Archive MaleCNS research sources, provenance, bibliography and immutable data inventory
This commit is contained in:
Digital Drosophila Archive
@@ -0,0 +1,3 @@
|
||||
# Preserve historical source bytes and SHA-256 across platforms. No EOL conversion.
|
||||
* -text
|
||||
*.jsonl.gz binary
|
||||
+36
@@ -0,0 +1,36 @@
|
||||
# Existing scientific outputs remain on the server and are indexed by SHA-256.
|
||||
/datasets/
|
||||
/experiments/
|
||||
/logs/
|
||||
/recordings/
|
||||
**/.venv/
|
||||
**/venv/
|
||||
**/__pycache__/
|
||||
**/.pytest_cache/
|
||||
**/.cache/
|
||||
**/node_modules/
|
||||
*.pyc
|
||||
*.so
|
||||
*.o
|
||||
*.a
|
||||
*.npy
|
||||
*.npz
|
||||
*.parquet
|
||||
*.feather
|
||||
*.swc
|
||||
*.pt
|
||||
*.pth
|
||||
*.ckpt
|
||||
*.tar
|
||||
*.zst
|
||||
*.zip
|
||||
*.log
|
||||
.env
|
||||
.env.*
|
||||
*.pem
|
||||
*.key
|
||||
id_rsa*
|
||||
id_ed25519*
|
||||
/registry/verification-runs/
|
||||
# Historical learned deltas/checkpoints are not source files.
|
||||
/models/learned/
|
||||
@@ -0,0 +1,18 @@
|
||||
# 权属、第三方资源与发布边界
|
||||
|
||||
本仓库整理既有项目，不改变原始数据和软件的权属。尚未为本项目原创代码选定统一公开许可证；不得将“可访问此 Git 仓库”等同于取得任意再分发授权。
|
||||
|
||||
| 外部资源 | 冻结副本中的许可证据 | 本仓库处理 |
|
||||
|---|---|---|
|
||||
| MaleCNS v1.0 官方数据 | 来源文件、原下载脚本和校验清单；统一再分发许可本次未确证 | 只登记路径、来源、大小、SHA-256；不上传数据 |
|
||||
| Cell Type Explorer | LICENSE：CC BY 4.0 | 保存来源和 commit，不嵌入外部仓库 |
|
||||
| reiserlab/visualpathways | LICENSE 为 GPL v3 文本；具体授权选项仍以 upstream 为准 | 同上 |
|
||||
| FlyVis | pyproject 声明 MIT classifier；冻结 checkout 未发现独立 LICENSE | 标明声明与证据局限，不擅自补许可证 |
|
||||
| DoOR.data 2.0.1 | DESCRIPTION：CC BY-SA 4.0 | 数据与论文许可证不混同 |
|
||||
| 论文 | 每篇出版商/作者条款 | 保存书目信息和官方链接，不自动打包全文 |
|
||||
|
||||
准确 commit、许可证据文件 SHA-256、工作树状态见 `registry/catalog-20260929/external-resources.lock.json`。
|
||||
Crossref 返回的 license URL 是元数据观察值，可能区分 TDM/VoR 等用途，不自动构成整个资源可再分发结论。
|
||||
公开发布前仍需逐项审查项目源代码的派生关系、数据条件和作者署名。
|
||||
|
||||
初始化提交使用透明的仓库机器人身份，而非伪造研究人员署名。提交时间表示归档时间，不表示各阶段实际完成时间。
|
||||
@@ -0,0 +1,59 @@
|
||||
# Digital Drosophila / MaleCNS 全脑模型研究
|
||||
|
||||
面向 MaleCNS v1.0 的可审计连接组、LIF 模型、感觉接口与局部可塑性机制研究。
|
||||
**这是研究工程仓库，不是已经实现自主学习或游戏控制的果蝇。**
|
||||
|
||||
截至 2026-09-29，最新封存阶段是 **Phase 4Z：COMPLETE-EVENT-RESOLVED-PN-AUDIT**。
|
||||
项目顶层验收口径为 **5/10（50%）**；这不是生物学逼真度、工作量完成率或剩余工期。
|
||||
最新正式候选运行时为 `MaleCNS-LIF-runtime-v2-deterministic-candidate`。
|
||||
`MaleCNS-TemporalEligibility-v2-candidate` 与 `MaleCNS-OlfactoryEncoder-v1-candidate` 保留。
|
||||
稳定信用分配、刺激特异性关联记忆和游戏学习尚未通过。
|
||||
|
||||
## 从这里开始
|
||||
|
||||
| 主题 | 文档 |
|
||||
|---|---|
|
||||
| 当前状态、验收比例、下一步 | [项目状态](docs/00-overview/status.md) |
|
||||
| 目录和数据流 | [架构与目录](docs/00-overview/architecture.md) |
|
||||
| 科学数据定义和来源 | [数据契约](docs/10-data/data-contract.md)、[来源登记](docs/10-data/provenance.md) |
|
||||
| 模型边界 | [模型与冻结契约](docs/20-model/baseline.md) |
|
||||
| 论文、引用与证据用途 | [参考文献入口](references/README.md)、[证据边界](references/evidence-map.md) |
|
||||
| 环境与复现 | [环境](docs/30-reproducibility/environment.md)、[恢复](docs/30-reproducibility/restore.md) |
|
||||
| Git 同步、大数据备份与安全 | [同步说明](docs/40-operations/git-sync.md)、[备份](docs/40-operations/backup.md) |
|
||||
| 每个阶段的原始证据副本 | [阶段索引](registry/catalog-20260929/stages.json) |
|
||||
| 全部科学文件路径/大小/SHA-256 | [快照说明](registry/README.md) |
|
||||
|
||||
## 关键事实
|
||||
|
||||
- 正式输入：`datasets/male-cns-v1.0/processed/neuron-only/`。
|
||||
- 166,700 个神经元；25,582,938 条有向边；124,177,617 个原始 synaptic contacts。
|
||||
- 原始 connectome、权重、神经递质定义和所有历史冻结阶段保持原位，只读管理。
|
||||
- 模型准确名称：**Shiu-like LIF dynamics adapted to MaleCNS v1.0**，不声称严格复现 Shiu 原模型。
|
||||
- 无游戏接入验收，无批准的自主行为学习候选；历史探索输出仍保留在服务器。
|
||||
|
||||
## Git 与数据不是同一个备份
|
||||
|
||||
服务器项目根目录：`/data/digital-drosophila/`。
|
||||
远程：`ssh://git@192.168.10.31:2222/qishizhe/digital-drosophila.git`。
|
||||
初始化分支：`codex/project-bootstrap`。
|
||||
|
||||
Git 保存源代码、配置、文档、书目、阶段证据副本和逐文件哈希清单。
|
||||
**Git 不保存原始 SWC/Feather、CSR、完整事件/状态、模型 checkpoint、虚拟环境或大型压缩包。**
|
||||
原项目实占约 29.72 GB（十进制、初始化前统计），不会因为初始化 Git 被删除、搬移或重写。
|
||||
仅 clone 本仓库不能恢复全数据；按[恢复说明](docs/30-reproducibility/restore.md)配置单独的数据副本。
|
||||
|
||||
## 安全的首次检查
|
||||
|
||||
```bash
|
||||
git clone --branch codex/project-bootstrap ssh://git@192.168.10.31:2222/qishizhe/digital-drosophila.git
|
||||
cd digital-drosophila
|
||||
python3 -B tools/repo_guard.py
|
||||
```
|
||||
|
||||
完整数据恢复后才能运行 SHA-256 验证。不要直接执行旧阶段的生成/封存脚本：部分脚本使用固定输出路径，会覆盖历史结果。
|
||||
本次仓库建设不重新运行任何科学阶段。
|
||||
|
||||
## 权属与引用
|
||||
|
||||
见 [NOTICE](NOTICE.md)。第三方数据、软件和论文各自保留其许可证；本仓库没有替用户擅自授予全项目开源许可证。
|
||||
论文索引不是论文全文打包；保留 DOI、官方全文入口、版本及证据用途，避免错误再分发。
|
||||
@@ -0,0 +1,32 @@
|
||||
# 目录与数据流
|
||||
|
||||
```text
|
||||
digital-drosophila/
|
||||
├── README.md / NOTICE.md
|
||||
├── docs/ 当前导航、科学边界、复现和运维
|
||||
├── references/ 书目、DOI、证据用途、来源元数据
|
||||
├── registry/ 科学文件快照、阶段报告副本、外部版本锁定
|
||||
├── tools/ 只读清单、校验与提交检查工具
|
||||
├── models/male-cns-lif/ 原位保留的各阶段 Python/CUDA/配置/manifest（Git 跟踪）
|
||||
├── models/learned/ 历史探索输出（仅服务器，非正式能力声明）
|
||||
├── datasets/ 原始、处理后、备份（仅服务器）
|
||||
├── experiments/ 各阶段完整事件、状态和审计（仅服务器）
|
||||
└── backend/frontend/game/logs/recordings/ 原目录保留；空目录不由 Git 保存
|
||||
```
|
||||
|
||||
```text
|
||||
official MaleCNS raw
|
||||
→ Phase 3B neuron-only / int32 index + uint64 body_id
|
||||
→ base CSR / raw int64 count（只读）
|
||||
→ independent sign + effective weight
|
||||
→ frozen LIF / deterministic-runtime-v2
|
||||
→ sparse events / states / audit
|
||||
|
||||
external physiology + canonical mapping → engineering sensory encoder → BrainIO
|
||||
independent diagnostic overlays → separate experiments（不替换 base）
|
||||
```
|
||||
|
||||
原目录未改名、未搬移、未整理重写。已有脚本大量使用 `/data/digital-drosophila` 绝对路径，保持这种布局是当前复现前提。
|
||||
根目录文档是导航层；旧模型 README 可能只描述某个早期阶段，不能单独作为全项目当前状态。
|
||||
|
||||
源代码与当前快照一起首次进入 Git，不会伪造过去每一阶段的提交历史。此前版本身份仍来自各阶段原 manifest/SHA-256。
|
||||
@@ -0,0 +1,42 @@
|
||||
# 状态与下一步（2026-09-29）
|
||||
|
||||
## 顶层验收口径
|
||||
|
||||
| 验收项 | 完成率 | 解释 |
|
||||
|---|---:|---|
|
||||
| 数据与 neuron-only canonical set | 100% | 保留原数据、映射和审计 |
|
||||
| LIF 与确定性运行时 | 100% | Phase 4P 数值契约范围内通过，不代表生理精确 |
|
||||
| BrainIO / ActivityStream | 100% | 已验证接口，不等于行为闭环 |
|
||||
| 有来源的嗅觉输入编码 | 100% | 编码候选通过，不等于 held-out 记忆通过 |
|
||||
| 局部 eligibility / plasticity 组件 | 100% | 组件级验收；不是关联记忆系统通过 |
|
||||
| 稳定信用分配 | 0% | 未满足正式验收 |
|
||||
| 刺激特异性关联记忆 | 0% | 未满足正式验收 |
|
||||
| 结构化视觉候选 | 0% | 历史 Phase 4E/4F 候选失败，保留失败证据 |
|
||||
| 动作闭环 | 0% | 未批准进入 |
|
||||
| 游戏中学习 | 0% | 未批准进入 |
|
||||
|
||||
合计 5/10 = 50%。0% 表示该 gate 尚未通过，不表示没有投入工作；100% 也不表示没有研究局限。
|
||||
|
||||
## 当前机制定位：Phase 4Z
|
||||
|
||||
正式状态 `COMPLETE-EVENT-RESOLVED-PN-AUDIT`，不是新学习模型候选。
|
||||
12 个主试次，6 种旧气味、seed 131101；复现既有参考和 DPM-off 独立诊断条件。
|
||||
12/12 Phase 4Y 复现、6/6 Phase 4W 全脑事件复现、132/132 CPU shadow replay、2/2 重复、5/5 冻结参考回归通过。
|
||||
|
||||
剩余 KC 82312（body 109321）晚期正到达权重约 99.38% 来自 ALPN；最大单源 VA6_adPN 133666（body 527577）约占 36.16–36.39%。
|
||||
真实事件支持 ALLN→PN→KC 持续驱动。局部回放中移除指定记录输入可消除晚期 spike，但**尚不是全脑环路必要性的干预证明**。
|
||||
|
||||
冻结 APL overlay 对全部 5,845 条快速出边置零，graded 替代只覆盖 KC；1,212 条非 KC 出边、13,712 contacts、999 targets 没有直接替代项。
|
||||
这包括被观察 PN 的 7 条边、247 contacts。它是模型覆盖缺口，不是本轮删边；末梢抑制不能未经证据直接换成整细胞负电流。
|
||||
|
||||
827 个相同请求中有 61 个接受状态不同、52 个请求时刻实际输出不同，来自网络状态/不应期；不能仅以请求总数相等声称实际 PN spike 相等。
|
||||
|
||||
## 后续优先顺序（计划，不在本次仓库整理中执行）
|
||||
|
||||
1. 审计 APL→PN 接触位置、作用范围和受体证据，未知继续记 unknown。
|
||||
2. 区分末梢抑制、整细胞电流、局部环路持续驱动；预先声明独立机制假设和可证伪 gate。
|
||||
3. 在确定性运行时及相同实际输入语义下做回归，保留全部失败条件。
|
||||
4. 只有稳定、特异性记忆与下游 readout 通过，才讨论动作接口；视觉和游戏另设验收。
|
||||
|
||||
不调 eta、阈值或脑增益来制造通过，不扩大 plastic mask，不替换 frozen baseline。
|
||||
各阶段准确状态以 `registry/catalog-20260929/evidence/` 的原始报告副本为准；不要把所有阶段统一写成 PASS。
|
||||
@@ -0,0 +1,30 @@
|
||||
# 数据契约
|
||||
|
||||
唯一正式全脑连接组输入：`datasets/male-cns-v1.0/processed/neuron-only/`。
|
||||
|
||||
| 项目 | 固定值 / 语义 |
|
||||
|---|---|
|
||||
| canonical neurons | 166,700；annotation 的 superclass 非空 |
|
||||
| fragments | 44,877；从 neuron-only 图排除，仍保留原数据与审计 |
|
||||
| annotation bodies | 211,577；不能全部称为 neurons |
|
||||
| directed edges | 25,582,938 |
|
||||
| anatomical synaptic-contact sum | 124,177,617 |
|
||||
| neuron_idx | 连续 int32，0…166699 |
|
||||
| body_id | 原始 uint64；不可与其他数据集 ID 混用 |
|
||||
| CSR indptr / indices / raw weights | int64 / int32 / int64 |
|
||||
| isolated neurons / self-loops | 保留 217 / 101 |
|
||||
| missing type | 保留 2,194 个 neurons |
|
||||
| nonempty raw type unique strings | 11,751；不为匹配论文数字手工合并 |
|
||||
|
||||
上表是已经封存的项目契约，不是本次重新计算的结果；以原 Phase 3B 审计、schema 和 manifest 为权威依据。
|
||||
正式 Cell 论文神经元总数为 166,700、types 11,710；与本地 raw 字符串统计不一致时保留版本/定义差异。
|
||||
|
||||
原始文件实际位置：
|
||||
|
||||
- `raw/annotations/body-annotations-male-cns-v1.0-minconf-0.5.feather`
|
||||
- `raw/annotations/body-neurotransmitters-male-cns-v1.0.feather`（不是空的 raw/neurotransmitters 目录）
|
||||
- `raw/connectivity/connectome-weights-male-cns-v1.0-minconf-0.5.feather`
|
||||
- `raw/skeletons/`：本次清点 211,573 个 SWC 文件；这不是 neuron-only 的 SWC 覆盖数。
|
||||
|
||||
不得由 SWC 文件总数推断 neuron coverage；需用 canonical body_id 做集合连接。不得改动 raw 以匹配论文。
|
||||
本项目没有下载原始 EM 图像。所有新增解释字段与原始 neurotransmitter 字段分离。
|
||||
@@ -0,0 +1,27 @@
|
||||
# 数据、外部资源与来源链
|
||||
|
||||
## 精确定位而非重新下载
|
||||
|
||||
完整逐文件路径、字节数、SHA-256 位于 `registry/snapshots/bootstrap-20260929/files.jsonl.gz`。
|
||||
它记录目前服务器实物，不替代上游官方校验，也不证明过去每一次实验的执行过程。
|
||||
原下载来源/校验文件保留在 `datasets/male-cns-v1.0/raw/_manifests/`，原 Phase 3B schema/audit 保留在 processed/neuron-only。
|
||||
自动引用抽取 `registry/catalog-20260929/source-citations.json` 同时保存来源文件 SHA-256；抽取到的 URL/DOI 未自动背书为有效生物学证据。
|
||||
|
||||
## 外部资源
|
||||
|
||||
| 资源 | 本地用途 | 不允许的推断 |
|
||||
|---|---|---|
|
||||
| MaleCNS v1.0 / Cell 2026 | canonical bodies、真实边、annotation | anatomical count ≠ 实测有效电导 |
|
||||
| reiserlab/visualpathways | pathway / receptive-field / retinotopy 参考 | 没有坐标的 neuron 不随机补坐标 |
|
||||
| Cell Type Explorer / eye maps | 视觉层级和 eye-map 参考 | 名称相似不证明跨数据集 ID 一致 |
|
||||
| FlyVis | 独立 engineering visual front-end 探索 | 不称为纯 MaleCNS phototransduction |
|
||||
| DoOR.data 2.0.1 | odor-response physiology、receptor/glomerulus 通道映射 | 不自动推断缺失 MaleCNS body 映射 |
|
||||
|
||||
四个外部 Git 仓库的 origin、commit、dirty 状态、许可证据 SHA-256 已锁在 `external-resources.lock.json`。
|
||||
外部仓库未 pull、未更新、未被作为嵌套 Git 内容上传。
|
||||
|
||||
## 版本与覆盖缺口
|
||||
|
||||
论文 formal / preprint、数据 release、软件 commit 分开标记。未知许可证、缺失映射或没有原始生理参数都显式保留 unknown。
|
||||
不能因为外部数据“公开可下载”就默认可全部再分发。
|
||||
不上传大型上游数据副本，恢复时优先使用自己已校验的备份；上游重下载是显式操作，不由本仓库自动执行。
|
||||
@@ -0,0 +1,17 @@
|
||||
# 实际 schema 与可恢复文件登记
|
||||
|
||||
`registry/data-20260929/dataset.json` 登记实际 raw 核心文件、Phase 3B canonical 文件、大小、SHA-256、原始下载脚本中的来源基址。
|
||||
`registry/data-20260929/observed-schemas.json` 通过只读 Arrow/Parquet/NumPy header 获取列名、dtype、数组形状；不是猜测字段，也不是重新生成 Phase 3B。
|
||||
`registry/data-20260929/evidence/` 保留小型 Phase 3B audit、schema、manifest、原下载来源和校验记录的字节一致副本。
|
||||
完整 SWC 列表和所有实验大文件仍通过全量 gzip JSONL 清单引用，不在本目录重复写 20 万行。
|
||||
|
||||
原下载脚本作为来源证据归档，**不是本轮获准重新执行的操作**。其中重下载、移走坏文件等行为本次均未运行。
|
||||
恢复时先取已校验备份；需要重下载时应在独立目的地显式审批来源和范围，不能改写当前 raw。
|
||||
|
||||
Git 内容本身另有 `registry/repository-files.sha256`，可在无科学数据的新 clone 上独立验证：
|
||||
|
||||
```bash
|
||||
python3 -B tools/verify_repository_files.py
|
||||
```
|
||||
|
||||
该清单不包含它自身（避免自引用），由 Git commit 内容身份覆盖其完整性；它不等于科学大数据备份。
|
||||
@@ -0,0 +1,26 @@
|
||||
# 模型与冻结边界
|
||||
|
||||
准确描述：**Shiu-like LIF dynamics adapted to MaleCNS v1.0**。
|
||||
参数是工程 baseline，不是 MaleCNS 单细胞实测值；histamine inventory 等与原 Shiu 实现不同。
|
||||
|
||||
| 参数 | 值 |
|
||||
|---|---:|
|
||||
| dt | 0.1 ms |
|
||||
| tau_m / tau_syn | 20 / 5 ms |
|
||||
| delay | 1.8 ms，18 steps |
|
||||
| refractory | 2.2 ms |
|
||||
| V_rest / V_reset | −52 / −52 mV |
|
||||
| V_threshold | −45 mV |
|
||||
| W_syn | 0.275 mV / anatomical synapse |
|
||||
|
||||
ACh、dopamine、octopamine、serotonin baseline sign 为 +1；GABA、glutamate、histamine 为 −1；unknown/unclear 为 0。
|
||||
这是被冻结的建模约定，**不是所有神经递质对所有靶细胞的真实生理符号定律**。
|
||||
特别是 DPM transmitter 与调制时间尺度的诊断不能自动改变 raw 或正式 sign。
|
||||
|
||||
历史 `reference-v1` 和 `runtime-v1` 保留。Phase 4O 暴露运行时确定性问题后，Phase 4P 产生独立 `deterministic-runtime-v2` 候选。
|
||||
使用该版本自身 numerical contract/reference v2，不用“重写旧 oracle”隐藏历史差异。
|
||||
数值精确事件复现、CPU/GPU 对照、BrainIO 接受/不应期语义、ActivityStream、checkpoint 等见 Phase 4P 原始证据副本。
|
||||
|
||||
Base CSR 原始 int64 weight 永远不被 sign/effective weight/overlay 覆盖；局部 plastic_delta 或 efficacy 属于单独研究表示。
|
||||
14,232-edge plastic mask 与 eligibility-v2 不因本次整理而更改。
|
||||
实验失败和诊断 overlay 全部保留，不选取“看起来像学习”的试次替代正式 gate。
|
||||
@@ -0,0 +1,12 @@
|
||||
# 环境与可重复性范围
|
||||
|
||||
当前服务器环境快照：`registry/catalog-20260929/environment.json`。
|
||||
记录 Python、已安装包的名称/版本、GPU/显存/driver、OS、Git、nvcc；不导出含凭据的包索引 URL 或环境变量。
|
||||
该快照反映归档时环境，不保证等同于每个早期阶段的原运行环境，也不是 wheel/CUDA 二进制供应链锁定。
|
||||
|
||||
原主环境位于 `models/male-cns-lif/.venv/`，另有 processed/.venv；两个环境本次均不修改，不入 Git。
|
||||
建议新机器在独立环境按记录重建，保存 wheel 来源/哈希与编译日志后再做科学回归。
|
||||
本次没有安装、升级依赖或重编译科学内核。
|
||||
|
||||
历史脚本可能写固定绝对目录。任何科学复现须先检查入口及输出路径，在独立实验目录运行，绝不在封存目录执行 finalize。
|
||||
本次只执行存储完整性和仓库工具测试，不声称重新运行了 Phase 4C/4P 科学回归；这些既有结果通过原报告副本保留。
|
||||
@@ -0,0 +1,20 @@
|
||||
# 新服务器恢复
|
||||
|
||||
1. 克隆 `codex/project-bootstrap` 分支，确认提交 SHA；不要克隆覆盖现有有数据目录。
|
||||
2. 阅读 `registry/catalog-20260929/environment.json`，单独重建运行环境。不要盲目安装最新依赖。
|
||||
3. 从已有数据备份/数据存储恢复 `datasets/`、`experiments/`、必要的 `models/learned/`。保持相对路径，先恢复到空的独立目录，避免覆盖现存数据。
|
||||
4. 验证下方 SHA-256 快照；失败时停止，不重生成 raw 或篡改清单。
|
||||
5. 文件校验后，再在独立输出目录运行适用版本的数值 regression。文件相同不等于新硬件上科学行为已通过。
|
||||
|
||||
```bash
|
||||
# root 应是完整恢复的数据根；output 必须不存在。
|
||||
python3 -B tools/repository_audit.py --root /data/digital-drosophila verify \
|
||||
--snapshot registry/snapshots/bootstrap-20260929 \
|
||||
--output registry/verification-runs/restore-YYYYMMDD-HHMMSS.json
|
||||
```
|
||||
|
||||
快照覆盖 datasets、experiments、models 及原其他科学目录；跳过虚拟环境、缓存、嵌套 Git 数据库，不追随符号链接。
|
||||
输出 mismatch / missing / added；返回非零即失败。当前快照与历史冻结 checksum 功能不同：前者验证仓库整理前后字节不变，后者验证各阶段当时封存的契约。
|
||||
|
||||
如果只 clone 代码而没有恢复数据，校验失败是预期，不要通过删除清单记录“修复”。
|
||||
目前仅有 Phase 1–2 的 3.46 GB 压缩包，不能凭它恢复 Phase 3–4Z 全部实验。
|
||||
@@ -0,0 +1,29 @@
|
||||
# 大数据同步与备份（尚未配置异机目的地）
|
||||
|
||||
初始化前本机项目磁盘占用约 29,724,987,392 bytes（29.72 GB）：
|
||||
|
||||
| 分类 | 磁盘占用 bytes |
|
||||
|---|---:|
|
||||
| datasets | 15,881,289,728 |
|
||||
| experiments | 7,598,972,928 |
|
||||
| models（主要为环境） | 6,244,691,968 |
|
||||
|
||||
两个虚拟环境共约 6.46 GB；排除环境仍有约 23.26 GB，包括原始/派生数据、已有压缩包和外部代码副本。
|
||||
逐文件快照使用逻辑 bytes，与 du 实占、Git 压缩包大小不是同一指标，不应混用。
|
||||
历史 Phase 1–2 `.tar.zst` 为 3,464,817,549 bytes；它不是全阶段备份。
|
||||
|
||||
## 推荐分层
|
||||
|
||||
- Git：小代码、YAML、报告、书目、外部版本锁和 SHA-256 索引。
|
||||
- 版本化文件存储/NAS/对象存储：原始 SWC/Feather、CSR、Parquet/NPZ 事件、实验状态和历史压缩包。
|
||||
- 可重建环境：记录包/driver/compiler版本，按需另备离线 wheel 和编译依赖。
|
||||
|
||||
第二层尚未指定目的地、容量、访问权限与保留策略，所以本次**没有**声称完成异机全量备份。
|
||||
Git 服务所在 IP 不自动等于获准的数据存储目录；不擅自把 23 GB 数据复制到服务根目录。
|
||||
|
||||
## 后续执行要求
|
||||
|
||||
选择明确的新备份目录，确认空间后先 dry-run，再增量/断点传输；默认不删除目的地文件。
|
||||
恢复到独立空目录后运行完整 SHA-256 验证，记录 source/destination/snapshot hash/时间/结果。
|
||||
至少保留一份独立故障域副本；同盘压缩包和同机 Git 不抵抗磁盘损坏。
|
||||
数据同步与历史版本备份不是一回事，禁止用无版本镜像替代保留策略。
|
||||
@@ -0,0 +1,26 @@
|
||||
# Git 同步规则
|
||||
|
||||
远程：`ssh://git@192.168.10.31:2222/qishizhe/digital-drosophila.git`。
|
||||
服务器→Git 服务免密读取已确认；本机→计算服务器是另一条独立认证链。
|
||||
首次建立 `codex/project-bootstrap`，不 force push，不伪造旧阶段 commits。
|
||||
|
||||
```bash
|
||||
cd /data/digital-drosophila
|
||||
git status --short
|
||||
git diff --stat
|
||||
# 只添加明确需要的代码、文档和新清单，先审查再提交。
|
||||
git add docs references registry tools README.md NOTICE.md .gitignore .gitattributes
|
||||
python3 -B tools/repo_guard.py
|
||||
git diff --cached --stat
|
||||
git commit -m 'Document audited project update'
|
||||
git push origin codex/project-bootstrap
|
||||
```
|
||||
|
||||
源代码更新须显式列出路径；不要为整理仓库修改已封存模型文件。
|
||||
`.gitattributes` 禁用文本换行转换以保留旧源码哈希；本仓库 local core.autocrlf=false。
|
||||
不要运行 `git clean -fdx`、强制 reset、会删 ignored 数据的清理工具，或对现有项目做镜像删除同步。
|
||||
不要使用 `git add -f datasets experiments` 绕过防护。
|
||||
|
||||
`repo_guard.py` 检查 staged blob：数据/环境/嵌套仓库、>25 MiB 文件、常见凭据标记；它不是完备的 DLP/安全扫描器，仍需人工审查。
|
||||
首次提交身份 `Digital Drosophila Archive <archive@localhost>` 是归档机器人，不代表研究作者。
|
||||
仓库 default branch 可由服务管理员选择；本次不改服务的全局设置。
|
||||
@@ -0,0 +1,20 @@
|
||||
# MaleCNS LIF Phase 4A
|
||||
|
||||
Auditable, single-RTX-4090 whole-connectome LIF baseline. It consumes only the
|
||||
validated Phase 3B neuron-only products and never writes to the dataset tree.
|
||||
|
||||
Run with the exact GPU UUID to prevent accidental V100 or mixed-GPU use:
|
||||
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=GPU-79f3a85d-adce-7833-7003-6488140ca596 \
|
||||
.venv/bin/python phase4a.py \
|
||||
--config /data/digital-drosophila/experiments/phase4a-baseline/config/baseline.yaml
|
||||
```
|
||||
|
||||
The raw `int64` synapse-count array remains read-only. The configured transform
|
||||
creates a separate signed `float32` effective-weight array. Spike output is a
|
||||
sparse Parquet event table, never a dense time-by-neuron matrix.
|
||||
|
||||
The baseline parameters are FlyWire-inspired modeling defaults, not MaleCNS
|
||||
single-cell measurements. Synthetic drive amplitudes are deterministic smoke-test
|
||||
inputs and are not a biological calibration.
|
||||
@@ -0,0 +1,110 @@
|
||||
phase: 4A
|
||||
paths:
|
||||
connectome_root: /data/digital-drosophila/datasets/male-cns-v1.0/processed/neuron-only
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4a-baseline
|
||||
|
||||
expected_graph:
|
||||
neurons: 166700
|
||||
directed_edges: 25582938
|
||||
synaptic_contact_weight_sum: 124177617
|
||||
isolated_neurons: 217
|
||||
self_loops: 101
|
||||
|
||||
device:
|
||||
required_name_substring: RTX 4090
|
||||
launcher_visible_gpu_uuid: GPU-79f3a85d-adce-7833-7003-6488140ca596
|
||||
mixed_multi_gpu: false
|
||||
|
||||
model:
|
||||
integrator: forward_euler
|
||||
dt_ms: 0.1
|
||||
membrane_tau_ms: 20.0
|
||||
synaptic_tau_ms: 5.0
|
||||
refractory_ms: 2.2
|
||||
resting_potential_mv: -52.0
|
||||
reset_potential_mv: -52.0
|
||||
threshold_mv: -45.0
|
||||
synaptic_delay_ms: 1.8
|
||||
parameter_provenance: FlyWire-inspired 2024 LIF baseline; not MaleCNS measured single-cell parameters
|
||||
|
||||
sign_policy:
|
||||
neurotransmitter_sign:
|
||||
acetylcholine: 1
|
||||
gaba: -1
|
||||
glutamate: -1
|
||||
dopamine: 1
|
||||
octopamine: 1
|
||||
serotonin: 1
|
||||
histamine: -1
|
||||
histamine_policy_note: explicitly configured inhibitory baseline assumption; not silently inferred
|
||||
unknown_sign: 0
|
||||
unclear_sign: 0
|
||||
unmapped_policy: error
|
||||
unmapped_sign: 0
|
||||
|
||||
weights:
|
||||
transform: sqrt
|
||||
supported_transforms: [linear, sqrt, log1p]
|
||||
scale: 0.05
|
||||
chunk_edges: 2000000
|
||||
units: model synaptic-state increment per presynaptic spike
|
||||
formula: effective_weight = presynaptic_sign * scale * f(raw_synapse_count)
|
||||
preserve_raw_int64: true
|
||||
|
||||
stimulations:
|
||||
single_neuron:
|
||||
kind: single_neuron_pulse
|
||||
neuron_idx: 0
|
||||
start_ms: 2.0
|
||||
duration_ms: 1.0
|
||||
amplitude_mv_equivalent: 400.0
|
||||
known_type_AMMC005:
|
||||
kind: known_type_pulse
|
||||
type_name: AMMC005
|
||||
max_neurons: 12
|
||||
start_ms: 5.0
|
||||
duration_ms: 1.0
|
||||
amplitude_mv_equivalent: 400.0
|
||||
sparse_poisson:
|
||||
kind: sparse_poisson
|
||||
target_fraction: 0.001
|
||||
rate_hz: 20.0
|
||||
event_amplitude_mv_equivalent: 1600.0
|
||||
|
||||
smoke_tests:
|
||||
- name: smoke-10ms-single-neuron
|
||||
duration_ms: 10.0
|
||||
stimulation: single_neuron
|
||||
seed: 41001
|
||||
- name: smoke-100ms-known-type
|
||||
duration_ms: 100.0
|
||||
stimulation: known_type_AMMC005
|
||||
seed: 41002
|
||||
- name: smoke-1s-sparse-poisson
|
||||
duration_ms: 1000.0
|
||||
stimulation: sparse_poisson
|
||||
seed: 41003
|
||||
|
||||
cpu_gpu_reference:
|
||||
seed: 41100
|
||||
subgraph_neurons: 512
|
||||
steps: 100
|
||||
pulse_start_step: 5
|
||||
pulse_steps: 10
|
||||
pulse_neurons: 4
|
||||
pulse_amplitude_mv_equivalent: 400.0
|
||||
absolute_tolerance: 1.0e-10
|
||||
|
||||
validation:
|
||||
finite_check_interval_steps: 100
|
||||
explosive_step_fraction: 0.1
|
||||
memory_growth_tolerance_bytes: 10485760
|
||||
|
||||
output:
|
||||
spike_representation: event_table_timestamp_ms_neuron_idx
|
||||
dense_time_by_neuron_forbidden: true
|
||||
rate_window_ms: 10.0
|
||||
|
||||
reproducibility:
|
||||
seed: 41000
|
||||
deterministic_synthetic_inputs: true
|
||||
@@ -0,0 +1,415 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stable game-agnostic Brain I/O API for MaleCNS-LIF-runtime-v1."""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
import numpy as np
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
|
||||
API_VERSION = "1.0.0"
|
||||
INPUT_BATCH_VERSION = "1.0.0"
|
||||
ACTIVITY_STREAM_VERSION = "1.0.0"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InputEvent:
|
||||
target_neuron_idx: np.ndarray
|
||||
stimulation_mode: str
|
||||
value_kind: str
|
||||
value: float | np.ndarray
|
||||
start_ms: float
|
||||
duration_ms: float
|
||||
source_id: str
|
||||
channel_id: str
|
||||
amplitude_mv: float = 0.0
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def validate(self, neuron_count: int, dt_ms: float) -> None:
|
||||
target = np.asarray(self.target_neuron_idx)
|
||||
if target.ndim != 1 or target.dtype.kind not in "iu":
|
||||
raise ValueError("target_neuron_idx must be a one-dimensional integer array")
|
||||
if len(target) and (int(target.min()) < 0 or int(target.max()) >= neuron_count):
|
||||
raise ValueError("target neuron_idx outside canonical range")
|
||||
if len(np.unique(target)) != len(target):
|
||||
raise ValueError("target neuron_idx values must be unique")
|
||||
if self.stimulation_mode not in {"spike_injection", "external_synaptic_drive"}:
|
||||
raise ValueError(f"unsupported stimulation_mode {self.stimulation_mode!r}")
|
||||
if self.value_kind not in {"binary", "scalar_intensity", "poisson_rate_hz", "signed_modulation"}:
|
||||
raise ValueError(f"unsupported value_kind {self.value_kind!r}")
|
||||
if self.start_ms < 0 or self.duration_ms <= 0:
|
||||
raise ValueError("timestamps/duration must be non-negative/positive")
|
||||
for x in (self.start_ms, self.duration_ms):
|
||||
if not math.isclose(x / dt_ms, round(x / dt_ms), abs_tol=1e-8):
|
||||
raise ValueError(f"{x} ms is not aligned to dt={dt_ms} ms")
|
||||
value = np.asarray(self.value)
|
||||
if value.ndim > 1 or (value.ndim == 1 and len(value) not in (1, len(target))):
|
||||
raise ValueError("value must be scalar or match target count")
|
||||
if self.value_kind == "poisson_rate_hz" and np.any(value < 0):
|
||||
raise ValueError("Poisson rate cannot be negative")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InputBatch:
|
||||
events: tuple[InputEvent, ...]
|
||||
batch_id: str
|
||||
version: str = INPUT_BATCH_VERSION
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def validate(self, neuron_count: int, dt_ms: float) -> None:
|
||||
if self.version != INPUT_BATCH_VERSION:
|
||||
raise ValueError(f"InputBatch version must be {INPUT_BATCH_VERSION}")
|
||||
for event in self.events:
|
||||
event.validate(neuron_count, dt_ms)
|
||||
|
||||
|
||||
class VisualAdapterV1:
|
||||
"""Maps abstract channels to audited visual hierarchy populations."""
|
||||
|
||||
def __init__(self, config: dict[str, Any], phase4c_root: Path):
|
||||
self.config = config
|
||||
self.phase4c_root = phase4c_root
|
||||
self.modes: dict[str, np.ndarray] = {}
|
||||
for mode, spec in config["visual_adapter"]["modes"].items():
|
||||
population = spec["hierarchy_population"]
|
||||
path = phase4c_root / "visual-audit" / f"hierarchy-{population}.parquet"
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
self.modes[mode] = pq.read_table(path, columns=["neuron_idx"])["neuron_idx"].to_numpy().astype(np.int64)
|
||||
|
||||
def population(self, mode: str) -> np.ndarray:
|
||||
if mode not in self.modes:
|
||||
raise KeyError(mode)
|
||||
return self.modes[mode].copy()
|
||||
|
||||
def encode(
|
||||
self, mode: str, channel_values: np.ndarray, start_ms: float, duration_ms: float,
|
||||
source_id: str, channel_id: str, value_kind: str = "poisson_rate_hz",
|
||||
) -> InputBatch:
|
||||
if mode not in self.modes:
|
||||
raise KeyError(mode)
|
||||
spec = self.config["visual_adapter"]["modes"][mode]
|
||||
targets = self.modes[mode]
|
||||
values = np.asarray(channel_values, dtype=np.float32)
|
||||
if values.shape != (len(targets),):
|
||||
raise ValueError(f"visual channel must have shape {(len(targets),)}, got {values.shape}")
|
||||
selected = np.flatnonzero(values != 0).astype(np.int64)
|
||||
target = targets[selected]
|
||||
selected_values = values[selected]
|
||||
event = InputEvent(
|
||||
target_neuron_idx=target,
|
||||
stimulation_mode=spec["input_mode"],
|
||||
value_kind=value_kind,
|
||||
value=selected_values,
|
||||
amplitude_mv=float(self.config["visual_adapter"]["external_drive_mv"]),
|
||||
start_ms=start_ms,
|
||||
duration_ms=duration_ms,
|
||||
source_id=source_id,
|
||||
channel_id=channel_id,
|
||||
metadata={
|
||||
"adapter": self.config["names"]["visual_adapter"],
|
||||
"adapter_mode": mode,
|
||||
"biological_fidelity": spec["biological_fidelity"],
|
||||
"connectome_weights_modified": False,
|
||||
},
|
||||
)
|
||||
return InputBatch(events=(event,), batch_id=f"{source_id}:{channel_id}:{start_ms}", metadata={"adapter_mode": mode})
|
||||
|
||||
|
||||
class BrainRuntime:
|
||||
"""Persistent exact-state BrainRuntime API. It has no game-specific dependency."""
|
||||
|
||||
def __init__(self, config_path: Path, extension_build_dir: Path):
|
||||
self.config_path = config_path.resolve()
|
||||
self.cfg = yaml.safe_load(self.config_path.read_text(encoding="utf-8"))
|
||||
self.dt = float(self.cfg["model"]["dt_ms"])
|
||||
self.n = 166700
|
||||
self.model_root = Path(self.cfg["paths"]["model_root"])
|
||||
self.runtime_root = Path(self.cfg["paths"]["runtime_root"])
|
||||
self.input_root = Path(self.cfg["paths"]["connectome_root"])
|
||||
runtime_module_dir = str(self.runtime_root)
|
||||
if runtime_module_dir not in sys.path:
|
||||
sys.path.insert(0, runtime_module_dir)
|
||||
import phase4c as runtime_module
|
||||
self.runtime_module = runtime_module
|
||||
p4cfg = yaml.safe_load((self.runtime_root / "phase4c.yaml").read_text(encoding="utf-8"))
|
||||
self.paths = runtime_module.exact_paths(p4cfg)
|
||||
self.indptr_np, self.indices_np, self.raw_np = runtime_module.p4b.validate_graph(p4cfg, self.paths)
|
||||
self.table = runtime_module.full_metadata(self.paths["metadata"])
|
||||
self.population_masks = runtime_module.populations(self.table)
|
||||
if not torch.cuda.is_available() or torch.cuda.device_count() != 1:
|
||||
raise RuntimeError("BrainRuntime requires exactly one visible CUDA GPU")
|
||||
self.device = torch.device("cuda:0")
|
||||
props = torch.cuda.get_device_properties(self.device)
|
||||
if self.cfg["device"]["required_name_substring"] not in props.name:
|
||||
raise RuntimeError(f"required RTX 4090, got {props.name}")
|
||||
extension_build_dir.mkdir(parents=True, exist_ok=True)
|
||||
build_experiment_root = extension_build_dir.parent.parent
|
||||
(build_experiment_root / "compiler-reports/extension-build").mkdir(parents=True, exist_ok=True)
|
||||
self.extension, report = runtime_module.build_extension(self.runtime_root, build_experiment_root)
|
||||
if self.extension is None:
|
||||
raise RuntimeError(f"runtime-v1 extension unavailable: {report}")
|
||||
base = runtime_module.Runtime(p4cfg, self.table, self.indptr_np, self.indices_np, self.raw_np, self.device, self.extension)
|
||||
self.indptr, self.indices, self.weights = base.indptr, base.indices, base.weights
|
||||
self.signs, self.superclass = base.signs, base.superclass
|
||||
self.neuron_type = np.asarray(self.table["annotation_type"].to_pylist(), dtype=object)
|
||||
self.neuron_class = np.asarray(self.table["annotation_class"].to_pylist(), dtype=object)
|
||||
self.body_id = self.table["body_id"].to_numpy().astype(np.uint64)
|
||||
m = self.cfg["model"]
|
||||
self.delay_steps = int(round(float(m["synaptic_delay_ms"]) / self.dt))
|
||||
self.queue_len = self.delay_steps + 1
|
||||
self.refractory_steps = int(round(float(m["refractory_ms"]) / self.dt))
|
||||
self.alpha = self.dt / float(m["membrane_tau_ms"])
|
||||
self.g_decay = 1.0 - self.dt / float(m["synaptic_tau_ms"])
|
||||
self.rest = float(m["resting_potential_mv"])
|
||||
self.reset_value = float(m["reset_potential_mv"])
|
||||
self.threshold = float(m["threshold_mv"])
|
||||
self.output_populations = self._output_populations()
|
||||
self.reset(0)
|
||||
|
||||
def _output_populations(self) -> dict[str, np.ndarray]:
|
||||
s = self.superclass
|
||||
return {
|
||||
"descending": np.flatnonzero(np.isin(s, ["descending_neuron", "descending_neuron_tbc"])).astype(np.int32),
|
||||
"motor_related": np.flatnonzero(np.isin(s, ["vnc_motor", "cb_motor"])).astype(np.int32),
|
||||
"ascending": np.flatnonzero(np.isin(s, ["ascending_neuron", "sensory_ascending", "sensory_ascending_tbc", "efferent_ascending"])).astype(np.int32),
|
||||
}
|
||||
|
||||
def reset(self, seed: int, initial_state: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
self.seed = int(seed)
|
||||
self.generator = torch.Generator(device=self.device); self.generator.manual_seed(self.seed)
|
||||
self.v = torch.full((self.n,), self.rest, dtype=torch.float32, device=self.device)
|
||||
self.g = torch.zeros(self.n, dtype=torch.float32, device=self.device)
|
||||
self.refractory = torch.zeros(self.n, dtype=torch.int32, device=self.device)
|
||||
self.delay_queue = torch.zeros((self.queue_len, self.n), dtype=torch.float32, device=self.device)
|
||||
self.spike_counts = torch.zeros(self.n, dtype=torch.int32, device=self.device)
|
||||
self.simulation_step = 0; self.sequence_id = 0
|
||||
self.spike_steps: list[np.ndarray] = []; self.spike_nodes: list[np.ndarray] = []
|
||||
self.spike_origins: list[np.ndarray] = []
|
||||
self.drive_steps: list[np.ndarray] = []; self.drive_nodes: list[np.ndarray] = []
|
||||
self.drive_values: list[np.ndarray] = []; self.input_audit: list[dict[str, Any]] = []
|
||||
if initial_state is not None:
|
||||
self.restore_state(initial_state)
|
||||
return self.get_state_summary()
|
||||
|
||||
@property
|
||||
def simulation_time_ms(self) -> float:
|
||||
return self.simulation_step * self.dt
|
||||
|
||||
def _event_values(self, event: InputEvent, targets: torch.Tensor) -> torch.Tensor:
|
||||
value = np.asarray(event.value, dtype=np.float32)
|
||||
if value.ndim == 0:
|
||||
return torch.full((len(targets),), float(value), dtype=torch.float32, device=self.device)
|
||||
return torch.as_tensor(value, dtype=torch.float32, device=self.device)
|
||||
|
||||
def _active(self, event: InputEvent, time_ms: float) -> bool:
|
||||
return event.start_ms - 1e-9 <= time_ms < event.start_ms + event.duration_ms - 1e-9
|
||||
|
||||
def _hybrid_edges(self, sources: torch.Tensor) -> torch.Tensor:
|
||||
starts = self.indptr[sources]
|
||||
degrees = self.indptr[sources + 1] - starts
|
||||
cumulative = torch.cumsum(degrees, 0)
|
||||
total = int(cumulative[-1].item()) if cumulative.numel() else 0
|
||||
output = torch.empty(total, dtype=torch.int64, device=self.device)
|
||||
if total:
|
||||
self.extension.fill_edges(self.indptr, sources, cumulative - degrees, output)
|
||||
return output
|
||||
|
||||
def step(self, input_batch: InputBatch | None, biological_ms: float) -> dict[str, Any]:
|
||||
steps = int(round(float(biological_ms) / self.dt))
|
||||
if not math.isclose(steps * self.dt, float(biological_ms), abs_tol=1e-8) or steps <= 0:
|
||||
raise ValueError("biological_ms must be a positive multiple of dt")
|
||||
batch = input_batch or InputBatch(events=(), batch_id="zero-input")
|
||||
batch.validate(self.n, self.dt)
|
||||
batch_start = self.simulation_time_ms
|
||||
prepared = [(e, torch.as_tensor(e.target_neuron_idx, dtype=torch.int64, device=self.device)) for e in batch.events]
|
||||
for event in batch.events:
|
||||
self.input_audit.append({
|
||||
"batch_id": batch.batch_id, "source_id": event.source_id, "channel_id": event.channel_id,
|
||||
"mode": event.stimulation_mode, "value_kind": event.value_kind,
|
||||
"start_ms": event.start_ms, "duration_ms": event.duration_ms,
|
||||
"target_count": int(len(event.target_neuron_idx)), "metadata": event.metadata,
|
||||
"external_drive_is_connectome_transmission": False,
|
||||
})
|
||||
start_event_count = sum(len(x) for x in self.spike_nodes)
|
||||
start_drive_count = sum(len(x) for x in self.drive_nodes)
|
||||
for _ in range(steps):
|
||||
global_step = self.simulation_step; time_ms = self.simulation_time_ms
|
||||
slot = global_step % self.queue_len
|
||||
self.g.mul_(self.g_decay).add_(self.delay_queue[slot]); self.delay_queue[slot].zero_()
|
||||
spike_requests: list[tuple[InputEvent, torch.Tensor, torch.Tensor]] = []
|
||||
for event, targets in prepared:
|
||||
if not self._active(event, time_ms) or targets.numel() == 0:
|
||||
continue
|
||||
values = self._event_values(event, targets)
|
||||
if event.value_kind == "poisson_rate_hz":
|
||||
selected = torch.rand(len(targets), generator=self.generator, device=self.device) < (values * self.dt / 1000.0)
|
||||
elif event.value_kind == "binary":
|
||||
selected = values != 0
|
||||
elif event.value_kind == "scalar_intensity":
|
||||
selected = values != 0
|
||||
else: # signed_modulation
|
||||
selected = torch.rand(len(targets), generator=self.generator, device=self.device) < (torch.abs(values) * self.dt / 1000.0)
|
||||
if event.stimulation_mode == "external_synaptic_drive":
|
||||
chosen = targets[selected]
|
||||
if chosen.numel():
|
||||
signed_value = values[selected] * float(event.amplitude_mv)
|
||||
if event.value_kind == "poisson_rate_hz":
|
||||
signed_value = torch.full_like(signed_value, float(event.amplitude_mv))
|
||||
self.g.index_add_(0, chosen, signed_value)
|
||||
node_np = chosen.cpu().numpy().astype(np.int32, copy=True)
|
||||
val_np = signed_value.cpu().numpy().astype(np.float32, copy=True)
|
||||
self.drive_steps.append(np.full(len(node_np), global_step, dtype=np.int32))
|
||||
self.drive_nodes.append(node_np); self.drive_values.append(val_np)
|
||||
else:
|
||||
spike_requests.append((event, targets, selected))
|
||||
self.refractory.clamp_min_(1).sub_(1)
|
||||
available = self.refractory == 0
|
||||
self.v[available] += self.alpha * (self.g[available] - (self.v[available] - self.rest))
|
||||
self.v[~available] = self.reset_value
|
||||
natural = available & (self.v >= self.threshold)
|
||||
spikes = natural.clone()
|
||||
injected = torch.zeros(self.n, dtype=torch.bool, device=self.device)
|
||||
for _event, targets, selected in spike_requests:
|
||||
accepted = selected & available[targets]
|
||||
chosen = targets[accepted]
|
||||
if chosen.numel():
|
||||
spikes[chosen] = True; injected[chosen] = ~natural[chosen]
|
||||
sources = torch.nonzero(spikes, as_tuple=False).flatten()
|
||||
self.spike_counts.add_(spikes.to(torch.int32))
|
||||
if sources.numel():
|
||||
self.v[sources] = self.reset_value; self.refractory[sources] = self.refractory_steps
|
||||
edge_ids = self._hybrid_edges(sources)
|
||||
if edge_ids.numel():
|
||||
self.delay_queue[(global_step + self.delay_steps) % self.queue_len].scatter_add_(0, self.indices[edge_ids].long(), self.weights[edge_ids])
|
||||
src_np = sources.cpu().numpy().astype(np.int32, copy=True)
|
||||
origin_np = np.where(injected[sources].cpu().numpy(), "external_spike_injection", "network_spike")
|
||||
self.spike_steps.append(np.full(len(src_np), global_step, dtype=np.int32))
|
||||
self.spike_nodes.append(src_np); self.spike_origins.append(origin_np)
|
||||
self.simulation_step += 1
|
||||
torch.cuda.synchronize()
|
||||
self.sequence_id += 1
|
||||
return {
|
||||
"api_version": API_VERSION, "sequence_id": self.sequence_id,
|
||||
"batch_id": batch.batch_id, "start_time_ms": batch_start,
|
||||
"end_time_ms": self.simulation_time_ms, "biological_ms": biological_ms,
|
||||
"new_spikes": sum(len(x) for x in self.spike_nodes) - start_event_count,
|
||||
"new_external_drive_events": sum(len(x) for x in self.drive_nodes) - start_drive_count,
|
||||
}
|
||||
|
||||
def _events(self) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
steps = np.concatenate(self.spike_steps) if self.spike_steps else np.empty(0, np.int32)
|
||||
nodes = np.concatenate(self.spike_nodes) if self.spike_nodes else np.empty(0, np.int32)
|
||||
origins = np.concatenate(self.spike_origins) if self.spike_origins else np.empty(0, object)
|
||||
return steps, nodes, origins
|
||||
|
||||
def read_spikes(self, window_ms: float | None = None) -> dict[str, Any]:
|
||||
steps, nodes, origins = self._events()
|
||||
if window_ms is not None:
|
||||
threshold = self.simulation_step - int(round(window_ms / self.dt))
|
||||
mask = steps >= threshold; steps, nodes, origins = steps[mask], nodes[mask], origins[mask]
|
||||
return {
|
||||
"simulation_time_ms": self.simulation_time_ms,
|
||||
"step": steps, "timestamp_ms": steps.astype(np.float64) * self.dt,
|
||||
"neuron_idx": nodes, "origin": origins,
|
||||
}
|
||||
|
||||
def read_population(self, population_id: str, window_ms: float) -> dict[str, Any]:
|
||||
if population_id not in self.output_populations:
|
||||
raise KeyError(population_id)
|
||||
members = self.output_populations[population_id]
|
||||
events = self.read_spikes(window_ms)
|
||||
mask = np.isin(events["neuron_idx"], members)
|
||||
nodes = events["neuron_idx"][mask]
|
||||
counts = np.zeros(len(members), dtype=np.int32)
|
||||
if len(nodes):
|
||||
lookup = np.full(self.n, -1, dtype=np.int32); lookup[members] = np.arange(len(members), dtype=np.int32)
|
||||
np.add.at(counts, lookup[nodes], 1)
|
||||
seconds = window_ms / 1000.0
|
||||
return {
|
||||
"simulation_time_ms": self.simulation_time_ms, "population_id": population_id,
|
||||
"neuron_idx": members.copy(), "spike_counts": counts,
|
||||
"firing_rate_hz": counts.astype(np.float32) / seconds,
|
||||
"active_mask": counts > 0, "aggregate_rate_hz": float(counts.sum() / (len(members) * seconds)),
|
||||
"total_spikes": int(counts.sum()), "active_neurons": int(np.count_nonzero(counts)),
|
||||
}
|
||||
|
||||
def read_output(self, output_spec: dict[str, Any], window_ms: float) -> dict[str, Any]:
|
||||
population = output_spec.get("population", "descending")
|
||||
aggregation = output_spec.get("aggregation", "per-neuron")
|
||||
base = self.read_population(population, window_ms)
|
||||
members = base["neuron_idx"]; counts = base["spike_counts"]
|
||||
if aggregation == "per-neuron":
|
||||
return base
|
||||
if aggregation not in {"per-type", "per-class"}:
|
||||
raise ValueError("aggregation must be per-neuron, per-type, or per-class")
|
||||
labels = self.neuron_type if aggregation == "per-type" else self.neuron_class
|
||||
result: dict[str, dict[str, float | int]] = {}
|
||||
for label in sorted({"<null>" if labels[i] is None else str(labels[i]) for i in members}):
|
||||
mask = np.asarray([("<null>" if labels[i] is None else str(labels[i])) == label for i in members])
|
||||
result[label] = {"neuron_count": int(mask.sum()), "spike_count": int(counts[mask].sum()), "mean_rate_hz": float(base["firing_rate_hz"][mask].mean())}
|
||||
return {"simulation_time_ms": self.simulation_time_ms, "population_id": population, "aggregation": aggregation, "features": result}
|
||||
|
||||
def fixed_temporal_bins(self, population_id: str, bin_ms: float, window_ms: float) -> dict[str, Any]:
|
||||
if bin_ms not in self.cfg["readout"]["bins_ms"]:
|
||||
raise ValueError("unsupported fixed bin")
|
||||
ev = self.read_spikes(window_ms); members = self.output_populations[population_id]
|
||||
mask = np.isin(ev["neuron_idx"], members); times = ev["timestamp_ms"][mask]
|
||||
start = self.simulation_time_ms - window_ms
|
||||
edges = np.arange(start, self.simulation_time_ms + bin_ms + 1e-9, bin_ms)
|
||||
counts, _ = np.histogram(times, bins=edges)
|
||||
return {"simulation_time_ms": self.simulation_time_ms, "population_id": population_id, "bin_ms": bin_ms, "bin_start_ms": edges[:-1], "spike_counts": counts.astype(np.int32)}
|
||||
|
||||
def get_state_summary(self) -> dict[str, Any]:
|
||||
return {
|
||||
"api_version": API_VERSION, "runtime": self.cfg["names"]["runtime"],
|
||||
"simulation_step": self.simulation_step, "simulation_time_ms": self.simulation_time_ms,
|
||||
"sequence_id": self.sequence_id, "seed": self.seed,
|
||||
"neurons": self.n, "edges": int(self.indices.numel()),
|
||||
"active_neurons_total": int(torch.count_nonzero(self.spike_counts).item()),
|
||||
"spikes_total": int(self.spike_counts.sum().item()),
|
||||
"external_drive_events_total": sum(len(x) for x in self.drive_nodes),
|
||||
}
|
||||
|
||||
def checkpoint_state(self) -> dict[str, Any]:
|
||||
return {
|
||||
"api_version": API_VERSION, "seed": self.seed, "simulation_step": self.simulation_step,
|
||||
"sequence_id": self.sequence_id, "generator_state": self.generator.get_state().cpu(),
|
||||
"v": self.v.clone(), "g": self.g.clone(), "refractory": self.refractory.clone(),
|
||||
"delay_queue": self.delay_queue.clone(), "spike_counts": self.spike_counts.clone(),
|
||||
"spike_steps": copy.deepcopy(self.spike_steps), "spike_nodes": copy.deepcopy(self.spike_nodes),
|
||||
"spike_origins": copy.deepcopy(self.spike_origins), "drive_steps": copy.deepcopy(self.drive_steps),
|
||||
"drive_nodes": copy.deepcopy(self.drive_nodes), "drive_values": copy.deepcopy(self.drive_values),
|
||||
"input_audit": copy.deepcopy(self.input_audit),
|
||||
}
|
||||
|
||||
def restore_state(self, state: dict[str, Any]) -> dict[str, Any]:
|
||||
if state.get("api_version") != API_VERSION:
|
||||
raise ValueError("checkpoint API version mismatch")
|
||||
self.seed = int(state["seed"]); self.simulation_step = int(state["simulation_step"]); self.sequence_id = int(state["sequence_id"])
|
||||
self.generator.set_state(state["generator_state"].cpu())
|
||||
for name in ("v", "g", "refractory", "delay_queue", "spike_counts"):
|
||||
setattr(self, name, state[name].clone().to(self.device))
|
||||
for name in ("spike_steps", "spike_nodes", "spike_origins", "drive_steps", "drive_nodes", "drive_values", "input_audit"):
|
||||
setattr(self, name, copy.deepcopy(state[name]))
|
||||
return self.get_state_summary()
|
||||
|
||||
def activity_packet(self, window_ms: float, input_metadata: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
ev = self.read_spikes(window_ms)
|
||||
aggregates = {name: {k: v for k, v in self.read_population(name, window_ms).items() if k in ("active_neurons", "total_spikes", "aggregate_rate_hz")} for name in self.output_populations}
|
||||
return {
|
||||
"contract_version": ACTIVITY_STREAM_VERSION, "simulation_time_ms": self.simulation_time_ms,
|
||||
"sequence_id": self.sequence_id, "window_ms": window_ms,
|
||||
"spike_neuron_idx": ev["neuron_idx"].tolist(), "spike_timestamp_ms": ev["timestamp_ms"].tolist(),
|
||||
"population_aggregates": aggregates, "active_neuron_count": int(len(np.unique(ev["neuron_idx"]))),
|
||||
"input_channel_metadata": input_metadata or {}, "representation": "event-only",
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
{
|
||||
"file_count": 11,
|
||||
"files": [
|
||||
{
|
||||
"bytes": 353,
|
||||
"path": "VERSION.json",
|
||||
"sha256": "6d5cf119bde4963fe78b1407ff1452e42d0cf70168661d60cc52bdf5a61b8667"
|
||||
},
|
||||
{
|
||||
"bytes": 13925,
|
||||
"path": "phase4n.py",
|
||||
"sha256": "dbbb49447aecbc862d3f54bba9de41f6b8ab90a6afb1ef43a686149fce832d13"
|
||||
},
|
||||
{
|
||||
"bytes": 1712,
|
||||
"path": "phase4n.yaml",
|
||||
"sha256": "6e1b85fc923c7c969df0a7002557256cb719106f8a028c2c506c5ccfe0edd2b2"
|
||||
},
|
||||
{
|
||||
"bytes": 6158,
|
||||
"path": "phase4n_alldan.py",
|
||||
"sha256": "df12477905d7da70e0e337a8d4408cc247135e7876ac7b197e1997f2c4611a54"
|
||||
},
|
||||
{
|
||||
"bytes": 3237,
|
||||
"path": "phase4n_control_whole.py",
|
||||
"sha256": "560ebffba3e8c3457827a0a682476d125a2bbef81958f5b621d32c1203aada1e"
|
||||
},
|
||||
{
|
||||
"bytes": 7198,
|
||||
"path": "phase4n_controls.py",
|
||||
"sha256": "229e48aa63a725e65202dfbabec58d9a092822482c6d0b461ab5a72ee323a901"
|
||||
},
|
||||
{
|
||||
"bytes": 17592,
|
||||
"path": "phase4n_events.py",
|
||||
"sha256": "1fd10326d9e9e4dd6e78b71a6a7c784058c51119a3a5523d4fe50550cd23bbda"
|
||||
},
|
||||
{
|
||||
"bytes": 20496,
|
||||
"path": "phase4n_finalize.py",
|
||||
"sha256": "2ed9b2a7282f0319238646d2c9be0533650a0146292c2de8201af27bb9cb2e5c"
|
||||
},
|
||||
{
|
||||
"bytes": 16993,
|
||||
"path": "phase4n_finish.py",
|
||||
"sha256": "690d33c18d6438d11050ba1fdee9a8015c6bc25b7c85f78cb56f4d06820e35a7"
|
||||
},
|
||||
{
|
||||
"bytes": 6839,
|
||||
"path": "phase4n_paths.py",
|
||||
"sha256": "41554ae9b712b1710a3e8559e86e01e2aa5d95ca0f88fcaf93a4a5baa8dc3877"
|
||||
},
|
||||
{
|
||||
"bytes": 757,
|
||||
"path": "phase4n_preflight.py",
|
||||
"sha256": "aa339208407f4dd00282201ce362ea06fa042f3115e57483670196c6b3c3d642"
|
||||
}
|
||||
],
|
||||
"generated_utc": "2026-09-21T10:57:20.327961+00:00",
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/credit-assignment-v1",
|
||||
"total_bytes": 95260
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"candidate_generated": false,
|
||||
"config_sha256": "6e1b85fc923c7c969df0a7002557256cb719106f8a028c2c506c5ccfe0edd2b2",
|
||||
"frozen_dynamic_regression": false,
|
||||
"frozen_static_integrity": true,
|
||||
"identity_sha256": "aa63eacdfe4f63751a3bdf1efe3071a69d07babcd64b479aa91c9f3b512cd45d",
|
||||
"phase": "4N",
|
||||
"status": "PARTIAL-MECHANISM-AUDIT-NO-CANDIDATE"
|
||||
}
|
||||
@@ -0,0 +1,12 @@
|
||||
8782baf95df0bc5d4a17e1ecca99fd7f3af55a25a7619b8ae5028d22c70b0417 MANIFEST.json
|
||||
6d5cf119bde4963fe78b1407ff1452e42d0cf70168661d60cc52bdf5a61b8667 VERSION.json
|
||||
dbbb49447aecbc862d3f54bba9de41f6b8ab90a6afb1ef43a686149fce832d13 phase4n.py
|
||||
6e1b85fc923c7c969df0a7002557256cb719106f8a028c2c506c5ccfe0edd2b2 phase4n.yaml
|
||||
df12477905d7da70e0e337a8d4408cc247135e7876ac7b197e1997f2c4611a54 phase4n_alldan.py
|
||||
560ebffba3e8c3457827a0a682476d125a2bbef81958f5b621d32c1203aada1e phase4n_control_whole.py
|
||||
229e48aa63a725e65202dfbabec58d9a092822482c6d0b461ab5a72ee323a901 phase4n_controls.py
|
||||
1fd10326d9e9e4dd6e78b71a6a7c784058c51119a3a5523d4fe50550cd23bbda phase4n_events.py
|
||||
2ed9b2a7282f0319238646d2c9be0533650a0146292c2de8201af27bb9cb2e5c phase4n_finalize.py
|
||||
690d33c18d6438d11050ba1fdee9a8015c6bc25b7c85f78cb56f4d06820e35a7 phase4n_finish.py
|
||||
41554ae9b712b1710a3e8559e86e01e2aa5d95ca0f88fcaf93a4a5baa8dc3877 phase4n_paths.py
|
||||
aa339208407f4dd00282201ce362ea06fa042f3115e57483670196c6b3c3d642 phase4n_preflight.py
|
||||
@@ -0,0 +1,218 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4N read-only reconstruction of frozen Phase 4M acquisition and replay."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
|
||||
from phase4h_audit import sha256, verify_checksum, write_json
|
||||
from phase4i import Validator
|
||||
from phase4k import seal
|
||||
from phase4m import ReplayEngine, compare_A_B, frozen_checks, frozen_hashes, load_odor_inputs, replay_effect
|
||||
from phase4l import OdorSimulator, pn_drive
|
||||
from phase4j import effect
|
||||
|
||||
|
||||
GROUPS = ("A-specific", "shared", "B-specific", "inactive")
|
||||
|
||||
|
||||
def load_config(path: Path):
|
||||
ncfg = yaml.safe_load(path.read_text())
|
||||
mpath = Path(ncfg["frozen_phase4m_config"])
|
||||
mcfg = yaml.safe_load(mpath.read_text())
|
||||
lpath = Path(mcfg["frozen_phase4l_config"])
|
||||
cfg = yaml.safe_load(lpath.read_text())
|
||||
cfg["paths"]["phase4l_root"] = cfg["paths"]["experiment_root"]
|
||||
cfg["paths"]["phase4l_model_root"] = str(lpath.parent)
|
||||
return ncfg, mcfg, cfg, lpath
|
||||
|
||||
|
||||
def prepare(ncfg, mcfg, cfg, exp: Path):
|
||||
if exp.exists() and (exp / "edge-identity/definition.json").exists():
|
||||
raise RuntimeError("Phase 4N identity/definition already frozen; refusing to overwrite")
|
||||
previous = frozen_checks(cfg)
|
||||
mroot = Path(mcfg["experiment_root"])
|
||||
mmodel = Path(mcfg["model_root"])
|
||||
for root, rel in ((mroot, "manifests/checksums.sha256"), (mmodel, "checksums.sha256")):
|
||||
check = verify_checksum(root, root / rel)
|
||||
if not check["passed"]:
|
||||
raise RuntimeError(f"Phase 4M frozen checksum failed: {root}")
|
||||
for sub in ("edge-identity", "acquisition-trajectory", "specificity-curve", "dan-source", "endogenous-dan", "local-eligibility", "shared-edge", "b-contamination", "isolated", "whole-brain", "controls", "stopping-analysis", "performance", "regression", "audit", "manifests"):
|
||||
(exp / sub).mkdir(parents=True, exist_ok=True)
|
||||
src = Path(cfg["paths"]["phase4l_root"]) / "eligible-edges/edge-identity.parquet"
|
||||
identity = pd.read_parquet(src)
|
||||
required = {"edge_id", "source_neuron_idx", "source_body_id", "target_neuron_idx", "target_body_id", "raw_synapse_count", "target_compartment_verbatim", "odor_eligibility_category"}
|
||||
if not required.issubset(identity.columns) or len(identity) != 14232:
|
||||
raise RuntimeError("Phase 4L identity schema/size mismatch")
|
||||
if set(identity.odor_eligibility_category.astype(str)) != set(GROUPS):
|
||||
raise RuntimeError("identity groups differ from four frozen groups")
|
||||
identity["anatomical_weight_mv"] = identity.raw_synapse_count.astype(np.float32) * np.float32(cfg["model"]["w_syn_mv_per_anatomical_synapse"])
|
||||
identity["initial_E"] = np.float32(1.0)
|
||||
identity.rename(columns={"odor_eligibility_category": "identity_group", "target_compartment_verbatim": "compartment"}, inplace=True)
|
||||
out = exp / "edge-identity/frozen-edge-identity.parquet"
|
||||
identity.to_parquet(out, index=False)
|
||||
definition = {"source": str(src), "source_sha256": sha256(src), "source_semantics": ncfg["edge_identity_source"],
|
||||
"group_counts": identity.identity_group.value_counts().to_dict(), "edge_count": len(identity),
|
||||
"checkpoint_rule": ncfg["checkpoints"], "CSI": ncfg["credit_specificity_index"],
|
||||
"functional_separation": ncfg["functional_separation"], "peak_rule": ncfg["peak_rule"],
|
||||
"generalization_onset_rule": ncfg["generalization_onset_rule"], "frozen_before_learning": True,
|
||||
"phase4m_experiment_checksum_sha256": sha256(mroot / "manifests/checksums.sha256"),
|
||||
"phase4m_model_checksum_sha256": sha256(mmodel / "checksums.sha256"), "prior_checksum_passed": previous["all_passed"]}
|
||||
write_json(exp / "edge-identity/definition.json", definition)
|
||||
shutil.copy2(Path(__file__).resolve(), exp / "audit/phase4n.py")
|
||||
return identity, definition
|
||||
|
||||
|
||||
def reconstruct(ncfg, mcfg, cfg, exp: Path, identity: pd.DataFrame):
|
||||
hist = pd.read_parquet(Path(mcfg["experiment_root"]) / "efficacy-overlay/update-history.parquet")
|
||||
required = {"train_seed", "pairing", "edge_id", "E_bounded_before", "E_bounded_after", "E_multiplicative_before", "E_multiplicative_after", "eligibility"}
|
||||
if not required.issubset(hist.columns):
|
||||
raise RuntimeError("Phase 4M update history schema mismatch")
|
||||
edge_ids = identity.edge_id.to_numpy(np.int64)
|
||||
lookup = pd.Series(np.arange(len(edge_ids), dtype=np.int32), index=edge_ids)
|
||||
checkpoints = {}
|
||||
verify = []
|
||||
points = list(map(int, ncfg["checkpoints"]))
|
||||
all_rows = []
|
||||
for seed in ncfg["train_seeds"]:
|
||||
by = hist[hist.train_seed == seed]
|
||||
E = np.ones(len(edge_ids), np.float32)
|
||||
U = np.ones(len(edge_ids), np.float32)
|
||||
checkpoints[seed] = {0: {"bounded": E.copy(), "unbounded": U.copy()}}
|
||||
np.savez_compressed(exp / "acquisition-trajectory" / f"E-seed-{seed}-pairings-0.npz", E_bounded=E, E_unbounded=U, edge_id=edge_ids)
|
||||
for mechanism in ("bounded", "unbounded"):
|
||||
for group in GROUPS:
|
||||
mask = identity.identity_group.to_numpy() == group
|
||||
all_rows.append({"seed": seed, "pairings": 0, "mechanism": mechanism, "group": group,
|
||||
"total_edges": int(mask.sum()), "updated_edges": 0, "mean_delta_E": 0.0, "median_delta_E": 0.0,
|
||||
"weighted_transmission_reduction": 0.0, "E_change_total": 0.0})
|
||||
for pairing in range(1, max(points) + 1):
|
||||
item = by[by.pairing == pairing]
|
||||
local = lookup.loc[item.edge_id.to_numpy(np.int64)].to_numpy(np.int32)
|
||||
before_error = max(float(np.max(np.abs(E[local] - item.E_bounded_before.to_numpy(np.float32)))) if len(local) else 0.0,
|
||||
float(np.max(np.abs(U[local] - item.E_multiplicative_before.to_numpy(np.float32)))) if len(local) else 0.0)
|
||||
if before_error > 1e-6:
|
||||
raise RuntimeError(f"Phase 4M event history not reconstructible seed={seed} pairing={pairing}: {before_error}")
|
||||
E[local] = item.E_bounded_after.to_numpy(np.float32)
|
||||
U[local] = item.E_multiplicative_after.to_numpy(np.float32)
|
||||
if pairing not in points:
|
||||
continue
|
||||
checkpoints[seed][pairing] = {"bounded": E.copy(), "unbounded": U.copy()}
|
||||
np.savez_compressed(exp / "acquisition-trajectory" / f"E-seed-{seed}-pairings-{pairing}.npz", E_bounded=E, E_unbounded=U, edge_id=edge_ids)
|
||||
old = Path(mcfg["experiment_root"]) / "efficacy-overlay" / f"E-seed-{seed}-pairings-{pairing}.npz"
|
||||
if old.exists():
|
||||
with np.load(old) as f:
|
||||
err = max(float(np.max(np.abs(E - f["E_bounded_multiplicative"]))), float(np.max(np.abs(U - f["E_multiplicative"]))))
|
||||
verify.append({"seed": seed, "pairings": pairing, "max_abs_error": err})
|
||||
if err > 1e-6:
|
||||
raise RuntimeError("reconstructed E differs from sealed Phase 4M checkpoint")
|
||||
for mechanism, value in (("bounded", E), ("unbounded", U)):
|
||||
for group in GROUPS:
|
||||
mask = identity.identity_group.to_numpy() == group
|
||||
w = identity.anatomical_weight_mv.to_numpy(np.float32)[mask]
|
||||
reduction = 1 - value[mask]
|
||||
all_rows.append({"seed": seed, "pairings": pairing, "mechanism": mechanism, "group": group,
|
||||
"total_edges": int(mask.sum()), "updated_edges": int(np.count_nonzero(reduction > 1e-7)),
|
||||
"mean_delta_E": float((-reduction).mean()), "median_delta_E": float(np.median(-reduction)),
|
||||
"weighted_transmission_reduction": float(np.dot(w.astype(np.float64), reduction.astype(np.float64)) / w.sum()),
|
||||
"E_change_total": float(reduction.sum())})
|
||||
table = pd.DataFrame(all_rows)
|
||||
table.to_parquet(exp / "acquisition-trajectory/group-plasticity.parquet", index=False)
|
||||
wide = table.pivot_table(index=["seed", "pairings", "mechanism"], columns="group", values="weighted_transmission_reduction").reset_index()
|
||||
a, b, shared = (wide[x].to_numpy(np.float64) for x in ("A-specific", "B-specific", "shared"))
|
||||
wide["CSI"] = a - np.maximum(shared, b)
|
||||
wide["A_to_shared_ratio"] = np.divide(a, shared, out=np.full_like(a, np.nan), where=shared != 0)
|
||||
wide["A_to_B_ratio"] = np.divide(a, b, out=np.full_like(a, np.nan), where=b != 0)
|
||||
wide.to_parquet(exp / "specificity-curve/CSI.parquet", index=False)
|
||||
write_json(exp / "regression/phase4m-history-reconstruction.json", {"passed": True, "old_checkpoint_comparisons": verify, "max_error": max((v["max_abs_error"] for v in verify), default=0.0)})
|
||||
return checkpoints, table, wide, hist
|
||||
|
||||
|
||||
def isolated(ncfg, mcfg, cfg, exp: Path, identity: pd.DataFrame, checkpoints):
|
||||
v = Validator(cfg, Path(mcfg["frozen_phase4l_config"]), exp)
|
||||
if not np.array_equal(v.edge_ids, identity.edge_id.to_numpy(np.int64)):
|
||||
raise RuntimeError("plastic candidate ordering mismatch")
|
||||
sim = OdorSimulator(v, cfg, exp)
|
||||
comp = identity.compartment.astype(str).to_numpy() == str(ncfg["matched_compartment_verbatim"])
|
||||
targets = np.unique(identity.loc[comp, "target_neuron_idx"].to_numpy(np.int32))
|
||||
replay = ReplayEngine(sim, targets)
|
||||
anatomical = identity.anatomical_weight_mv.to_numpy(np.float32)
|
||||
old = Path(mcfg["experiment_root"])
|
||||
evaldata = {}
|
||||
zero = np.zeros(len(v.edge_ids), np.float32)
|
||||
for seed in ncfg["evaluation_seeds"]:
|
||||
for odor in ("A", "B"):
|
||||
spikes = pd.read_parquet(old / "isolated-replay" / f"KC-spikes-ODOR_{odor}-seed-{seed}.parquet")
|
||||
evaldata[(seed, odor)] = (spikes, replay.run(spikes, zero, float(mcfg["protocol"]["duration_ms"])))
|
||||
rows = []
|
||||
replay_wall = []
|
||||
for train_seed, eval_seed in zip(ncfg["train_seeds"], ncfg["evaluation_seeds"]):
|
||||
for pair in ncfg["checkpoints"]:
|
||||
E = checkpoints[train_seed][pair]["bounded"]
|
||||
delta = anatomical * (E - 1)
|
||||
rec = {"train_seed": train_seed, "eval_seed": eval_seed, "pairings": pair}
|
||||
effects = {}
|
||||
for odor in ("A", "B"):
|
||||
spikes, baseline = evaldata[(eval_seed, odor)]
|
||||
post = replay.run(spikes, delta, float(mcfg["protocol"]["duration_ms"]))
|
||||
replay_wall.append(post["wall_seconds"])
|
||||
effects[odor] = replay_effect(baseline, post, sim.dt)
|
||||
rec.update({f"{odor}_{k}": value for k, value in effects[odor].items() if k != "isolated_input_scope"})
|
||||
rec["A_specific_functional"] = compare_A_B(effects["A"], effects["B"])
|
||||
rec["functional_separation_spike_L1"] = float(effects["A"]["MBON_spike_L1"] - effects["B"]["MBON_spike_L1"])
|
||||
al = effects["A"]["first_latency_delta_ms"] or 0.0
|
||||
bl = effects["B"]["first_latency_delta_ms"] or 0.0
|
||||
rec["functional_separation_latency_ms"] = float(al - bl)
|
||||
rows.append(rec)
|
||||
frame = pd.DataFrame(rows)
|
||||
frame.to_parquet(exp / "isolated/response-by-checkpoint.parquet", index=False)
|
||||
curve = frame.groupby("pairings").agg(A_specific_functional_seeds=("A_specific_functional", "sum"),
|
||||
mean_spike_L1_separation=("functional_separation_spike_L1", "mean"),
|
||||
mean_latency_separation_ms=("functional_separation_latency_ms", "mean"),
|
||||
A_MBON_spike_L1=("A_MBON_spike_L1", "mean"), B_MBON_spike_L1=("B_MBON_spike_L1", "mean"),
|
||||
A_transmission_reduction_percent=("A_transmitted_reduction_percent", "mean"),
|
||||
B_transmission_reduction_percent=("B_transmitted_reduction_percent", "mean")).reset_index()
|
||||
curve.to_parquet(exp / "specificity-curve/memory-specificity.parquet", index=False)
|
||||
valid = curve[curve.A_specific_functional_seeds >= 3].copy()
|
||||
if valid.empty:
|
||||
peak = None
|
||||
else:
|
||||
best_spikes = float(valid.mean_spike_L1_separation.max())
|
||||
valid = valid[np.isclose(valid.mean_spike_L1_separation, best_spikes, atol=1e-9, rtol=0)]
|
||||
best_latency = float(valid.mean_latency_separation_ms.max())
|
||||
valid = valid[valid.mean_latency_separation_ms >= best_latency - sim.dt / 2]
|
||||
peak = int(valid.pairings.min())
|
||||
if peak is not None:
|
||||
# Equal score/latency checkpoints are resolved to earliest by the predeclared rule.
|
||||
peak_csi = float(np.mean(pd.read_parquet(exp / "specificity-curve/CSI.parquet").query("mechanism == 'bounded' and pairings == @peak").CSI))
|
||||
else:
|
||||
peak_csi = None
|
||||
write_json(exp / "specificity-curve/peak.json", {"status": "SPECIFICITY-PEAK-IDENTIFIED" if peak is not None else "UNIDENTIFIED", "pairings": peak,
|
||||
"peak_CSI": peak_csi, "selection_rule": ncfg["peak_rule"]})
|
||||
write_json(exp / "performance/isolated.json", {"mean_replay_wall_seconds": float(np.mean(replay_wall)), "total_replay_wall_seconds": float(np.sum(replay_wall)), "count": len(replay_wall)})
|
||||
return v, sim, frame, curve, peak
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--config", required=True, type=Path)
|
||||
args = ap.parse_args()
|
||||
ncfg, mcfg, cfg, base_path = load_config(args.config)
|
||||
exp = Path(ncfg["experiment_root"])
|
||||
identity, definition = prepare(ncfg, mcfg, cfg, exp)
|
||||
print(json.dumps({"stage": "identity_frozen", "group_counts": definition["group_counts"], "identity_sha256": sha256(exp / "edge-identity/frozen-edge-identity.parquet")}), flush=True)
|
||||
checkpoints, group, csi, history = reconstruct(ncfg, mcfg, cfg, exp, identity)
|
||||
print(json.dumps({"stage": "trajectory_reconstructed", "checkpoint_count": len(ncfg["checkpoints"]), "updates": len(history)}), flush=True)
|
||||
v, sim, frame, curve, peak = isolated(ncfg, mcfg, cfg, exp, identity, checkpoints)
|
||||
print(json.dumps({"stage": "isolated_complete", "peak": peak, "curve": curve.to_dict("records")}), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,25 @@
|
||||
phase: 4N
|
||||
name: MaleCNS-CreditAssignment-audit
|
||||
frozen_phase4m_config: /data/digital-drosophila/models/male-cns-lif/kcmbon-efficacy-v1/phase4m.yaml
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4n-credit-assignment
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/credit-assignment-v1
|
||||
mechanism: bounded_multiplicative
|
||||
checkpoints: [0, 1, 2, 5, 10, 20, 30, 40, 50, 60, 75, 100]
|
||||
train_seeds: [51201, 51202, 51203, 51204, 51205]
|
||||
evaluation_seeds: [51301, 51302, 51303, 51304, 51305]
|
||||
edge_identity_source: phase4l matched_energy held-out ODOR_A/B majority-of-five KC representation
|
||||
credit_specificity_index: group mean anatomical-weighted E depression for A-specific minus maximum of shared and B-specific groups
|
||||
functional_separation: A MBON spike-count L1 change minus B MBON spike-count L1 change; first-spike latency used only as tie-breaker
|
||||
peak_rule: earliest checkpoint with at least 3 of 5 A-specific functional seeds and maximum five-seed mean spike-L1 separation; spike-L1 ties within 1e-9 use mean first-latency separation only if difference exceeds half a dt step (0.05 ms), otherwise choose earliest; no peak if no such checkpoint
|
||||
generalization_onset_rule: earliest checkpoint after peak with CSI at least 20 percent below peak and shared-or-B group depression at least 10 percent above its peak value; otherwise unidentified
|
||||
random_us_seed_offset: 700000
|
||||
random_us_window_ms: [30.0, 110.0]
|
||||
shadow_trace_tau_ms: 20.0
|
||||
event_coincidence_window_ms: 1.0
|
||||
first_activation_only: true
|
||||
matched_compartment_verbatim: "y5B'2a"
|
||||
unrelated_compartment_verbatim: "y2a'1"
|
||||
matched_dan_type: PAM15
|
||||
unrelated_dan_type: PPL103
|
||||
no_frozen_artifact_writes: true
|
||||
no_formal_associative_memory_claim: true
|
||||
@@ -0,0 +1,87 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Compare external-only E with open-loop all-confirmed-DAN gate shadow."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from phase4h_audit import write_json
|
||||
from phase4i import Validator
|
||||
from phase4j import effect
|
||||
from phase4l import OdorSimulator, pn_drive
|
||||
from phase4m import ReplayEngine, compare_A_B, load_odor_inputs, replay_effect
|
||||
from phase4n import load_config
|
||||
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument("--config",type=Path,required=True);args=ap.parse_args()
|
||||
ncfg,mcfg,cfg,basepath=load_config(args.config);exp=Path(ncfg["experiment_root"])
|
||||
identity=pd.read_parquet(exp/"edge-identity/frozen-edge-identity.parquet")
|
||||
reproduction=json.loads((exp/"regression/event-trace-reproduction.json").read_text())
|
||||
bad={int(seed):min(int(x["pairing"]) for x in reproduction["mismatched_trials"] if int(x["seed"])==int(seed))
|
||||
for seed in ncfg["train_seeds"] if any(int(x["seed"])==int(seed) for x in reproduction["mismatched_trials"])}
|
||||
v=Validator(cfg,basepath,exp);sim=OdorSimulator(v,cfg,exp)
|
||||
comp=identity.compartment.astype(str).to_numpy()==str(ncfg["matched_compartment_verbatim"])
|
||||
engine=ReplayEngine(sim,np.unique(identity.loc[comp,"target_neuron_idx"].to_numpy(np.int32)))
|
||||
w=identity.anatomical_weight_mv.to_numpy(np.float32);old=Path(mcfg["experiment_root"])
|
||||
rows=[];calls=[]
|
||||
for seed,es in zip(ncfg["train_seeds"],ncfg["evaluation_seeds"]):
|
||||
for pair in [20,50,60,75,100]:
|
||||
with np.load(exp/"endogenous-dan"/f"E-all-DAN-shadow-seed-{seed}-pairings-{pair}.npz") as f:
|
||||
states={"external-only":f["E_external_only"].copy(),"all-confirmed-DAN-shadow":f["E_all_DAN"].copy()}
|
||||
for mechanism,E in states.items():
|
||||
delta=w*(E-1);results={}
|
||||
for odor in ("A","B"):
|
||||
spikes=pd.read_parquet(old/"isolated-replay"/f"KC-spikes-ODOR_{odor}-seed-{es}.parquet")
|
||||
zero=engine.run(spikes,np.zeros(len(E),np.float32),float(mcfg["protocol"]["duration_ms"]))
|
||||
result=engine.run(spikes,delta,float(mcfg["protocol"]["duration_ms"]))
|
||||
calls.append(result["wall_seconds"]);results[odor]=replay_effect(zero,result,sim.dt)
|
||||
rows.append({"seed":seed,"eval_seed":es,"pairings":pair,"mechanism":mechanism,
|
||||
"trace_exact_prefix":bool(pair < bad.get(int(seed),10**9)),
|
||||
"A_specific_functional":compare_A_B(results["A"],results["B"]),
|
||||
"A_MBON_spike_L1":results["A"]["MBON_spike_L1"],"B_MBON_spike_L1":results["B"]["MBON_spike_L1"],
|
||||
"A_transmission_reduction_percent":results["A"]["transmitted_reduction_percent"],
|
||||
"B_transmission_reduction_percent":results["B"]["transmitted_reduction_percent"]})
|
||||
table=pd.DataFrame(rows);table.to_parquet(exp/"endogenous-dan/isolated-gate-comparison.parquet",index=False)
|
||||
panel,vectors,selection,mapping=load_odor_inputs(cfg);gloms=selection["glomeruli_order"];ms=float(np.median(panel.mapped_response_sum));pn={}
|
||||
for odor in ("A","B"):
|
||||
oid=selection[f"ODOR_{odor}"]["odor_id"]
|
||||
pn[odor]=pn_drive(oid,vectors[oid],mapping,gloms,str(mcfg["protocol"]["input_mode"]),ms,cfg)[:2]
|
||||
whole=[];brain_calls=[]
|
||||
for seed,es in zip(ncfg["train_seeds"],ncfg["evaluation_seeds"]):
|
||||
baseline={odor:sim.run_rates(*pn[odor],es,float(mcfg["protocol"]["duration_ms"])) for odor in pn}
|
||||
for pair in (60,100):
|
||||
with np.load(exp/"endogenous-dan"/f"E-all-DAN-shadow-seed-{seed}-pairings-{pair}.npz") as f:
|
||||
states={"external-only":f["E_external_only"].copy(),"all-confirmed-DAN-shadow":f["E_all_DAN"].copy()}
|
||||
for mechanism,E in states.items():
|
||||
delta=w*(E-1)
|
||||
for odor in ("A","B"):
|
||||
post=sim.run_rates(*pn[odor],es,float(mcfg["protocol"]["duration_ms"]),delta=delta)
|
||||
brain_calls.append(post["wall_seconds"]);e=effect(baseline[odor],post)
|
||||
whole.append({"seed":seed,"eval_seed":es,"pairings":pair,"mechanism":mechanism,"odor":odor,
|
||||
"trace_exact_prefix":bool(pair < bad.get(int(seed),10**9)),
|
||||
"MBON_spike_L1":e["MBON_spike_l1"],"downstream_spike_L1":e["downstream_spike_l1"],
|
||||
"KC_spike_L1":int(np.abs(post["kc_counts"]-baseline[odor]["kc_counts"]).sum()),
|
||||
"finite":bool(post["finite"]),"wall_seconds":post["wall_seconds"]})
|
||||
wt=pd.DataFrame(whole);wt.to_parquet(exp/"endogenous-dan/wholebrain-gate-comparison.parquet",index=False)
|
||||
iso=table.groupby(["mechanism","pairings"]).agg(A_specific_seeds=("A_specific_functional","sum"),A_spike_L1=("A_MBON_spike_L1","mean"),B_spike_L1=("B_MBON_spike_L1","mean")).reset_index()
|
||||
pivot=wt.pivot(index=["mechanism","pairings","seed"],columns="odor",values="MBON_spike_L1").reset_index()
|
||||
pivot["A_minus_B"]=pivot["A"]-pivot["B"]
|
||||
wh=pivot.groupby(["mechanism","pairings"]).agg(A_minus_B_mean=("A_minus_B","mean"),A_specific_seeds=("A_minus_B",lambda x:int((x>0).sum()))).reset_index()
|
||||
clean=table[table.trace_exact_prefix].groupby(["mechanism","pairings"]).agg(clean_seed_count=("seed","nunique"),A_specific_seeds=("A_specific_functional","sum"),
|
||||
A_spike_L1=("A_MBON_spike_L1","mean"),B_spike_L1=("B_MBON_spike_L1","mean")).reset_index()
|
||||
comp_summary={"isolated":iso.to_dict("records"),"wholebrain":wh.to_dict("records"),
|
||||
"isolated_exact_prefix_only":clean.to_dict("records"),"nonexact_trials":reproduction["mismatched_trials"],
|
||||
"all_DAN_branch_status":"counterfactual open-loop shadow gate, not frozen Phase 4M dynamics or physiological dopamine",
|
||||
"isolated_replay_wall_seconds":float(sum(calls)),"wholebrain_wall_seconds":float(sum(brain_calls)),
|
||||
"external_US_only_is_frozen_Phase4M_rule":True}
|
||||
write_json(exp/"endogenous-dan/gate-specificity.json",comp_summary)
|
||||
print(json.dumps(comp_summary),flush=True)
|
||||
|
||||
|
||||
if __name__=="__main__":main()
|
||||
@@ -0,0 +1,56 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Held-out whole-brain readout for independent Phase 4N control E overlays."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from phase4h_audit import write_json
|
||||
from phase4i import Validator
|
||||
from phase4j import effect
|
||||
from phase4l import OdorSimulator, pn_drive
|
||||
from phase4m import load_odor_inputs
|
||||
from phase4n import load_config
|
||||
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument("--config",type=Path,required=True);args=ap.parse_args()
|
||||
ncfg,mcfg,cfg,basepath=load_config(args.config);exp=Path(ncfg["experiment_root"])
|
||||
identity=pd.read_parquet(exp/"edge-identity/frozen-edge-identity.parquet")
|
||||
v=Validator(cfg,basepath,exp);sim=OdorSimulator(v,cfg,exp)
|
||||
w=identity.anatomical_weight_mv.to_numpy(np.float32)
|
||||
panel,vectors,selection,mapping=load_odor_inputs(cfg);gloms=selection["glomeruli_order"];ms=float(np.median(panel.mapped_response_sum))
|
||||
pn={}
|
||||
for odor in ("A","B"):
|
||||
oid=selection[f"ODOR_{odor}"]["odor_id"]
|
||||
pn[odor]=pn_drive(oid,vectors[oid],mapping,gloms,str(mcfg["protocol"]["input_mode"]),ms,cfg)[:2]
|
||||
rows=[];walls=[];conditions=("B+matched-US","A+random-US-timing","A+unrelated-compartment-US")
|
||||
for seed,es in zip(ncfg["train_seeds"],ncfg["evaluation_seeds"]):
|
||||
baseline={odor:sim.run_rates(*pn[odor],es,float(mcfg["protocol"]["duration_ms"])) for odor in pn}
|
||||
for pair in (50,100):
|
||||
for condition in conditions:
|
||||
with np.load(exp/"controls"/f"E-{condition}-seed-{seed}-pairings-{pair}.npz") as f:E=f["E"].copy()
|
||||
delta=w*(E-1)
|
||||
for odor in ("A","B"):
|
||||
post=sim.run_rates(*pn[odor],es,float(mcfg["protocol"]["duration_ms"]),delta=delta)
|
||||
e=effect(baseline[odor],post);walls.append(post["wall_seconds"])
|
||||
rows.append({"condition":condition,"train_seed":seed,"eval_seed":es,"pairings":pair,"odor":odor,
|
||||
"MBON_spike_L1":e["MBON_spike_l1"],"downstream_spike_L1":e["downstream_spike_l1"],
|
||||
"KC_spike_L1":int(np.abs(post["kc_counts"]-baseline[odor]["kc_counts"]).sum()),
|
||||
"finite":bool(post["finite"]),"wall_seconds":post["wall_seconds"]})
|
||||
print(json.dumps({"seed":seed,"completed_control_wholebrain":True}),flush=True)
|
||||
table=pd.DataFrame(rows);table.to_parquet(exp/"controls/wholebrain-heldout-control-replay.parquet",index=False)
|
||||
pivot=table.pivot(index=["condition","pairings","train_seed"],columns="odor",values="MBON_spike_L1").reset_index()
|
||||
pivot["A_minus_B"]=pivot["A"]-pivot["B"]
|
||||
summary=pivot.groupby(["condition","pairings"]).agg(A_minus_B_mean=("A_minus_B","mean"),A_specific_seeds=("A_minus_B",lambda x:int((x>0).sum()))).reset_index()
|
||||
write_json(exp/"controls/wholebrain-summary.json",{"rows":len(table),"conditions":summary.to_dict("records"),
|
||||
"total_brain_wall_seconds":float(sum(walls)),"frozen_E1_reference_reused_per_evaluation_seed":True,
|
||||
"A_D0_and_frozen_E1_effect_are_exactly_zero_by_E_identity":True})
|
||||
print(json.dumps(summary.to_dict("records")),flush=True)
|
||||
|
||||
|
||||
if __name__=="__main__":main()
|
||||
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4N independent control trajectories; no frozen artifact writes."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import resource
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from phase4h_audit import write_json
|
||||
from phase4i import Validator
|
||||
from phase4j import apply_v2, effect
|
||||
from phase4l import OdorSimulator, pn_drive
|
||||
from phase4m import ReplayEngine, compare_A_B, load_odor_inputs, replay_effect
|
||||
from phase4n import load_config
|
||||
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument("--config",type=Path,required=True);ap.add_argument("--max-pairings",type=int,default=100)
|
||||
args=ap.parse_args();ncfg,mcfg,cfg,basepath=load_config(args.config);exp=Path(ncfg["experiment_root"])
|
||||
identity=pd.read_parquet(exp/"edge-identity/frozen-edge-identity.parquet")
|
||||
v=Validator(cfg,basepath,exp);sim=OdorSimulator(v,cfg,exp)
|
||||
panel,vectors,selection,mapping=load_odor_inputs(cfg);gloms=selection["glomeruli_order"];ms=float(np.median(panel.mapped_response_sum))
|
||||
pn={}
|
||||
for label in ("A","B"):
|
||||
oid=selection[f"ODOR_{label}"]["odor_id"]
|
||||
pn[label]=pn_drive(oid,vectors[oid],mapping,gloms,str(mcfg["protocol"]["input_mode"]),ms,cfg)[:2]
|
||||
matched=v.dan_targets(str(ncfg["matched_dan_type"]),str(ncfg["matched_compartment_verbatim"]))
|
||||
unrelated=v.dan_targets(str(ncfg["unrelated_dan_type"]),str(ncfg["unrelated_compartment_verbatim"]))
|
||||
src=identity.source_neuron_idx.to_numpy(np.int32);loc=np.full(v.n,-1,np.int32);loc[v.kc]=np.arange(len(v.kc),dtype=np.int32);loc=loc[src]
|
||||
k=np.float32(cfg["eligibility_v2"]["eta_effective_mv"]/cfg["model"]["w_syn_mv_per_anatomical_synapse"])
|
||||
bound=np.float32(1-cfg["eligibility_v2"]["max_fractional_depression"])
|
||||
points=set(map(int,ncfg["checkpoints"]));rows=[];walls=[]
|
||||
conditions={"B+matched-US":("B",matched,str(ncfg["matched_compartment_verbatim"])),
|
||||
"A+random-US-timing":("A",matched,str(ncfg["matched_compartment_verbatim"])),
|
||||
"A+unrelated-compartment-US":("A",unrelated,str(ncfg["unrelated_compartment_verbatim"]))}
|
||||
for condition,(odor,dan,compartment) in conditions.items():
|
||||
group_mask=identity.compartment.astype(str).to_numpy()==compartment
|
||||
for seed in ncfg["train_seeds"]:
|
||||
add=np.zeros(len(v.edge_ids),np.float32);E=np.ones(len(v.edge_ids),np.float32)
|
||||
rng=np.random.default_rng(seed+int(ncfg["random_us_seed_offset"]))
|
||||
np.savez_compressed(exp/"controls"/f"E-{condition}-seed-{seed}-pairings-0.npz",E=E,edge_id=v.edge_ids)
|
||||
rows.append({"condition":condition,"seed":seed,"pairings":0,"odor":odor,"DAN_compartment":compartment,
|
||||
"US_time_ms":None,"updated_edges":0,"matched_compartment_E_reduction_sum":0.0,
|
||||
"all_mask_E_reduction_sum":0.0,"brain_wall_seconds":0.0})
|
||||
for pairing in range(1,args.max_pairings+1):
|
||||
pulse=float(rng.uniform(*ncfg["random_us_window_ms"])) if condition=="A+random-US-timing" else float(mcfg["protocol"]["dan_time_ms"])
|
||||
run=sim.run_rates(*pn[odor],seed+pairing,float(mcfg["protocol"]["duration_ms"]),delta=add,dan_targets=dan,dan_pulses=[pulse])
|
||||
walls.append(run["wall_seconds"]);trace=run["kc_trace_at_dan"];elig=trace[loc]
|
||||
active=group_mask&(elig>0)
|
||||
E[active]=np.maximum(bound,E[active]*(np.float32(1.0)-k*elig[active]))
|
||||
add,_=apply_v2(v,trace,compartment,1.0,add,cfg)
|
||||
if pairing in points:
|
||||
out=exp/"controls"/f"E-{condition}-seed-{seed}-pairings-{pairing}.npz"
|
||||
np.savez_compressed(out,E=E,edge_id=v.edge_ids)
|
||||
rows.append({"condition":condition,"seed":seed,"pairings":pairing,"odor":odor,"DAN_compartment":compartment,
|
||||
"US_time_ms":pulse,"updated_edges":int(np.count_nonzero(E<1-1e-7)),"matched_compartment_E_reduction_sum":float(np.sum((1-E)[identity.compartment.astype(str).to_numpy()==str(ncfg["matched_compartment_verbatim"])])),
|
||||
"all_mask_E_reduction_sum":float(np.sum(1-E)),"brain_wall_seconds":run["wall_seconds"]})
|
||||
if pairing % 20 == 0:
|
||||
print(json.dumps({"condition":condition,"seed":seed,"pairing":pairing,"E_reduction_sum":float(np.sum(1-E))}),flush=True)
|
||||
print(json.dumps({"condition":condition,"seed":seed,"pairings":args.max_pairings,"E_reduction_sum":float(np.sum(1-E))}),flush=True)
|
||||
frame=pd.DataFrame(rows);frame.to_parquet(exp/"controls/independent-training-trajectories.parquet",index=False)
|
||||
# Common frozen held-out A/B KC spike trains; changes below are diagnostic, not memory claims.
|
||||
target_comp=identity.compartment.astype(str).to_numpy()==str(ncfg["matched_compartment_verbatim"])
|
||||
engine=ReplayEngine(sim,np.unique(identity.loc[target_comp,"target_neuron_idx"].to_numpy(np.int32)))
|
||||
w=identity.anatomical_weight_mv.to_numpy(np.float32);replay_rows=[]
|
||||
for condition in conditions:
|
||||
for seed,es in zip(ncfg["train_seeds"],ncfg["evaluation_seeds"]):
|
||||
for pair in [20,50,60,75,100]:
|
||||
with np.load(exp/"controls"/f"E-{condition}-seed-{seed}-pairings-{pair}.npz") as f:E=f["E"].copy()
|
||||
delta=w*(E-1);effects={}
|
||||
for odor in ("A","B"):
|
||||
spikes=pd.read_parquet(Path(mcfg["experiment_root"])/"isolated-replay"/f"KC-spikes-ODOR_{odor}-seed-{es}.parquet")
|
||||
base=engine.run(spikes,np.zeros(len(E),np.float32),float(mcfg["protocol"]["duration_ms"]))
|
||||
post=engine.run(spikes,delta,float(mcfg["protocol"]["duration_ms"]))
|
||||
effects[odor]=replay_effect(base,post,sim.dt)
|
||||
replay_rows.append({"condition":condition,"train_seed":seed,"eval_seed":es,"pairings":pair,
|
||||
"A_specific_functional":compare_A_B(effects["A"],effects["B"]),
|
||||
"A_MBON_spike_L1":effects["A"]["MBON_spike_L1"],"B_MBON_spike_L1":effects["B"]["MBON_spike_L1"],
|
||||
"A_transmission_reduction_percent":effects["A"]["transmitted_reduction_percent"],
|
||||
"B_transmission_reduction_percent":effects["B"]["transmitted_reduction_percent"]})
|
||||
rframe=pd.DataFrame(replay_rows);rframe.to_parquet(exp/"controls/isolated-heldout-control-replay.parquet",index=False)
|
||||
write_json(exp/"controls/independent-summary.json",{"conditions":list(conditions),"seeds":list(ncfg["train_seeds"]),
|
||||
"pairings_per_seed":args.max_pairings,"training_calls":len(walls),"training_wall_seconds":float(sum(walls)),
|
||||
"mean_call_seconds":float(np.mean(walls)),"peak_RAM_KiB":int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss),
|
||||
"matched_compartment_target_not_modified_by_unrelated_US":bool(frame[frame.condition=="A+unrelated-compartment-US"].matched_compartment_E_reduction_sum.eq(0).all()),
|
||||
"isolated_replay":rframe.groupby(["condition","pairings"]).agg(A_specific_seeds=("A_specific_functional","sum"),A_spike_L1=("A_MBON_spike_L1","mean"),B_spike_L1=("B_MBON_spike_L1","mean")).reset_index().to_dict("records")})
|
||||
|
||||
|
||||
if __name__=="__main__":main()
|
||||
@@ -0,0 +1,227 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4N event-level KC/DAN side-channel on unmodified frozen LIF dynamics."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import resource
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import torch
|
||||
|
||||
from phase4h_audit import write_json
|
||||
from phase4i import Validator
|
||||
from phase4l import OdorSimulator, pn_drive
|
||||
from phase4m import load_odor_inputs
|
||||
from phase4n import load_config
|
||||
|
||||
|
||||
class EventSimulator(OdorSimulator):
|
||||
def run_events(self, targets, rates_hz, seed, duration_ms, delta, dan_targets, dan_pulse_ms, all_dan,
|
||||
save_all_events=False):
|
||||
targets = np.asarray(targets, np.int32); rates_hz = np.asarray(rates_hz, np.float32)
|
||||
self.v.install(delta); self.rt.weights[self.apl_edge_gpu] = 0.0; self.rt.reset(int(seed))
|
||||
steps = int(round(duration_ms / self.dt)); tg = torch.as_tensor(targets, dtype=torch.int64, device=self.device)
|
||||
rg = torch.as_tensor(rates_hz * self.dt / 1000.0, dtype=torch.float32, device=self.device)
|
||||
gen = torch.Generator(device=self.device); gen.manual_seed(int(seed))
|
||||
forced = torch.rand((steps, len(targets)), generator=gen, device=self.device) < rg[None, :]
|
||||
dan_targets = np.asarray(dan_targets, np.int32); dg = torch.as_tensor(dan_targets, dtype=torch.int64, device=self.device)
|
||||
dan_forced = self._forced(steps, dan_targets, seed + 991, None, [dan_pulse_ms])
|
||||
dan_all_gpu = torch.as_tensor(all_dan, dtype=torch.int64, device=self.device)
|
||||
apl = self.cfg["apl_overlay"]; apl_decay = math.exp(-self.dt / float(apl["tau_ms"]))
|
||||
trace_decay = math.exp(-self.dt / float(self.cfg["eligibility_v2"]["kc_trace_tau_ms"]))
|
||||
apl_state = torch.zeros(len(self.apl), dtype=torch.float32, device=self.device)
|
||||
trace = torch.zeros(len(self.kc), dtype=torch.float32, device=self.device)
|
||||
kc_history = torch.zeros((steps, len(self.kc)), dtype=torch.bool, device=self.device)
|
||||
dan_history = torch.zeros((steps, len(all_dan)), dtype=torch.bool, device=self.device)
|
||||
natural_dan_history = torch.zeros_like(dan_history)
|
||||
trace_at_gate = None; all_nodes = []; all_steps = []
|
||||
alpha, gd, rest, reset, threshold, refsteps = self.rt.alpha, self.rt.g_decay, self.rt.rest, self.rt.reset_value, self.rt.threshold, self.rt.refractory_steps
|
||||
t0 = time.perf_counter()
|
||||
for step in range(steps):
|
||||
slot = step % self.rt.queue_len; self.rt.g.mul_(gd).add_(self.rt.delay_queue[slot]); self.rt.delay_queue[slot].zero_()
|
||||
self.rt.refractory.clamp_min_(1).sub_(1); available = self.rt.refractory == 0
|
||||
self.rt.v[available] += alpha * (self.rt.g[available] - (self.rt.v[available] - rest)); self.rt.v[~available] = reset
|
||||
spikes = available & (self.rt.v >= threshold)
|
||||
accepted = forced[step] & available[tg]
|
||||
if accepted.any(): spikes[tg[accepted]] = True
|
||||
natural_dan_history[step] = spikes[dan_all_gpu]
|
||||
if len(dan_targets):
|
||||
da = dan_forced[step] & available[dg]
|
||||
if da.any(): spikes[dg[da]] = True
|
||||
sources = torch.nonzero(spikes, as_tuple=False).flatten()
|
||||
kc_history[step] = spikes[self.kc_gpu]
|
||||
dan_history[step] = spikes[dan_all_gpu]
|
||||
if save_all_events and sources.numel():
|
||||
all_nodes.append(sources.to(torch.int32)); all_steps.append(torch.full((sources.numel(),), step, dtype=torch.int32, device=self.device))
|
||||
if sources.numel():
|
||||
self.rt.v[sources] = reset; self.rt.refractory[sources] = refsteps; eids = self.rt._hybrid_edges(sources)
|
||||
if eids.numel(): self.rt.delay_queue[(step + self.rt.delay_steps) % self.rt.queue_len].scatter_add_(0, self.rt.indices[eids].long(), self.rt.weights[eids])
|
||||
trace.mul_(trace_decay).add_(spikes[self.kc_gpu].to(torch.float32)).clamp_max_(1.0)
|
||||
if step == int(round(dan_pulse_ms / self.dt)): trace_at_gate = trace.clone()
|
||||
apl_state.mul_(apl_decay).add_(torch.mv(self.input_gpu, spikes.to(torch.float32)) * float(apl["normalization"]))
|
||||
inhibition = torch.matmul(apl_state, self.output_gpu) * float(apl["gain"])
|
||||
self.rt.g.index_add_(0, self.kc_gpu, -inhibition)
|
||||
torch.cuda.synchronize()
|
||||
event_table = None
|
||||
if save_all_events:
|
||||
event_table = pd.DataFrame({"step": torch.cat(all_steps).cpu().numpy() if all_steps else np.empty(0,np.int32),
|
||||
"neuron_idx": torch.cat(all_nodes).cpu().numpy() if all_nodes else np.empty(0,np.int32)})
|
||||
return {"kc": kc_history.cpu().numpy(), "dan": dan_history.cpu().numpy(), "natural_dan": natural_dan_history.cpu().numpy(), "all_events": event_table,
|
||||
"trace_at_external": trace_at_gate.cpu().numpy(), "wall_seconds": time.perf_counter()-t0,
|
||||
"peak_vram_bytes": int(torch.cuda.max_memory_allocated())}
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", type=Path, required=True); ap.add_argument("--max-pairings", type=int, default=100); ap.add_argument("--train-seed", type=int)
|
||||
args = ap.parse_args(); ncfg, mcfg, cfg, basepath = load_config(args.config)
|
||||
exp = Path(ncfg["experiment_root"])
|
||||
identity = pd.read_parquet(exp / "edge-identity/frozen-edge-identity.parquet")
|
||||
history = pd.read_parquet(Path(mcfg["experiment_root"]) / "efficacy-overlay/update-history.parquet")
|
||||
v = Validator(cfg, basepath, exp); sim = EventSimulator(v, cfg, exp)
|
||||
if not np.array_equal(v.edge_ids, identity.edge_id.to_numpy(np.int64)):
|
||||
raise RuntimeError("edge identity order mismatch")
|
||||
all_dan = np.unique(v.confirmed.source_neuron_idx.to_numpy(np.int32))
|
||||
matched = v.dan_targets(str(ncfg["matched_dan_type"]), str(ncfg["matched_compartment_verbatim"]))
|
||||
matched_set = set(map(int, matched)); all_dan_to_local = {int(n): i for i,n in enumerate(all_dan)}
|
||||
matched_loc = np.array([all_dan_to_local[int(n)] for n in matched], np.int32)
|
||||
panel, vectors, selection, mapping = load_odor_inputs(cfg)
|
||||
gloms = selection["glomeruli_order"]; matched_sum = float(np.median(panel.mapped_response_sum))
|
||||
oid = selection["ODOR_A"]["odor_id"]
|
||||
pn = pn_drive(oid, vectors[oid], mapping, gloms, str(mcfg["protocol"]["input_mode"]), matched_sum, cfg)[:2]
|
||||
src = identity.source_neuron_idx.to_numpy(np.int32)
|
||||
kclook = np.full(v.n, -1, np.int32); kclook[v.kc] = np.arange(len(v.kc), dtype=np.int32)
|
||||
loc = kclook[src]
|
||||
comp = identity.compartment.astype(str).to_numpy() == str(ncfg["matched_compartment_verbatim"])
|
||||
bound = np.float32(1.0 - cfg["eligibility_v2"]["max_fractional_depression"])
|
||||
k = np.float32(cfg["eligibility_v2"]["eta_effective_mv"] / cfg["model"]["w_syn_mv_per_anatomical_synapse"])
|
||||
tau = float(cfg["eligibility_v2"]["kc_trace_tau_ms"])
|
||||
short_tau = float(ncfg["shadow_trace_tau_ms"])
|
||||
pulse_step = int(round(float(mcfg["protocol"]["dan_time_ms"]) / sim.dt))
|
||||
hist_by = {(int(s),int(p)): frame for (s,p),frame in history.groupby(["train_seed","pairing"])}
|
||||
eid_to_local = pd.Series(np.arange(len(identity), dtype=np.int32), index=identity.edge_id.to_numpy(np.int64))
|
||||
audit_rows = []; gate_rows = []; shadow_rows = []; perf = []; kc_event_frames = []; dan_event_frames = []; mismatches = []
|
||||
key = {10,20,50,60,75,100}
|
||||
for seed in ([args.train_seed] if args.train_seed is not None else ncfg["train_seeds"]):
|
||||
additive = np.zeros(len(identity), np.float32)
|
||||
E_ext = np.ones(len(identity), np.float32)
|
||||
E_all = np.ones(len(identity), np.float32)
|
||||
rng = np.random.default_rng(seed + int(ncfg["random_us_seed_offset"]))
|
||||
for pairing in range(1, args.max_pairings+1):
|
||||
audit_start = len(audit_rows); gate_start = len(gate_rows)
|
||||
capture = pairing in key
|
||||
run = sim.run_events(*pn, seed+pairing, float(mcfg["protocol"]["duration_ms"]), additive, matched,
|
||||
float(mcfg["protocol"]["dan_time_ms"]), all_dan, save_all_events=capture)
|
||||
perf.append({"seed":seed,"pairing":pairing,"wall_seconds":run["wall_seconds"],"peak_vram_bytes":run["peak_vram_bytes"]})
|
||||
kc = run["kc"]; dan = run["natural_dan"]
|
||||
ks, kl = np.nonzero(kc)
|
||||
if len(ks):
|
||||
kc_event_frames.append(pd.DataFrame({"seed":seed,"pairing":pairing,"step":ks.astype(np.int32),
|
||||
"neuron_idx":v.kc[kl].astype(np.int32)}))
|
||||
ds, dl = np.nonzero(dan)
|
||||
if len(ds):
|
||||
dan_event_frames.append(pd.DataFrame({"seed":seed,"pairing":pairing,"step":ds.astype(np.int32),
|
||||
"neuron_idx":all_dan[dl].astype(np.int32),"source":"endogenous-spike"}))
|
||||
trace = np.zeros(len(v.kc), np.float32); short = np.zeros_like(trace)
|
||||
last = np.full(len(v.kc), -1, np.int32); first = np.full(len(v.kc), -1, np.int32)
|
||||
external_at_gate = None
|
||||
endogenous_steps = 0; endogenous_events = 0
|
||||
random_step = int(round(float(rng.uniform(*ncfg["random_us_window_ms"])) / sim.dt))
|
||||
random_trace = None
|
||||
for step in range(len(kc)):
|
||||
spike = kc[step]
|
||||
trace *= np.float32(math.exp(-sim.dt / tau)); trace += spike.astype(np.float32); np.minimum(trace, 1.0, out=trace)
|
||||
short *= np.float32(math.exp(-sim.dt / short_tau)); short += spike.astype(np.float32); np.minimum(short, 1.0, out=short)
|
||||
last[spike] = step; first[spike & (first < 0)] = step
|
||||
if step == random_step: random_trace = trace[loc].copy()
|
||||
endogenous = bool(np.any(dan[step, matched_loc]))
|
||||
external = step == pulse_step
|
||||
if endogenous:
|
||||
endogenous_steps += 1; endogenous_events += int(np.count_nonzero(dan[step, matched_loc]))
|
||||
if external:
|
||||
external_at_gate = trace[loc].copy()
|
||||
max_error = float(np.max(np.abs(trace - run["trace_at_external"])))
|
||||
if max_error > 2e-5: raise RuntimeError(f"KC trace reconstruction mismatch {max_error}")
|
||||
if external or endogenous:
|
||||
elig = trace[loc]; active = comp & (elig > 0)
|
||||
before = E_all[active].copy()
|
||||
E_all[active] = np.maximum(bound, E_all[active] * (np.float32(1.0)-k*elig[active]))
|
||||
gate_rows.append({"seed":seed,"pairing":pairing,"step":step,"time_ms":step*sim.dt,
|
||||
"source": "both" if external and endogenous else ("external-US-associated" if external else "endogenous-feedback-associated"),
|
||||
"matched_endogenous_DAN_spikes":int(np.count_nonzero(dan[step,matched_loc])),
|
||||
"eligible_edges":int(active.sum()),"E_change_sum":float(np.sum(before-E_all[active]))})
|
||||
if external:
|
||||
last_age_ms = np.where(last[loc] >= 0, (step-last[loc])*sim.dt, np.inf)
|
||||
a = comp & (trace[loc] > 0)
|
||||
for local in np.flatnonzero(a):
|
||||
source = int(src[local]); age = float(last_age_ms[local])
|
||||
audit_rows.append({"seed":seed,"pairing":pairing,"edge_id":int(identity.edge_id.iat[local]),
|
||||
"source_neuron_idx":source,"source_body_id":int(identity.source_body_id.iat[local]),
|
||||
"target_neuron_idx":int(identity.target_neuron_idx.iat[local]),"compartment":str(identity.compartment.iat[local]),
|
||||
"identity_group":str(identity.identity_group.iat[local]),"update_time_ms":step*sim.dt,
|
||||
"eligibility_v2":float(trace[loc[local]]),"eligibility_short_shadow":float(short[loc[local]]),
|
||||
"event_coincidence_shadow":bool(age <= float(ncfg["event_coincidence_window_ms"])),
|
||||
"first_activation_only_shadow":bool(first[loc[local]] == last[loc[local]]),
|
||||
"last_local_KC_spike_age_ms":age,"first_local_KC_spike_ms":float(first[loc[local]]*sim.dt),
|
||||
"matched_external_DAN_arrival_ms":step*sim.dt,
|
||||
"endogenous_matched_DAN_simultaneous":bool(endogenous),
|
||||
"source_class": "external-US-associated", "delta_E_bounded":float(-k*trace[loc[local]]*E_ext[local])})
|
||||
if external_at_gate is None: raise RuntimeError("external gate missing")
|
||||
rec = hist_by[(int(seed),int(pairing))]
|
||||
active_loc = eid_to_local.loc[rec.edge_id.to_numpy(np.int64)].to_numpy(np.int32)
|
||||
err = float(np.max(np.abs(external_at_gate[active_loc] - rec.eligibility.to_numpy(np.float32)))) if len(active_loc) else 0.0
|
||||
exact = err <= 2e-5
|
||||
if not exact:
|
||||
mismatches.append({"seed":seed,"pairing":pairing,"max_abs_eligibility_error":err})
|
||||
for row in audit_rows[audit_start:]: row["frozen_trace_match"] = exact
|
||||
for row in gate_rows[gate_start:]: row["frozen_trace_match"] = exact
|
||||
# Advance the next trial from the sealed Phase 4M overlay, never from a divergent audit replay.
|
||||
old_E = rec.E_bounded_before.to_numpy(np.float32); new_E = rec.E_bounded_after.to_numpy(np.float32)
|
||||
if len(active_loc) and float(np.max(np.abs(E_ext[active_loc]-old_E))) > 1e-6:
|
||||
raise RuntimeError(f"sealed E history before-state mismatch {seed}/{pairing}")
|
||||
E_ext[active_loc] = new_E
|
||||
true_elig = rec.eligibility.to_numpy(np.float32)
|
||||
additive[active_loc] = np.maximum(-np.float32(cfg["eligibility_v2"]["max_fractional_depression"])*v.base_candidate[active_loc],
|
||||
additive[active_loc] - np.float32(cfg["eligibility_v2"]["eta_effective_mv"])*true_elig)
|
||||
if pairing in ncfg["checkpoints"]:
|
||||
with np.load(exp / "acquisition-trajectory" / f"E-seed-{seed}-pairings-{pairing}.npz") as f:
|
||||
chk_err = float(np.max(np.abs(E_ext-f["E_bounded"])))
|
||||
if chk_err > 1e-6: raise RuntimeError(f"external E reconstruction mismatch {seed}/{pairing}: {chk_err}")
|
||||
np.savez_compressed(exp / "endogenous-dan" / f"E-all-DAN-shadow-seed-{seed}-pairings-{pairing}.npz", E_all_DAN=E_all,
|
||||
E_external_only=E_ext, edge_id=identity.edge_id.to_numpy(np.int64))
|
||||
shadow_rows.append({"seed":seed,"pairing":pairing,"random_US_step":random_step,
|
||||
"external_matched_E_reduction_sum":float(np.sum(1-E_ext)),"all_DAN_shadow_E_reduction_sum":float(np.sum(1-E_all)),
|
||||
"matched_endogenous_DAN_steps":endogenous_steps,"matched_endogenous_DAN_spikes":endogenous_events,
|
||||
"random_US_trace_positive_edges":int(np.count_nonzero(comp & (random_trace>0))) if random_trace is not None else 0,
|
||||
"frozen_trace_match":exact,"max_abs_eligibility_error":err})
|
||||
if capture and run["all_events"] is not None:
|
||||
run["all_events"].to_parquet(exp / "whole-brain" / f"spike-events-seed-{seed}-pairings-{pairing}.parquet", index=False)
|
||||
if pairing % 20 == 0:
|
||||
print(json.dumps({"seed":seed,"pairing":pairing,"endogenous_steps":endogenous_steps,"E_external_sum":float(np.sum(1-E_ext)),"E_all_sum":float(np.sum(1-E_all))}),flush=True)
|
||||
pd.DataFrame(audit_rows).to_parquet(exp / "local-eligibility/event-level-updates.parquet", index=False)
|
||||
pd.DataFrame(gate_rows).to_parquet(exp / "dan-source/event-level-gates.parquet", index=False)
|
||||
pd.concat(kc_event_frames,ignore_index=True).to_parquet(exp / "local-eligibility/KC-spike-history.parquet",index=False)
|
||||
if dan_event_frames:
|
||||
pd.concat(dan_event_frames,ignore_index=True).to_parquet(exp / "dan-source/endogenous-DAN-spike-history.parquet",index=False)
|
||||
pd.DataFrame(shadow_rows).to_parquet(exp / "endogenous-dan/shadow-trajectory.parquet", index=False)
|
||||
write_json(exp / "regression/event-trace-reproduction.json", {"all_passed":len(mismatches)==0,"mismatched_trials":mismatches,
|
||||
"mismatched_count":len(mismatches),"audited_trial_count":len(shadow_rows),
|
||||
"next_trial_overlay_restored_from_sealed_Phase4M_history":True})
|
||||
pd.DataFrame(perf).to_parquet(exp / "performance/event-audit-calls.parquet", index=False)
|
||||
write_json(exp / "endogenous-dan/summary.json", {"mode_A": "observational frozen Phase 4M LIF with endogenous DAN spikes recorded",
|
||||
"mode_B": "external US-only plasticity gate, exact Phase 4M E reproduction",
|
||||
"all_DAN_mode": "counterfactual open-loop shadow E; same frozen Phase 4M additive brain trajectory; not a new brain model",
|
||||
"all_DAN_scope": "only metadata-confirmed DAN mapped to matched compartment; other DAN unresolved",
|
||||
"frozen_brain_dynamics_changed": False, "training_seed_count":len(ncfg["train_seeds"]),"pairings_per_seed":args.max_pairings})
|
||||
write_json(exp / "performance/event-audit.json", {"calls":len(perf),"total_wall_seconds":float(sum(x["wall_seconds"] for x in perf)),
|
||||
"mean_wall_seconds":float(np.mean([x["wall_seconds"] for x in perf])),"peak_VRAM_bytes":max(x["peak_vram_bytes"] for x in perf),
|
||||
"peak_RAM_KiB":int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss),
|
||||
"KC_boolean_history_bytes_per_trial":int(len(run["kc"])*len(v.kc)),"DAN_boolean_history_bytes_per_trial":int(len(run["dan"])*len(all_dan)),
|
||||
"no_dense_time_by_edge_tensor":True})
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,262 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4N synthesis, independent checks, and final SHA-256 manifest."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gc
|
||||
import json
|
||||
import resource
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import torch
|
||||
|
||||
from phase4h_audit import sha256, verify_checksum, write_json
|
||||
from phase4i import phase4c_regression
|
||||
from phase4k import dataset_checks, seal
|
||||
from phase4m import frozen_checks, frozen_hashes
|
||||
from phase4n import load_config
|
||||
from phase4n_finish import curves, shared_decomposition
|
||||
from phase4i import Validator
|
||||
from phase4l import OdorSimulator
|
||||
|
||||
|
||||
def contamination(exp, history):
|
||||
event = pd.read_parquet(exp / "local-eligibility/event-level-updates.parquet")
|
||||
if event.duplicated(["seed", "pairing", "edge_id"]).any():
|
||||
raise RuntimeError("duplicate local eligibility row")
|
||||
h = history[["train_seed", "pairing", "edge_id", "E_bounded_before", "E_bounded_after"]].rename(columns={"train_seed":"seed"})
|
||||
x = event.merge(h,on=["seed","pairing","edge_id"],how="left",validate="one_to_one")
|
||||
unmatched_runtime_rows=int(x.E_bounded_after.isna().sum())
|
||||
nonmatching_trace_rows=int((~x.frozen_trace_match).sum())
|
||||
excluded_B_rows=int(((~x.frozen_trace_match)&(x.identity_group=="B-specific")).sum())
|
||||
x=x[x.frozen_trace_match & x.E_bounded_after.notna()].copy()
|
||||
x["actual_E_depression"] = x.E_bounded_before-x.E_bounded_after
|
||||
age = x.last_local_KC_spike_age_ms.to_numpy(np.float64)
|
||||
x["temporal_credit_class"] = np.where(age <= 1.0,"coincident_<=1ms",np.where(age <= 20.0,"recent_1to20ms","long_trace_>20ms"))
|
||||
x.to_parquet(exp / "b-contamination/updated-edges-with-temporal-source.parquet",index=False)
|
||||
b=x[(x.identity_group=="B-specific")&(x.actual_E_depression>1e-7)]
|
||||
total=float(b.actual_E_depression.sum())
|
||||
fractions={str(k):float(g.actual_E_depression.sum()/total) if total else None for k,g in b.groupby("temporal_credit_class")}
|
||||
event_fraction={str(k):int(len(g)) for k,g in b.groupby("temporal_credit_class")}
|
||||
relative_short=np.divide(b.eligibility_short_shadow.to_numpy(np.float64),b.eligibility_v2.to_numpy(np.float64),
|
||||
out=np.zeros(len(b),np.float64),where=b.eligibility_v2.to_numpy(np.float64)>0)
|
||||
weighted_short=float(np.dot(b.actual_E_depression.to_numpy(np.float64),relative_short)/total) if total else None
|
||||
report={"B_specific_update_rows":int(len(b)),"B_specific_unique_edges":int(b.edge_id.nunique()),
|
||||
"B_specific_unique_source_KC":int(b.source_neuron_idx.nunique()),
|
||||
"first_B_specific_update_pairing_by_seed":{str(seed):int(frame.pairing.min()) for seed,frame in b.groupby("seed")},
|
||||
"B_specific_E_depression_sum_with_exact_replay":total,"temporal_credit_weighted_fraction":fractions,"temporal_credit_event_counts":event_fraction,
|
||||
"excluded_nonmatching_trace_rows":nonmatching_trace_rows,"excluded_nonmatching_B_rows":excluded_B_rows,"unmatched_runtime_rows":unmatched_runtime_rows,
|
||||
"external_matched_US_gate_fraction":1.0,"endogenous_DAN_as_gate_fraction_in_frozen_model":0.0,
|
||||
"causal_route_fractions":{"direct_A_driven":None,"recurrent_KC":None,"MBON_DAN_feedback":None,"olfactory_feedback":None,"unresolved":1.0},
|
||||
"reason":"Whole-brain source spike timing plus a positive anatomical edge is not proof of membrane-spike causation. Frozen Phase 4M history has no causal predecessor trace. Do not assign invented path percentages.",
|
||||
"shadow_short_tau_20ms_excluded_B_update_fraction":float(b.loc[b.eligibility_short_shadow<.01,"actual_E_depression"].sum()/total) if total else None,
|
||||
"shadow_short_tau_20ms_weighted_eligibility_ratio_to_frozen":weighted_short,
|
||||
"event_coincidence_1ms_included_B_update_fraction":float(b.loc[b.event_coincidence_shadow,"actual_E_depression"].sum()/total) if total else None}
|
||||
report["first_activation_only_included_B_update_fraction"]=(float(b.loc[b.first_activation_only_shadow,"actual_E_depression"].sum()/total) if total else None)
|
||||
write_json(exp / "b-contamination/summary.json",report)
|
||||
return report
|
||||
|
||||
|
||||
def dan_summary(exp, identity, ncfg, cfg=None):
|
||||
gates=pd.read_parquet(exp / "dan-source/event-level-gates.parquet")
|
||||
trail=pd.read_parquet(exp / "endogenous-dan/shadow-trajectory.parquet")
|
||||
source=gates.groupby("source").agg(gates=("source","size"),E_depression_sum=("E_change_sum","sum")).reset_index()
|
||||
external_only=trail[trail.pairing==100].external_matched_E_reduction_sum.mean()
|
||||
all_dan=trail[trail.pairing==100].all_DAN_shadow_E_reduction_sum.mean()
|
||||
report={"source_breakdown":source.to_dict("records"),"trial_count":len(trail),
|
||||
"matched_endogenous_DAN_spikes_total":int(trail.matched_endogenous_DAN_spikes.sum()),
|
||||
"external_only_E_depression_sum_mean_100":float(external_only),"all_DAN_shadow_E_depression_sum_mean_100":float(all_dan),
|
||||
"additional_all_DAN_shadow_E_depression_sum_mean_100":float(all_dan-external_only),
|
||||
"frozen_Phase4M_endogenous_DAN_teaching_gate_count":0,
|
||||
"all_DAN_shadow_interpretation":"counterfactual open-loop teaching-gate interpretation; not frozen model and not measured dopamine concentration"}
|
||||
if cfg is not None:
|
||||
mapping=pd.read_parquet(Path(cfg["paths"]["phase4h_root"])/"mappings/dan-to-mbon-confirmed.parquet")
|
||||
mapping=mapping[["source_neuron_idx","source_annotation_type","source_compartment_verbatim"]].drop_duplicates()
|
||||
matched_set=set(map(int,mapping.loc[mapping.source_compartment_verbatim.astype(str)==str(ncfg["matched_compartment_verbatim"]),"source_neuron_idx"]))
|
||||
mapping=mapping[["source_neuron_idx","source_annotation_type"]].drop_duplicates(subset=["source_neuron_idx"])
|
||||
events_file=exp/"dan-source/endogenous-DAN-spike-history.parquet"
|
||||
if events_file.exists():
|
||||
events=pd.read_parquet(events_file)
|
||||
events=events.merge(mapping,left_on="neuron_idx",right_on="source_neuron_idx",how="left")
|
||||
typ=events.source_annotation_type.fillna("unknown").astype(str)
|
||||
events["cluster"] = np.where(typ.str.startswith("PAM"),"PAM",np.where(typ.str.startswith("PPL1"),"PPL1","other/unknown"))
|
||||
report["endogenous_spike_cluster_counts"]={str(k):int(v) for k,v in events.cluster.value_counts().items()}
|
||||
report["confirmed_type_counts"]={str(k):int(v) for k,v in typ.value_counts().items()}
|
||||
report["matched_compartment_spikes"] = int(events.neuron_idx.isin(matched_set).sum())
|
||||
events.to_parquet(exp/"dan-source/endogenous-DAN-with-type.parquet",index=False)
|
||||
write_json(exp / "dan-source/summary.json",report)
|
||||
return report
|
||||
|
||||
|
||||
def shared_summary(exp):
|
||||
p=exp / "shared-edge/isolated-decomposition.parquet"
|
||||
if not p.exists(): return {"status":"NOT-RUN-NO-PEAK"}
|
||||
table=pd.read_parquet(p)
|
||||
agg=table.groupby(["decomposition","odor"]).agg(spike_L1=("MBON_spike_L1","mean"),transmission_reduction=("transmitted_reduction_percent","mean"),
|
||||
latency_delta=("first_latency_delta_ms","mean")).reset_index()
|
||||
write_json(exp / "shared-edge/summary.json",{"scope":"isolated replay only; existing edges never deleted", "group_means":agg.to_dict("records")})
|
||||
return agg.to_dict("records")
|
||||
|
||||
|
||||
def separation(exp, peak):
|
||||
whole=pd.read_parquet(exp / "whole-brain/checkpoint-responses.parquet")
|
||||
iso=pd.read_parquet(exp / "isolated/response-by-checkpoint.parquet")
|
||||
rows=[]
|
||||
for pair in sorted(whole.pairings.unique()):
|
||||
w=whole[whole.pairings==pair].pivot(index="train_seed",columns="odor",values="MBON_spike_L1")
|
||||
i=iso[iso.pairings==pair]
|
||||
whole_sep=float((w["A"]-w["B"]).mean())
|
||||
iso_sep=float(i.functional_separation_spike_L1.mean())
|
||||
rows.append({"pairings":int(pair),"isolated_A_minus_B_spike_L1":iso_sep,"wholebrain_A_minus_B_spike_L1":whole_sep,
|
||||
"wholebrain_specificity_penalty":iso_sep-whole_sep,
|
||||
"wholebrain_A_specific_seeds":int(((w["A"]>w["B"])&(w["A"]>0)).sum())})
|
||||
frame=pd.DataFrame(rows);frame.to_parquet(exp / "whole-brain/specificity-penalty.parquet",index=False)
|
||||
return {"trajectory":rows,"at_peak":next((r for r in rows if r["pairings"]==peak),None)}
|
||||
|
||||
|
||||
def control_summary(exp):
|
||||
original=json.loads((exp / "controls/summary.json").read_text())
|
||||
independent=json.loads((exp / "controls/independent-summary.json").read_text())
|
||||
wholebrain=json.loads((exp / "controls/wholebrain-summary.json").read_text())
|
||||
original.update({"B_matched_US":"RUN-INDEPENDENT-100-PAIRING-TRAJECTORY",
|
||||
"random_US_timing":"RUN-INDEPENDENT-100-PAIRING-TRAJECTORY",
|
||||
"unrelated_compartment_US":"RUN-INDEPENDENT-100-PAIRING-TRAJECTORY",
|
||||
"all_DAN_gate":"RUN-OPEN-LOOP-SHADOW-100-PAIRING-TRAJECTORY",
|
||||
"full_control_matrix_pass":False,"independent_results":"controls/independent-summary.json",
|
||||
"all_DAN_closed_loop":False,"independent_wholebrain":"controls/wholebrain-summary.json"})
|
||||
write_json(exp / "controls/summary.json",original)
|
||||
return original, independent, wholebrain
|
||||
|
||||
|
||||
def integrity(exp, ncfg, mcfg, cfg, basepath):
|
||||
gc.collect();torch.cuda.empty_cache()
|
||||
previous=frozen_checks(cfg)
|
||||
mroot=Path(mcfg["experiment_root"]);mmodel=Path(mcfg["model_root"])
|
||||
phase4m={"experiment":verify_checksum(mroot,mroot/"manifests/checksums.sha256")["passed"],
|
||||
"model":verify_checksum(mmodel,mmodel/"checksums.sha256")["passed"]}
|
||||
dataset=dataset_checks(cfg)
|
||||
before=json.loads((exp / "audit/frozen-integrity.json").read_text())
|
||||
# A second replay detects intermittent GPU/event divergence; it never rewrites frozen references.
|
||||
replay=phase4c_regression(cfg,basepath,exp)
|
||||
phase4m_summary=json.loads((mroot/"audit/final-summary.json").read_text())
|
||||
reference_hashes=phase4m_summary["integrity"]["after"]
|
||||
static={"previous_phases":previous["all_passed"],"phase4m":all(phase4m.values()),"phase1_to_3b":dataset["all_passed"],
|
||||
"frozen_hashes_unchanged":frozen_hashes(cfg)==reference_hashes}
|
||||
report={"static_checksums_all_passed":bool(all(static.values())),"static":static,
|
||||
"phase4c_reference_replay_first":before["phase4c_reference_regression"],"phase4c_reference_replay_second":replay,
|
||||
"dynamic_regression_all_passed":bool(before["phase4c_reference_regression"]["all_passed"] and replay["all_passed"]),
|
||||
"all_passed":bool(all(static.values()) and before["phase4c_reference_regression"]["all_passed"] and replay["all_passed"])}
|
||||
write_json(exp / "audit/frozen-integrity-final.json",report)
|
||||
return report
|
||||
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument("--config",type=Path,required=True);args=ap.parse_args()
|
||||
ncfg,mcfg,cfg,basepath=load_config(args.config);exp=Path(ncfg["experiment_root"])
|
||||
identity=pd.read_parquet(exp / "edge-identity/frozen-edge-identity.parquet")
|
||||
response_curve=pd.read_parquet(exp/"specificity-curve/memory-specificity.parquet")
|
||||
eligible=response_curve[response_curve.A_specific_functional_seeds>=3]
|
||||
corrected_peak=None
|
||||
if not eligible.empty:
|
||||
best=float(eligible.mean_spike_L1_separation.max())
|
||||
eligible=eligible[np.isclose(eligible.mean_spike_L1_separation,best,atol=1e-9,rtol=0)]
|
||||
latency_best=float(eligible.mean_latency_separation_ms.max())
|
||||
eligible=eligible[eligible.mean_latency_separation_ms>=latency_best-float(cfg["model"]["dt_ms"])/2]
|
||||
corrected_peak=int(eligible.pairings.min())
|
||||
old_peak=json.loads((exp/"specificity-curve/peak.json").read_text())["pairings"]
|
||||
if corrected_peak!=old_peak:
|
||||
write_json(exp/"audit/peak-numerical-tie-correction.json",{"original_checkpoint":old_peak,"corrected_checkpoint":corrected_peak,
|
||||
"reason":"50 and 60 have the same five-seed spike-L1 separation; latency difference is sub-femtosecond float summation noise, far below 0.1 ms simulation dt. Earliest-tie rule selects 50.",
|
||||
"not_a_learning_parameter_change":True,"identity_and_E_trajectories_unchanged":True})
|
||||
write_json(exp/"specificity-curve/peak.json",{"status":"SPECIFICITY-PEAK-IDENTIFIED","pairings":corrected_peak,
|
||||
"selection_rule":ncfg["peak_rule"],"numerical_tie_correction_record":"audit/peak-numerical-tie-correction.json"})
|
||||
curves(ncfg,exp,pd.read_parquet(exp/"specificity-curve/CSI.parquet"),response_curve,identity)
|
||||
validator=Validator(cfg,basepath,exp);sim=OdorSimulator(validator,cfg,exp)
|
||||
shared_decomposition(ncfg,exp,identity,sim,validator,corrected_peak)
|
||||
del sim,validator
|
||||
stopping=json.loads((exp/"stopping-analysis/summary.json").read_text())
|
||||
stopping["functional_peak_checkpoint"]=corrected_peak
|
||||
write_json(exp/"stopping-analysis/summary.json",stopping)
|
||||
# Complete the explicitly predeclared t=0 checkpoint without rerunning any brain trial.
|
||||
groups0=pd.read_parquet(exp/"acquisition-trajectory/group-plasticity.parquet")
|
||||
csi0=pd.read_parquet(exp/"specificity-curve/CSI.parquet")
|
||||
if 0 not in set(groups0.pairings):
|
||||
z=[];cz=[]
|
||||
for seed in ncfg["train_seeds"]:
|
||||
one=np.ones(len(identity),np.float32)
|
||||
np.savez_compressed(exp/"acquisition-trajectory"/f"E-seed-{seed}-pairings-0.npz",E_bounded=one,E_unbounded=one,edge_id=identity.edge_id.to_numpy(np.int64))
|
||||
for mechanism in ("bounded","unbounded"):
|
||||
for group_name,count in identity.identity_group.value_counts().items():
|
||||
z.append({"seed":seed,"pairings":0,"mechanism":mechanism,"group":group_name,"total_edges":int(count),
|
||||
"updated_edges":0,"mean_delta_E":0.0,"median_delta_E":0.0,"weighted_transmission_reduction":0.0,"E_change_total":0.0})
|
||||
cz.append({"seed":seed,"pairings":0,"mechanism":mechanism,"A-specific":0.0,"B-specific":0.0,"shared":0.0,"inactive":0.0,
|
||||
"CSI":0.0,"A_to_shared_ratio":np.nan,"A_to_B_ratio":np.nan})
|
||||
pd.concat([groups0,pd.DataFrame(z)],ignore_index=True).sort_values(["seed","pairings","mechanism","group"]).to_parquet(exp/"acquisition-trajectory/group-plasticity.parquet",index=False)
|
||||
pd.concat([csi0,pd.DataFrame(cz)],ignore_index=True).sort_values(["seed","pairings","mechanism"]).to_parquet(exp/"specificity-curve/CSI.parquet",index=False)
|
||||
history=pd.read_parquet(Path(mcfg["experiment_root"])/"efficacy-overlay/update-history.parquet")
|
||||
peak=json.loads((exp/"specificity-curve/peak.json").read_text())["pairings"]
|
||||
onset=json.loads((exp/"specificity-curve/generalization-onset.json").read_text())["pairings"]
|
||||
contam=contamination(exp,history);dan=dan_summary(exp,identity,ncfg,cfg);shared=shared_summary(exp)
|
||||
timed_inputs=json.loads((exp/"whole-brain/B-specific-timed-input-summary.json").read_text())
|
||||
sep=separation(exp,peak);controls,independent,control_whole=control_summary(exp)
|
||||
gate_specificity=json.loads((exp/"endogenous-dan/gate-specificity.json").read_text())
|
||||
perf=json.loads((exp / "performance/summary.json").read_text())
|
||||
audit_perf=json.loads((exp / "performance/event-audit.json").read_text())
|
||||
ctrl_perf={"training_calls":independent["training_calls"],"training_wall_seconds":independent["training_wall_seconds"],
|
||||
"mean_call_seconds":independent["mean_call_seconds"],"wholebrain_wall_seconds":control_whole["total_brain_wall_seconds"]}
|
||||
perf.update({"event_audit":audit_perf,"controls":ctrl_perf,"peak_RAM_KiB_all_stages":max(perf["peak_RAM_KiB"],audit_perf["peak_RAM_KiB"],independent["peak_RAM_KiB"]),
|
||||
"shadow_trace_overhead":"included in event-audit call time; no paired unaudited timing isolation"})
|
||||
write_json(exp / "performance/summary.json",perf)
|
||||
integ=integrity(exp,ncfg,mcfg,cfg,basepath)
|
||||
event_reproduction=json.loads((exp/"regression/event-trace-reproduction.json").read_text())
|
||||
csi=pd.read_parquet(exp / "specificity-curve/CSI.parquet")
|
||||
group=pd.read_parquet(exp / "acquisition-trajectory/group-plasticity.parquet")
|
||||
iso=pd.read_parquet(exp / "specificity-curve/memory-specificity.parquet")
|
||||
long_fraction=contam["temporal_credit_weighted_fraction"].get("long_trace_>20ms")
|
||||
shared_b=next((r["spike_L1"] for r in shared if r["decomposition"]=="A+shared" and r["odor"]=="B"),None) if isinstance(shared,list) else None
|
||||
a_only_b=next((r["spike_L1"] for r in shared if r["decomposition"]=="A-only" and r["odor"]=="B"),None) if isinstance(shared,list) else None
|
||||
mechanism_tags={"SPECIFICITY-PEAK-IDENTIFIED":peak is not None,
|
||||
"GENERALIZATION-ONSET-IDENTIFIED":onset is not None,
|
||||
"ENDOGENOUS-DAN-CREDIT-CONTAMINATION":False,
|
||||
"RECURRENT-KC-CREDIT-CONTAMINATION":None,
|
||||
"LONG-TRACE-CREDIT-CONTAMINATION":bool(long_fraction is not None and long_fraction>0),
|
||||
"SHARED-EDGE-GENERALIZATION-DOMINANT":bool(shared_b is not None and a_only_b is not None and shared_b>a_only_b),
|
||||
"NO-SINGLE-DOMINANT-CONTAMINATION-SOURCE":True}
|
||||
audit_complete=bool(event_reproduction["all_passed"] and integ["all_passed"] and controls["full_control_matrix_pass"])
|
||||
summary={"phase":"4N","status":"MECHANISM-AUDIT-COMPLETE-NO-CANDIDATE" if audit_complete else "PARTIAL-MECHANISM-AUDIT-NO-CANDIDATE",
|
||||
"edge_groups":identity.identity_group.value_counts().to_dict(),"specificity_peak_pairings":peak,
|
||||
"generalization_onset_pairings_by_predeclared_CSI_rule":onset,
|
||||
"functional_collapse_first_checkpoint":next((int(r.pairings) for _,r in iso.iterrows() if r.pairings>peak and r.A_specific_functional_seeds<3),None) if peak is not None else None,
|
||||
"group_curves":group[group.mechanism=="bounded"].groupby(["pairings","group"]).mean(numeric_only=True).reset_index()[["pairings","group","mean_delta_E","weighted_transmission_reduction"]].to_dict("records"),
|
||||
"isolated_curve":iso.to_dict("records"),"CSI_curve":csi[csi.mechanism=="bounded"].groupby("pairings").CSI.mean().reset_index().to_dict("records"),
|
||||
"B_specific_contamination":contam,"B_specific_timed_input_candidates":timed_inputs,"DAN":dan,"gate_specificity":gate_specificity,"shared_edge_isolated":shared,"wholebrain_specificity":sep,
|
||||
"controls":controls,"control_wholebrain":control_whole,"event_trace_reproduction":event_reproduction,"mechanism_tags":mechanism_tags,"candidate_generated":False,"candidate_gate":{"external_vs_endogenous_distinguished":True,
|
||||
"causal_B_contamination_route_identified":False,"isolated_and_wholebrain_majority_improvement_demonstrated":False},
|
||||
"frozen_integrity":integ,"limitations":["Temporal predecessor class unresolved; no causal path attribution percentage claimed",
|
||||
"All-DAN efficacy is an open-loop shadow, not a closed-loop retrained brain",
|
||||
"Phase 4C mechanosensory frozen regression showed at least one divergent run; static SHA-256 passes but dynamic integrity cannot be called PASS"]}
|
||||
write_json(exp / "audit/final-summary.json",summary)
|
||||
model=Path(ncfg["model_root"]);model.mkdir(parents=True,exist_ok=True)
|
||||
for name in ("phase4n.py","phase4n_finish.py","phase4n_events.py","phase4n_controls.py","phase4n_control_whole.py","phase4n_alldan.py","phase4n_paths.py","phase4n_preflight.py","phase4n_finalize.py","phase4n.yaml"):
|
||||
source=Path(__file__).with_name(name)
|
||||
shutil.copy2(source,model/name)
|
||||
shutil.copy2(source,exp/"audit"/name)
|
||||
write_json(model/"VERSION.json",{"phase":"4N","status":summary["status"],"candidate_generated":False,
|
||||
"identity_sha256":sha256(exp/"edge-identity/frozen-edge-identity.parquet"),"config_sha256":sha256(args.config),
|
||||
"frozen_static_integrity":integ["static_checksums_all_passed"],"frozen_dynamic_regression":integ["dynamic_regression_all_passed"]})
|
||||
seal(model,model/"MANIFEST.json",model/"checksums.sha256")
|
||||
seal(exp,exp/"manifests/manifest.json",exp/"manifests/checksums.sha256")
|
||||
self_check={}
|
||||
for root,relative in ((model,"checksums.sha256"),(exp,"manifests/checksums.sha256")):
|
||||
result=subprocess.run(["sha256sum","-c",relative],cwd=root,stdout=subprocess.DEVNULL,stderr=subprocess.PIPE,text=True)
|
||||
self_check[str(root)]=result.returncode==0
|
||||
if not all(self_check.values()):raise RuntimeError(f"Phase4N self-checksum failed {self_check}")
|
||||
print(json.dumps({"status":summary["status"],"peak":peak,"onset":onset,"B_contamination":contam,
|
||||
"DAN":dan,"wholebrain_penalty":sep["at_peak"],"integrity":integ,"self_checks":self_check}),flush=True)
|
||||
|
||||
|
||||
if __name__=="__main__":main()
|
||||
@@ -0,0 +1,254 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4N checkpoint-specific whole-brain/credit audit and conservative sealing."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gc
|
||||
import json
|
||||
import resource
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import torch
|
||||
|
||||
from phase4h_audit import sha256, verify_checksum, write_json
|
||||
from phase4i import Validator, phase4c_regression
|
||||
from phase4j import effect
|
||||
from phase4k import dataset_checks, seal
|
||||
from phase4l import OdorSimulator, pn_drive
|
||||
from phase4m import frozen_checks, frozen_hashes, load_odor_inputs
|
||||
from phase4n import GROUPS, load_config
|
||||
|
||||
|
||||
def state(exp, seed, pair):
|
||||
if pair == 0:
|
||||
return np.ones(14232, np.float32)
|
||||
with np.load(exp / "acquisition-trajectory" / f"E-seed-{seed}-pairings-{pair}.npz") as f:
|
||||
return f["E_bounded"].copy()
|
||||
|
||||
|
||||
def curves(ncfg, exp, csi, replay, identity):
|
||||
csi = csi[csi.mechanism == "bounded"]
|
||||
peak = json.loads((exp / "specificity-curve/peak.json").read_text())["pairings"]
|
||||
onset = None
|
||||
if peak is not None:
|
||||
p = csi[csi.pairings == peak]
|
||||
baseline_csi = p.CSI.mean()
|
||||
baseline_other = max(p.shared.mean(), p["B-specific"].mean())
|
||||
for pair in ncfg["checkpoints"]:
|
||||
if pair <= peak:
|
||||
continue
|
||||
q = csi[csi.pairings == pair]
|
||||
other = max(q.shared.mean(), q["B-specific"].mean())
|
||||
if q.CSI.mean() <= .8 * baseline_csi and other >= 1.1 * baseline_other and baseline_csi > 0 and baseline_other > 0:
|
||||
onset = pair
|
||||
break
|
||||
write_json(exp / "specificity-curve/generalization-onset.json", {"status": "GENERALIZATION-ONSET-IDENTIFIED" if onset is not None else "UNIDENTIFIED",
|
||||
"pairings": onset, "rule": ncfg["generalization_onset_rule"]})
|
||||
groups = pd.read_parquet(exp / "acquisition-trajectory/group-plasticity.parquet")
|
||||
groups = groups[groups.mechanism == "bounded"]
|
||||
# Dependency-free vector plot; values remain in the accompanying Parquet files.
|
||||
parts = ['<svg xmlns="http://www.w3.org/2000/svg" width="900" height="640" viewBox="0 0 900 640">',
|
||||
'<rect width="900" height="640" fill="white"/>', '<text x="55" y="30" font-size="17">Phase 4N acquisition and specificity</text>']
|
||||
def panel(series, y0, height, title):
|
||||
allv = np.concatenate([np.asarray(v, np.float64) for _, v, _ in series]); lo, hi = float(np.min(allv)), float(np.max(allv))
|
||||
if hi <= lo: hi = lo + 1.0
|
||||
parts.append(f'<text x="55" y="{y0-10}" font-size="13">{title}</text>')
|
||||
parts.append(f'<path d="M60 {y0} V{y0+height} H850" fill="none" stroke="black"/>')
|
||||
for label, vals, color in series:
|
||||
xs = np.asarray(ncfg["checkpoints"], np.float64)
|
||||
ys = np.asarray(vals, np.float64)
|
||||
coords = ' '.join(f'{60+790*x/100:.1f},{y0+height-20-(height-35)*(y-lo)/(hi-lo):.1f}' for x,y in zip(xs,ys))
|
||||
parts.append(f'<polyline points="{coords}" fill="none" stroke="{color}" stroke-width="2"/>')
|
||||
parts.append(f'<text x="{625}" y="{y0+18+17*len([s for s in series if s[0] < label])}" font-size="11" fill="{color}">{label}</text>')
|
||||
for p,color in ((peak,"green"),(onset,"red")):
|
||||
if p is not None:
|
||||
x = 60 + 790*p/100
|
||||
parts.append(f'<path d="M{x:.1f} {y0} V{y0+height}" stroke="{color}" stroke-dasharray="4,4"/>')
|
||||
first = []
|
||||
for name,color in zip(GROUPS,("#1261a0","#d4871e","#b72230","#777777")):
|
||||
g = groups[groups.group == name].groupby("pairings").weighted_transmission_reduction.mean().reindex(ncfg["checkpoints"]).fillna(0.0)
|
||||
first.append((name,g.to_numpy(),color))
|
||||
panel(first, 80, 210, "Mean anatomical-weighted E reduction")
|
||||
panel([("A-B spike L1",replay.mean_spike_L1_separation.to_numpy(),"#2059a6"),
|
||||
("A-B latency ms",replay.mean_latency_separation_ms.to_numpy(),"#b72230")], 370, 210, "Functional MBON effect difference")
|
||||
parts.append('</svg>')
|
||||
(exp / "specificity-curve/trajectory.svg").write_text('\n'.join(parts), encoding="utf-8")
|
||||
return peak, onset
|
||||
|
||||
|
||||
def shared_decomposition(ncfg, exp, identity, sim, v, peak):
|
||||
if peak is None:
|
||||
write_json(exp / "shared-edge/summary.json", {"status": "NOT-RUN-NO-SPECIFICITY-PEAK"})
|
||||
return None
|
||||
from phase4m import ReplayEngine, replay_effect
|
||||
comp = identity.compartment.astype(str).to_numpy() == str(ncfg["matched_compartment_verbatim"])
|
||||
targets = np.unique(identity.loc[comp, "target_neuron_idx"].to_numpy(np.int32))
|
||||
engine = ReplayEngine(sim, targets)
|
||||
w = identity.anatomical_weight_mv.to_numpy(np.float32)
|
||||
groups = identity.identity_group.astype(str).to_numpy()
|
||||
rows = []
|
||||
from phase4n import load_config
|
||||
_, mcfg, _, _ = load_config(Path(__file__).with_name("phase4n.yaml"))
|
||||
for seed, es in zip(ncfg["train_seeds"], ncfg["evaluation_seeds"]):
|
||||
E = state(exp, seed, peak)
|
||||
for odor in ("A", "B"):
|
||||
spikes = pd.read_parquet(Path(mcfg["experiment_root"]) / "isolated-replay" / f"KC-spikes-ODOR_{odor}-seed-{es}.parquet")
|
||||
baseline = engine.run(spikes, np.zeros(len(E), np.float32), float(mcfg["protocol"]["duration_ms"]))
|
||||
for label, included in (("A-only", ("A-specific",)), ("A+shared", ("A-specific", "shared")),
|
||||
("A+shared+B", ("A-specific", "shared", "B-specific")), ("all-updated", GROUPS)):
|
||||
delta = np.where(np.isin(groups, included), w * (E - 1), 0).astype(np.float32)
|
||||
post = engine.run(spikes, delta, float(mcfg["protocol"]["duration_ms"]))
|
||||
metrics = replay_effect(baseline, post, sim.dt)
|
||||
rows.append({"train_seed": seed, "eval_seed": es, "pairings": peak, "odor": odor, "decomposition": label,
|
||||
**{k: val for k, val in metrics.items() if k != "isolated_input_scope"}})
|
||||
pd.DataFrame(rows).to_parquet(exp / "shared-edge/isolated-decomposition.parquet", index=False)
|
||||
return pd.DataFrame(rows)
|
||||
|
||||
|
||||
def wholebrain(ncfg, mcfg, cfg, exp, identity, sim, v, peak):
|
||||
wanted = sorted(set([10, 20, 50, 75, 100] + ([peak] if peak is not None else [])))
|
||||
panel, vectors, selection, mapping = load_odor_inputs(cfg)
|
||||
gloms = selection["glomeruli_order"]
|
||||
matched = float(np.median(panel.mapped_response_sum))
|
||||
pn = {}
|
||||
for label in ("A", "B"):
|
||||
oid = selection[f"ODOR_{label}"]["odor_id"]
|
||||
pn[label] = pn_drive(oid, vectors[oid], mapping, gloms, str(mcfg["protocol"]["input_mode"]), matched, cfg)[:2]
|
||||
dan = np.unique(v.confirmed.source_neuron_idx.to_numpy(np.int32))
|
||||
group_sources = {name: np.unique(identity.loc[identity.identity_group == name, "source_neuron_idx"].to_numpy(np.int32)) for name in GROUPS}
|
||||
w = identity.anatomical_weight_mv.to_numpy(np.float32)
|
||||
rows, calls = [], []
|
||||
for seed, es in zip(ncfg["train_seeds"], ncfg["evaluation_seeds"]):
|
||||
baseline = {label: sim.run_rates(*pn[label], es, float(mcfg["protocol"]["duration_ms"]), collect_kc_spikes=True) for label in pn}
|
||||
for pair in wanted:
|
||||
delta = w * (state(exp, seed, pair) - 1)
|
||||
for label in ("A", "B"):
|
||||
post = sim.run_rates(*pn[label], es, float(mcfg["protocol"]["duration_ms"]), delta=delta, collect_kc_spikes=True)
|
||||
calls.append(post["wall_seconds"])
|
||||
eff = effect(baseline[label], post)
|
||||
row = {"train_seed": seed, "eval_seed": es, "pairings": pair, "odor": label,
|
||||
"MBON_spike_L1": eff["MBON_spike_l1"], "MBON_g_max_abs_mv": eff["MBON_g_max_abs"],
|
||||
"MBON_v_max_abs_mv": eff["MBON_v_max_abs"], "MBON_latency_delta_ms": eff["MBON_first_latency_delta_ms"],
|
||||
"downstream_spike_L1": eff["downstream_spike_l1"], "KC_spike_L1": int(np.abs(post["kc_counts"]-baseline[label]["kc_counts"]).sum()),
|
||||
"endogenous_DAN_spikes": int(post["counts"][dan].sum()), "baseline_endogenous_DAN_spikes": int(baseline[label]["counts"][dan].sum()),
|
||||
"finite": bool(post["finite"]), "peak_vram_bytes": int(post["peak_vram_bytes"]), "wall_seconds": post["wall_seconds"]}
|
||||
for group, nodes in group_sources.items():
|
||||
row[f"{group}_KC_spikes"] = int(post["counts"][nodes].sum())
|
||||
row[f"{group}_KC_newly_active_vs_E1"] = int(np.count_nonzero((post["counts"][nodes] > 0) & (baseline[label]["counts"][nodes] == 0)))
|
||||
row["MBON_spikes"] = int(post["mbon_counts"].sum())
|
||||
row["other_feedback_spikes"] = int(post["downstream_counts"].sum())
|
||||
rows.append(row)
|
||||
frame = pd.DataFrame(rows)
|
||||
frame.to_parquet(exp / "whole-brain/checkpoint-responses.parquet", index=False)
|
||||
write_json(exp / "performance/whole-brain.json", {"calls": len(calls), "mean_wall_seconds": float(np.mean(calls)),
|
||||
"sum_wall_seconds": float(np.sum(calls)), "max_peak_vram_bytes": int(frame.peak_vram_bytes.max())})
|
||||
return frame
|
||||
|
||||
|
||||
def audit_updates(ncfg, mcfg, exp, identity, history, csi, peak):
|
||||
group = identity.set_index("edge_id").identity_group
|
||||
hist = history.copy()
|
||||
hist["identity_group"] = group.loc[hist.edge_id.to_numpy(np.int64)].to_numpy()
|
||||
hist["delta_E_bounded"] = hist.E_bounded_after - hist.E_bounded_before
|
||||
hist["DAN_source"] = "external-US-associated"
|
||||
hist["matched_compartment"] = True
|
||||
hist["endogenous_DAN_used_as_gate"] = False
|
||||
hist["local_KC_trace_nonzero"] = hist.eligibility > 0
|
||||
hist.to_parquet(exp / "local-eligibility/phase4m-updates-classified.parquet", index=False)
|
||||
first = hist[(hist.identity_group == "B-specific") & (hist.delta_E_bounded < -1e-7)].groupby("train_seed").pairing.min().reset_index(name="first_B_specific_update_pairing")
|
||||
first.to_parquet(exp / "b-contamination/first-B-updates.parquet", index=False)
|
||||
causes = {"direct_A_driven_KC_activity": None, "recurrent_KC_reactivation": None, "MBON_DAN_feedback": None,
|
||||
"olfactory_feedback": None, "long_eligibility_trace": None, "endogenous_DAN_teaching_interpretation": 0.0,
|
||||
"unresolved_runtime_KC_recruitment_or_trace": 1.0}
|
||||
write_json(exp / "b-contamination/provenance.json", {"classification": causes,
|
||||
"reason": "Frozen Phase 4M update history records trace at forced matched-US but no event-level causal predecessor or KC spike times. Assigning a recurrent/feedback percentage would be invented.",
|
||||
"earliest_B_update_by_seed": first.to_dict("records"), "all_updates_gated_by_external_matched_US": True})
|
||||
write_json(exp / "dan-source/phase4m-gate-audit.json", {"external_US_gates": int(history[["train_seed", "pairing"]].drop_duplicates().shape[0]),
|
||||
"endogenous_DAN_as_teaching_gate": 0, "policy": "Phase 4M implementation opens plasticity once at imposed 90 ms US, not at endogenous DAN spike times",
|
||||
"observed_endogenous_DAN_activity_is_not_equivalent_to_dopamine_concentration": True})
|
||||
score = csi[csi.mechanism == "bounded"].groupby("pairings").CSI.mean()
|
||||
write_json(exp / "stopping-analysis/summary.json", {"status": "POST-HOC-ONLY-NO-ADAPTIVE-POLICY",
|
||||
"peak_CSI_checkpoint": int(score.idxmax()), "peak_CSI": float(score.max()), "functional_peak_checkpoint": peak,
|
||||
"online_validated_criterion": False, "heldout_training_validation_required": True})
|
||||
|
||||
|
||||
def controls(exp, identity, ncfg, mcfg):
|
||||
rows = []
|
||||
for seed in ncfg["train_seeds"]:
|
||||
for pair in ncfg["checkpoints"]:
|
||||
E = state(exp, seed, pair)
|
||||
rows += [{"seed": seed, "pairings": pair, "condition": "A+matched-US", "E_changed": int(np.count_nonzero(E < 1 - 1e-7)), "status": "RUN-RECONSTRUCTED-FROZEN-PHASE4M"},
|
||||
{"seed": seed, "pairings": pair, "condition": "A+D0", "E_changed": 0, "status": "EXACT-ZERO-GATE-IDENTITY"},
|
||||
{"seed": seed, "pairings": pair, "condition": "frozen-E1", "E_changed": 0, "status": "EXACT-FROZEN-IDENTITY"},
|
||||
{"seed": seed, "pairings": pair, "condition": "external-US-only-gate", "E_changed": int(np.count_nonzero(E < 1 - 1e-7)), "status": "IDENTICAL-TO-FROZEN-PHASE4M-RULE"}]
|
||||
pd.DataFrame(rows).to_parquet(exp / "controls/trajectory-status.parquet", index=False)
|
||||
report = {"A_matched_US": "RUN-PHASE4M-TRAJECTORY-RECONSTRUCTED", "A_D0": "ANALYTIC-EXACT-IDENTITY", "frozen_E1": "ANALYTIC-EXACT-IDENTITY",
|
||||
"external_US_only": "RUN-IDENTICAL-TO-PHASE4M", "B_matched_US": "NOT-RUN", "random_US_timing": "NOT-RUN",
|
||||
"unrelated_compartment_US": "NOT-RUN", "all_DAN_gate": "NOT-RUN", "full_control_matrix_pass": False}
|
||||
write_json(exp / "controls/summary.json", report)
|
||||
return report
|
||||
|
||||
|
||||
def integrity(ncfg, mcfg, cfg, base_path, exp, frozen_before):
|
||||
gc.collect(); torch.cuda.empty_cache()
|
||||
checks = frozen_checks(cfg)
|
||||
mroot, mm = Path(mcfg["experiment_root"]), Path(mcfg["model_root"])
|
||||
mchecks = {"phase4m_exp": verify_checksum(mroot, mroot / "manifests/checksums.sha256")["passed"],
|
||||
"phase4m_model": verify_checksum(mm, mm / "checksums.sha256")["passed"]}
|
||||
dataset = dataset_checks(cfg)
|
||||
regression = phase4c_regression(cfg, base_path, exp)
|
||||
after = frozen_hashes(cfg)
|
||||
report = {"all_passed": bool(checks["all_passed"] and all(mchecks.values()) and dataset["all_passed"] and regression["all_passed"] and frozen_before == after),
|
||||
"phase4c_to_4l": checks, "phase4m": mchecks, "phase1_to_3b": dataset, "phase4c_reference_regression": regression,
|
||||
"before_after_identical": frozen_before == after, "base_csr_unmodified": True, "APL_v2_encoder_mask_unmodified": True}
|
||||
write_json(exp / "audit/frozen-integrity.json", report)
|
||||
return report
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", type=Path, required=True); args = ap.parse_args()
|
||||
ncfg, mcfg, cfg, base_path = load_config(args.config)
|
||||
exp = Path(ncfg["experiment_root"])
|
||||
identity = pd.read_parquet(exp / "edge-identity/frozen-edge-identity.parquet")
|
||||
csi = pd.read_parquet(exp / "specificity-curve/CSI.parquet")
|
||||
replay = pd.read_parquet(exp / "specificity-curve/memory-specificity.parquet")
|
||||
history = pd.read_parquet(Path(mcfg["experiment_root"]) / "efficacy-overlay/update-history.parquet")
|
||||
before = frozen_hashes(cfg)
|
||||
peak, onset = curves(ncfg, exp, csi, replay, identity)
|
||||
print(json.dumps({"stage": "specificity", "peak": peak, "generalization_onset": onset}), flush=True)
|
||||
v = Validator(cfg, base_path, exp); sim = OdorSimulator(v, cfg, exp)
|
||||
shared = shared_decomposition(ncfg, exp, identity, sim, v, peak)
|
||||
print(json.dumps({"stage": "shared_decomposition", "rows": 0 if shared is None else len(shared)}), flush=True)
|
||||
whole = wholebrain(ncfg, mcfg, cfg, exp, identity, sim, v, peak)
|
||||
print(json.dumps({"stage": "wholebrain", "rows": len(whole)}), flush=True)
|
||||
audit_updates(ncfg, mcfg, exp, identity, history, csi, peak)
|
||||
control = controls(exp, identity, ncfg, mcfg)
|
||||
del sim, v
|
||||
integ = integrity(ncfg, mcfg, cfg, base_path, exp, before)
|
||||
ru = resource.getrusage(resource.RUSAGE_SELF)
|
||||
perf = {"peak_RAM_KiB": int(ru.ru_maxrss), "edge_audit_rows": int(len(history)),
|
||||
"edge_audit_memory_bytes": int(history.memory_usage(deep=True).sum()), "shadow_trace_overhead": "NOT-MEASURED-NOT-RUN",
|
||||
"wholebrain": json.loads((exp / "performance/whole-brain.json").read_text()),
|
||||
"isolated": json.loads((exp / "performance/isolated.json").read_text())}
|
||||
write_json(exp / "performance/summary.json", perf)
|
||||
summary = {"phase": "4N", "status": "PARTIAL-MECHANISM-AUDIT-NO-CANDIDATE", "peak": peak, "generalization_onset": onset,
|
||||
"edge_groups": identity.identity_group.value_counts().to_dict(), "controls": control, "frozen_integrity": integ["all_passed"],
|
||||
"candidate_generated": False, "limitations": ["Endogenous DAN event-level gate overlay not run", "Full matched control trajectories not run", "Runtime causal predecessor and local shadow traces not captured; contamination source percentages unresolved"]}
|
||||
write_json(exp / "audit/final-summary.json", summary)
|
||||
model = Path(ncfg["model_root"]); model.mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(args.config, model / "phase4n.yaml"); shutil.copy2(Path(__file__).resolve(), model / "phase4n_finish.py")
|
||||
shutil.copy2(Path(__file__).with_name("phase4n.py"), model / "phase4n.py")
|
||||
shutil.copy2(Path(__file__).resolve(), exp / "audit/phase4n_finish.py")
|
||||
write_json(model / "VERSION.json", {"phase": "4N", "status": summary["status"], "candidate_generated": False,
|
||||
"identity_sha256": sha256(exp / "edge-identity/frozen-edge-identity.parquet"), "config_sha256": sha256(args.config)})
|
||||
seal(model, model / "MANIFEST.json", model / "checksums.sha256")
|
||||
seal(exp, exp / "manifests/manifest.json", exp / "manifests/checksums.sha256")
|
||||
print(json.dumps(summary), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,103 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Timing-compatible incoming KC drive paths; not unique causal-spike proofs."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
from phase4h_audit import write_json
|
||||
from phase4n import load_config
|
||||
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument("--config",type=Path,required=True);args=ap.parse_args()
|
||||
ncfg,mcfg,cfg,_=load_config(args.config);exp=Path(ncfg["experiment_root"])
|
||||
identity=pd.read_parquet(exp/"edge-identity/frozen-edge-identity.parquet")
|
||||
bnodes=np.unique(identity.loc[identity.identity_group=="B-specific","source_neuron_idx"].to_numpy(np.int32))
|
||||
root=Path(cfg["paths"]["connectome_root"])
|
||||
indptr=np.load(root/"gpu/csr-indptr.int64.npy",mmap_mode="r")
|
||||
indices=np.load(root/"gpu/csr-indices.int32.npy",mmap_mode="r")
|
||||
raw=np.load(root/"gpu/csr-weights.int64.npy",mmap_mode="r")
|
||||
signs=np.load("/data/digital-drosophila/experiments/phase4a-baseline/outputs/derived/synaptic-sign.int8.npy",mmap_mode="r")
|
||||
meta=pd.read_parquet(root/"neuron-index/unified-metadata.parquet",columns=["neuron_idx","body_id","annotation_class","annotation_type"])
|
||||
meta=meta.sort_values("neuron_idx").reset_index(drop=True)
|
||||
if len(signs)!=len(meta) or len(indptr)!=len(meta)+1:raise RuntimeError("frozen graph/schema mismatch")
|
||||
dan=pd.read_parquet(Path(cfg["paths"]["phase4h_root"])/"mappings/dan-to-mbon-confirmed.parquet",columns=["source_neuron_idx"])
|
||||
danset=set(map(int,dan.source_neuron_idx.unique()))
|
||||
edge_ids=np.flatnonzero(np.isin(np.asarray(indices),bnodes)).astype(np.int64)
|
||||
targets=np.asarray(indices[edge_ids],np.int32)
|
||||
sources=(np.searchsorted(indptr,edge_ids,side="right")-1).astype(np.int32)
|
||||
weights=np.asarray(raw[edge_ids],np.float32)*np.asarray(signs[sources],np.float32)*np.float32(cfg["model"]["w_syn_mv_per_anatomical_synapse"])
|
||||
by_target={int(n):np.flatnonzero(targets==n) for n in bnodes}
|
||||
classes=meta.annotation_class.fillna("").astype(str).to_numpy()
|
||||
def source_class(n):
|
||||
cls=classes[n]
|
||||
if cls in ("ALPN","olfactory"):return "PN/olfactory"
|
||||
if cls=="Kenyon_Cell":return "KC recurrent"
|
||||
if cls=="MBON":return "MBON feedback"
|
||||
if n in danset:return "DAN"
|
||||
if "APL" in str(meta.annotation_type.iat[n]):return "APL anatomical"
|
||||
return "other CNS"
|
||||
rows=[];pairings=[10,20,50,60,75,100]
|
||||
dt=float(cfg["model"]["dt_ms"]);delay=int(round(float(cfg["model"]["synaptic_delay_ms"])/dt))
|
||||
tau=float(cfg["model"]["synaptic_tau_ms"]);window=int(round(5*tau/dt))
|
||||
for seed in ncfg["train_seeds"]:
|
||||
for pair in pairings:
|
||||
file=exp/"whole-brain"/f"spike-events-seed-{seed}-pairings-{pair}.parquet"
|
||||
events=pd.read_parquet(file)
|
||||
steps_by_node={int(n):g.step.to_numpy(np.int32) for n,g in events.groupby("neuron_idx")}
|
||||
for target in bnodes:
|
||||
times=steps_by_node.get(int(target))
|
||||
if times is None or not len(times):continue
|
||||
t=int(times[0])
|
||||
if t>int(round(float(mcfg["protocol"]["dan_time_ms"])/dt)):continue
|
||||
cat_totals=defaultdict(float);rank=[]
|
||||
for position in by_target[int(target)]:
|
||||
src=int(sources[position]);w=float(weights[position]);
|
||||
if w<=0:continue
|
||||
source_steps=steps_by_node.get(src)
|
||||
if source_steps is None:continue
|
||||
arrived=source_steps+delay
|
||||
active=arrived[(arrived<=t)&(arrived>=t-window)]
|
||||
if not len(active):continue
|
||||
contribution=float(np.sum(w*np.exp(-(t-active)*dt/tau)))
|
||||
category=source_class(src);cat_totals[category]+=contribution
|
||||
latest=int(active[-1])
|
||||
rank.append((contribution,int(edge_ids[position]),src,latest,category,w))
|
||||
rank.sort(reverse=True)
|
||||
top=rank[0] if rank else None
|
||||
rows.append({"seed":seed,"pairing":pair,"B_specific_KC_neuron_idx":int(target),"B_specific_KC_body_id":int(meta.body_id.iat[target]),
|
||||
"first_activation_step":t,"first_activation_ms":t*dt,"candidate_predecessor_neuron_idx":None if top is None else top[2],
|
||||
"candidate_predecessor_body_id":None if top is None else int(meta.body_id.iat[top[2]]),
|
||||
"actual_edge_id":None if top is None else top[1],"arrival_ms":None if top is None else top[3]*dt,
|
||||
"candidate_predecessor_population":None if top is None else top[4],
|
||||
"candidate_predecessor_type":None if top is None else str(meta.annotation_type.iat[top[2]]),
|
||||
"anatomical_effective_weight_mv":None if top is None else top[5],
|
||||
"decayed_positive_input_at_spike_mv":None if top is None else top[0],
|
||||
"positive_input_by_source_class_json":json.dumps(dict(cat_totals),sort_keys=True),
|
||||
"APL_graded_inhibition_accounted":False,"unique_causal_predecessor_proven":False,
|
||||
"reason":"Multiple delayed inputs and APL graded inhibition converge; strongest positive timed edge is not a counterfactual causal proof."})
|
||||
table=pd.DataFrame(rows);table.to_parquet(exp/"whole-brain/B-specific-timed-input-candidates.parquet",index=False)
|
||||
updates=pd.read_parquet(exp/"local-eligibility/event-level-updates.parquet")
|
||||
updates=updates[(updates.identity_group=="B-specific")&updates.pairing.isin(pairings)].copy()
|
||||
linked=updates.merge(table,left_on=["seed","pairing","source_neuron_idx"],
|
||||
right_on=["seed","pairing","B_specific_KC_neuron_idx"],how="left",validate="many_to_one")
|
||||
linked.to_parquet(exp/"whole-brain/B-specific-credit-timed-input.parquet",index=False)
|
||||
distribution=table.candidate_predecessor_population.fillna("none").value_counts().to_dict() if len(table) else {}
|
||||
write_json(exp/"whole-brain/B-specific-timed-input-summary.json",{"audited_B_specific_KC":int(len(bnodes)),
|
||||
"rows":int(len(table)),"source_class_of_strongest_timing_compatible_positive_input":distribution,
|
||||
"linked_B_specific_plastic_update_rows":int(len(linked)),
|
||||
"linked_updates_with_timed_input":int(linked.actual_edge_id.notna().sum()),
|
||||
"causal_predecessor_unambiguously_identified":False,
|
||||
"APL_graded_inhibition_in_source_decomposition":False,
|
||||
"five_synaptic_tau_lookback_ms":5*tau,"exact_frozen_edge_and_delay_used":True,
|
||||
"not_interpretable_as_causal_source_percentages":True})
|
||||
print(json.dumps({"B_nodes":len(bnodes),"rows":len(table),"timed_positive_source_distribution":distribution}),flush=True)
|
||||
|
||||
|
||||
if __name__=="__main__":main()
|
||||
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4N early consistency check for frozen event and DAN audit tables."""
|
||||
from pathlib import Path
|
||||
import json
|
||||
import pandas as pd
|
||||
from phase4n import load_config
|
||||
from phase4n_finalize import contamination, dan_summary, shared_summary, separation
|
||||
|
||||
ncfg, mcfg, cfg, _ = load_config(Path(__file__).with_name("phase4n.yaml"))
|
||||
exp = Path(ncfg["experiment_root"])
|
||||
identity = pd.read_parquet(exp / "edge-identity/frozen-edge-identity.parquet")
|
||||
history = pd.read_parquet(Path(mcfg["experiment_root"]) / "efficacy-overlay/update-history.parquet")
|
||||
print(json.dumps({"contamination": contamination(exp, history), "DAN": dan_summary(exp, identity, ncfg),
|
||||
"shared": shared_summary(exp), "wholebrain": separation(exp, 50)}))
|
||||
@@ -0,0 +1,6 @@
|
||||
6e51c6df2ab7b41a97633ae895a2f02277bba638b6fa11919cac85e054539449 __pycache__/phase4q.cpython-310.pyc
|
||||
2cd10fb215d7ed624f4a4323453d20c4327bea490fa52eb3afc5d7fa0176a6e5 phase4q.py
|
||||
2cd10fb215d7ed624f4a4323453d20c4327bea490fa52eb3afc5d7fa0176a6e5 phase4q.py.frozen
|
||||
2cf3f799a5c1d24c20ed7aa1ff920daffa792b189002d0d7dcee1e51a4d49bd7 phase4q.yaml
|
||||
2cf3f799a5c1d24c20ed7aa1ff920daffa792b189002d0d7dcee1e51a4d49bd7 phase4q.yaml.frozen
|
||||
d2bb488498663c367ac1620c238e4fe83573358ce3bdc132718fd85510cc6da2 phase4q_seal.py
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"files": [
|
||||
{
|
||||
"path": "__pycache__/phase4q.cpython-310.pyc",
|
||||
"sha256": "6e51c6df2ab7b41a97633ae895a2f02277bba638b6fa11919cac85e054539449",
|
||||
"size": 32322
|
||||
},
|
||||
{
|
||||
"path": "phase4q.py",
|
||||
"sha256": "2cd10fb215d7ed624f4a4323453d20c4327bea490fa52eb3afc5d7fa0176a6e5",
|
||||
"size": 38764
|
||||
},
|
||||
{
|
||||
"path": "phase4q.py.frozen",
|
||||
"sha256": "2cd10fb215d7ed624f4a4323453d20c4327bea490fa52eb3afc5d7fa0176a6e5",
|
||||
"size": 38764
|
||||
},
|
||||
{
|
||||
"path": "phase4q.yaml",
|
||||
"sha256": "2cf3f799a5c1d24c20ed7aa1ff920daffa792b189002d0d7dcee1e51a4d49bd7",
|
||||
"size": 2607
|
||||
},
|
||||
{
|
||||
"path": "phase4q.yaml.frozen",
|
||||
"sha256": "2cf3f799a5c1d24c20ed7aa1ff920daffa792b189002d0d7dcee1e51a4d49bd7",
|
||||
"size": 2607
|
||||
},
|
||||
{
|
||||
"path": "phase4q_seal.py",
|
||||
"sha256": "d2bb488498663c367ac1620c238e4fe83573358ce3bdc132718fd85510cc6da2",
|
||||
"size": 740
|
||||
}
|
||||
],
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/credit-v3"
|
||||
}
|
||||
@@ -0,0 +1,484 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4Q: local terminal credit assignment v3 on deterministic-runtime-v2."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import resource
|
||||
import shutil
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow as pa
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
import phase4p
|
||||
import phase4p_finalize
|
||||
from phase4h_audit import verify_checksum
|
||||
from phase4p_revalidate import DeterministicAPLSim, IsolatedReplay, door_inputs, make_drive
|
||||
|
||||
|
||||
ROOT = Path("/data/digital-drosophila")
|
||||
|
||||
|
||||
def save(path: Path, x: object) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(x, indent=2, sort_keys=True, default=str) + "\n")
|
||||
|
||||
|
||||
def sha(*arrays: np.ndarray) -> str:
|
||||
h = hashlib.sha256()
|
||||
for x in arrays: h.update(np.ascontiguousarray(x).tobytes())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def dirs(exp: Path) -> None:
|
||||
for name in ("calibration", "timing", "eligibility-v3", "unit-tests", "edge-credit", "random-us",
|
||||
"distant-us", "isolated", "shared-edge", "b-contamination", "whole-brain", "controls",
|
||||
"performance", "regression", "audit", "manifests"):
|
||||
(exp / name).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def load(cfgpath: Path):
|
||||
cfg = yaml.safe_load(cfgpath.read_text()); exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
dirs(exp); model.mkdir(parents=True, exist_ok=True)
|
||||
pcfg = yaml.safe_load((Path(cfg["paths"]["deterministic_runtime"]) / "phase4p.yaml").read_text())
|
||||
rt = phase4p.CanonicalRuntime(pcfg, Path(cfg["paths"]["deterministic_runtime"]), exp)
|
||||
sim = DeterministicAPLSim(rt)
|
||||
identity = pd.read_parquet(cfg["paths"]["edge_identity"]).sort_values("edge_id").reset_index(drop=True)
|
||||
expected = cfg["frozen_update"]["groups"]
|
||||
actual = identity.identity_group.value_counts().to_dict()
|
||||
if any(int(actual.get(k, 0)) != int(v) for k, v in expected.items()): raise RuntimeError(f"frozen group mismatch {actual}")
|
||||
if not np.array_equal(identity.edge_id.to_numpy(np.int64), sim.identity.sort_values("edge_id").edge_id.to_numpy(np.int64)):
|
||||
raise RuntimeError("plastic edge identity mismatch")
|
||||
# sim identity order already follows frozen parquet; preserve it for E arrays.
|
||||
identity = sim.identity.copy().reset_index(drop=True)
|
||||
return cfg, exp, model, pcfg, rt, sim, identity
|
||||
|
||||
|
||||
def random_us_steps(base: int, seed: int, count: int, nsteps: int) -> np.ndarray:
|
||||
rng = np.random.default_rng(base + seed)
|
||||
return rng.integers(0, nsteps, size=count, dtype=np.int32)
|
||||
|
||||
|
||||
def last_spike_trace(steps: np.ndarray, nodes: np.ndarray, kc_lookup: np.ndarray, us_step: int,
|
||||
window_ms: float, tau_ms: float, dt_ms: float) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Causal v3: only the last source-terminal spike at/before the US counts."""
|
||||
last = np.full(int(kc_lookup.max()) + 1, -1, np.int32)
|
||||
valid = steps <= us_step
|
||||
for step, node in zip(steps[valid], nodes[valid]):
|
||||
loc = kc_lookup[int(node)]
|
||||
if loc >= 0: last[loc] = int(step)
|
||||
age_ms = np.where(last >= 0, (us_step - last).astype(np.float32) * np.float32(dt_ms), np.float32(np.inf))
|
||||
trace = np.zeros(len(last), np.float32); ok = (age_ms >= 0) & (age_ms <= np.float32(window_ms))
|
||||
trace[ok] = np.exp(-age_ms[ok] / np.float32(tau_ms)).astype(np.float32)
|
||||
return trace, last
|
||||
|
||||
|
||||
def v2_trace(steps: np.ndarray, nodes: np.ndarray, kc_lookup: np.ndarray, us_step: int, dt_ms: float) -> np.ndarray:
|
||||
trace = np.zeros(int(kc_lookup.max()) + 1, np.float32); decay = np.float32(math.exp(-dt_ms / 100.0))
|
||||
by = {}
|
||||
valid = steps <= us_step
|
||||
for step in np.unique(steps[valid]): by[int(step)] = kc_lookup[nodes[(steps == step) & valid]]
|
||||
for step in range(us_step + 1):
|
||||
trace = np.float32(trace * decay); active = by.get(step)
|
||||
if active is not None and len(active): trace[active] = np.minimum(np.float32(1), np.float32(trace[active] + np.float32(1)))
|
||||
return trace
|
||||
|
||||
|
||||
def edge_elig(trace: np.ndarray, source_local: np.ndarray, compartment_mask: np.ndarray, teaching: bool = True) -> np.ndarray:
|
||||
out = np.zeros(len(source_local), np.float32)
|
||||
if teaching: out[compartment_mask] = trace[source_local[compartment_mask]]
|
||||
return out
|
||||
|
||||
|
||||
def update_E(E: np.ndarray, elig: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
|
||||
before = E.copy(); active = elig > 0
|
||||
E[active] = np.maximum(np.float32(0.75), np.float32(E[active] * np.float32(1.0 - np.float32(0.01) * elig[active])))
|
||||
return E, np.float32(E - before)
|
||||
|
||||
|
||||
def group_metrics(E: np.ndarray, base: np.ndarray, identity: pd.DataFrame) -> list[dict[str, Any]]:
|
||||
rows = []
|
||||
for group in ("A-specific", "shared", "B-specific", "inactive"):
|
||||
mask = identity.identity_group.astype(str).to_numpy() == group; d = np.float32(E[mask] - 1)
|
||||
rows.append({"group": group, "edges": int(mask.sum()), "updated_edges": int(np.count_nonzero(d)),
|
||||
"mean_delta_E": float(np.mean(d)), "total_abs_delta_E": float(np.abs(d).sum()),
|
||||
"total_weighted_delta_mv": float(np.sum(base[mask].astype(np.float64) * d.astype(np.float64)))})
|
||||
return rows
|
||||
|
||||
|
||||
def formal_odor_ids(panel: pd.DataFrame, selection: dict[str, Any]) -> tuple[str, str]:
|
||||
return str(selection["ODOR_A"]["odor_id"]), str(selection["ODOR_B"]["odor_id"])
|
||||
|
||||
|
||||
def unit_tests(cfg, exp, sim, identity, window, tau, source_local, comp_mask, write=True) -> dict[str, Any]:
|
||||
dt = sim.dt; kc0 = int(sim.kc[0]); lookup = np.full(sim.n, -1, np.int32); lookup[sim.kc] = np.arange(len(sim.kc), dtype=np.int32)
|
||||
us = 900; recent_step = us - int(round(min(window / 2, 2.0) / dt)); far = us - int(round((window + 50) / dt))
|
||||
def tr(ss, nn, gate=True, w=window, t=tau, u=us):
|
||||
x, _ = last_spike_trace(np.asarray(ss, np.int32), np.asarray(nn, np.int32), lookup, u, w, t, dt)
|
||||
return edge_elig(x, source_local, comp_mask, gate)
|
||||
active_edges = comp_mask & (identity.source_neuron_idx.to_numpy(np.int32) == kc0)
|
||||
if not active_edges.any():
|
||||
kc0 = int(identity.loc[comp_mask, "source_neuron_idx"].iloc[0]); recent_step = us - 10
|
||||
active_edges = comp_mask & (identity.source_neuron_idx.to_numpy(np.int32) == kc0)
|
||||
recent = tr([recent_step], [kc0]); inactive = tr([], []); dzero = tr([recent_step], [kc0], False)
|
||||
unrelated = edge_elig(last_spike_trace(np.asarray([recent_step],np.int32),np.asarray([kc0],np.int32),lookup,us,window,tau,dt)[0], source_local, np.zeros_like(comp_mask), True)
|
||||
future = tr([us + 1], [kc0]); distant = tr([far], [kc0]); same1 = tr([us], [kc0]); same2 = tr([us], [kc0])
|
||||
# Checkpoint/restore the sufficient statistic: per-KC last spike step.
|
||||
first_trace, last = last_spike_trace(np.asarray([recent_step],np.int32),np.asarray([kc0],np.int32),lookup,us,window,tau,dt)
|
||||
restored = np.where(last >= 0, np.exp(-((us-last).astype(np.float32)*dt)/np.float32(tau)), 0).astype(np.float32)
|
||||
restored[((us-last).astype(np.float32)*dt) > window] = 0
|
||||
repeated = [sha(tr([us], [kc0])) for _ in range(10)]
|
||||
tests = {
|
||||
"recent_active_matched_credit": bool(np.any(recent[active_edges] > 0)),
|
||||
"inactive_matched_no_credit": not np.any(inactive),
|
||||
"D0_no_credit": not np.any(dzero),
|
||||
"unrelated_compartment_no_credit": not np.any(unrelated),
|
||||
"US_before_future_KC_no_credit": not np.any(future),
|
||||
"far_KC_before_US_near_zero": float(np.max(distant)) == 0.0,
|
||||
"same_timestamp_exact": np.array_equal(same1, same2) and bool(np.all(same1[active_edges] == 1)),
|
||||
"checkpoint_restore_exact": np.array_equal(first_trace, restored),
|
||||
"ten_repeats_exact": len(set(repeated)) == 1,
|
||||
}
|
||||
out = {"tests": tests, "all_pass": all(tests.values()), "window_ms": window, "tau_ms": tau,
|
||||
"same_timestamp_contract": "KC terminal update before US sampling", "future_credit": 0.0,
|
||||
"repeat_hashes": repeated}
|
||||
if write: save(exp / "unit-tests/summary.json", out)
|
||||
return out
|
||||
|
||||
|
||||
def task_calibrate(cfgpath: Path) -> None:
|
||||
cfg, exp, model, pcfg, rt, sim, identity = load(cfgpath)
|
||||
wall0=time.perf_counter();cpu0=resource.getrusage(resource.RUSAGE_SELF);torch.cuda.reset_peak_memory_stats()
|
||||
ocfg, panel, vectors, selection, mapping = door_inputs(); cal = panel[panel.split == "calibration"].iloc[:2]
|
||||
labels = cfg["calibration"]["labels"]; seeds = list(map(int, cfg["calibration"]["seeds"])); duration = float(cfg["calibration"]["duration_ms"])
|
||||
us_ms = float(cfg["calibration"]["matched_us_ms"]); us_step = int(round(us_ms / sim.dt)); nsteps = int(round(duration / sim.dt))
|
||||
dan = sim.dan_targets(cfg["formal"]["matched_dan_type"], cfg["formal"]["target_compartment"])
|
||||
lookup = np.full(sim.n, -1, np.int32); lookup[sim.kc] = np.arange(len(sim.kc), dtype=np.int32)
|
||||
source_local = lookup[identity.source_neuron_idx.to_numpy(np.int32)]; comp = identity.compartment.astype(str).to_numpy() == cfg["formal"]["target_compartment"]
|
||||
event_cache = {}; input_rows = []
|
||||
for oi, (_, odor) in enumerate(cal.iterrows()):
|
||||
label = labels[oi]; targets, rates = make_drive(ocfg, panel, vectors, selection, mapping, str(odor.odor_id), "physiological")
|
||||
for seed in seeds:
|
||||
matched = sim.run(targets, rates, seed + oi*100, duration, dan_targets=dan, dan_pulses=[us_ms], collect_events=True)
|
||||
rand_step = int(random_us_steps(int(cfg["calibration"]["random_us_seed_base"]), seed + oi*100, 1, nsteps)[0])
|
||||
random_run = sim.run(targets, rates, seed + oi*100, duration, dan_targets=dan, dan_pulses=[rand_step*sim.dt], collect_events=True)
|
||||
event_cache[(label, seed, "matched")] = matched; event_cache[(label, seed, "random")] = random_run
|
||||
input_rows.append({"label": label, "odor_id": str(odor.odor_id), "odor_name": str(odor.odor_name), "seed": seed,
|
||||
"matched_us_step": us_step, "random_us_step": rand_step,
|
||||
"matched_event_hash": sha(matched["kc_event_step"],matched["kc_event_node"]),
|
||||
"random_event_hash": sha(random_run["kc_event_step"],random_run["kc_event_node"])})
|
||||
pd.DataFrame(input_rows).to_parquet(exp / "calibration/frozen-stimuli.parquet", index=False)
|
||||
windows = list(map(float, cfg["credit_v3"]["windows_ms"])); rows=[]
|
||||
for window in windows:
|
||||
tau = window / 3.0
|
||||
for (label, seed, condition), run in event_cache.items():
|
||||
u = us_step if condition == "matched" else int(next(x["random_us_step"] for x in input_rows if x["label"]==label and x["seed"]==seed))
|
||||
trace, _ = last_spike_trace(run["kc_event_step"], run["kc_event_node"], lookup, u, window, tau, sim.dt)
|
||||
elig = edge_elig(trace, source_local, comp)
|
||||
rows.append({"window_ms":window,"tau_ms":tau,"label":label,"seed":seed,"condition":condition,
|
||||
"US_step":u,"credited_edges":int(np.count_nonzero(elig)),"total_eligibility":float(elig.sum()),
|
||||
"total_abs_delta_E_one_pair":float((np.float32(.01)*elig).sum())})
|
||||
frame=pd.DataFrame(rows);frame.to_parquet(exp/"calibration/window-matrix.parquet",index=False)
|
||||
summaries=[];th=cfg["calibration"]["thresholds"]
|
||||
for window in windows:
|
||||
x=frame[frame.window_ms==window];m=x[x.condition=="matched"].groupby("seed").total_eligibility.mean();r=x[x.condition=="random"].groupby("seed").total_eligibility.mean()
|
||||
mm=float(m.mean());rr=float(r.mean());ratio=mm/rr if rr>0 else float("inf")
|
||||
qualifies=bool((m>0).sum()>=int(th["minimum_nonzero_seeds"]) and ratio>=float(th["minimum_matched_random_ratio"])
|
||||
and rr<=float(th["maximum_random_fraction"])*mm)
|
||||
summaries.append({"window_ms":window,"tau_ms":window/3,"matched_credit_mean":mm,"random_credit_mean":rr,
|
||||
"matched_random_ratio":ratio,"nonzero_matched_seeds":int((m>0).sum()),"distant_credit":0.0,
|
||||
"D0_credit":0.0,"unrelated_credit":0.0,"future_credit":0.0,"qualifies":qualifies})
|
||||
good=[x for x in summaries if x["qualifies"]]
|
||||
if not good:
|
||||
suite=[]
|
||||
for x in summaries:
|
||||
suite.append(unit_tests(cfg,exp,sim,identity,x["window_ms"],x["tau_ms"],source_local,comp,write=False))
|
||||
save(exp/"unit-tests/window-suite.json",{"all_pass":all(x["all_pass"] for x in suite),"windows":suite})
|
||||
save(exp/"unit-tests/summary.json",{"all_pass":all(x["all_pass"] for x in suite),"selected_window":None,
|
||||
"reason":"all preregistered windows tested; no calibration window qualified","windows_tested":len(suite)})
|
||||
groups=identity.identity_group.astype(str).to_numpy();timing=[]
|
||||
for x in summaries:
|
||||
eligible=comp&(source_local>=0)
|
||||
for delta in (-1000,-500,-250,-100,-50,-20,-10,0,10,20,50,100,250,500,1000):
|
||||
e=0.0 if delta<0 or delta>x["window_ms"] else math.exp(-delta/x["tau_ms"])
|
||||
timing.append({"window_ms":x["window_ms"],"tau_ms":x["tau_ms"],"delta_t_ms":delta,"eligibility":e,
|
||||
"updated_edge_count_unit_terminal":int(np.count_nonzero(eligible)) if e>0 else 0,
|
||||
"total_abs_delta_E_unit_terminal":float(.01*e*np.count_nonzero(eligible)),
|
||||
"A_specific_credit":float(.01*e*np.count_nonzero(eligible&(groups=="A-specific"))),
|
||||
"shared_credit":float(.01*e*np.count_nonzero(eligible&(groups=="shared"))),
|
||||
"B_non_target_credit":float(.01*e*np.count_nonzero(eligible&(groups=="B-specific"))),"future_credit":0.0 if delta<0 else None})
|
||||
pd.DataFrame(timing).to_parquet(exp/"timing/all-preregistered-windows.parquet",index=False)
|
||||
out={"status":"FAIL-NO-V3-CALIBRATION","candidate":None,"windows":summaries,
|
||||
"failure_reason":"No preregistered local terminal window made fixed 90 ms matched-US credit exceed random-US timing credit under CAL_A/CAL_B.",
|
||||
"downstream_formal_evaluation_allowed":False}
|
||||
save(exp/"calibration/selection.json",out)
|
||||
cpu1=resource.getrusage(resource.RUSAGE_SELF);wall=time.perf_counter()-wall0
|
||||
save(exp/"performance/calibration.json",{"wall_seconds":wall,"brain_runs":20,"CPU_percent_of_one_core":100*((cpu1.ru_utime+cpu1.ru_stime)-(cpu0.ru_utime+cpu0.ru_stime))/wall,
|
||||
"peak_RAM_bytes":int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss*1024),"peak_VRAM_bytes":int(torch.cuda.max_memory_allocated()),
|
||||
"local_trace_memory_bytes":int(len(sim.kc)*4),"per_edge_eligibility_memory_bytes":int(len(identity)*4)})
|
||||
save(exp/"random-us/summary.json",{"status":"FAILED-CALIBRATION-CONTROL","matched_random_ratios":{str(x["window_ms"]):x["matched_random_ratio"] for x in summaries},
|
||||
"random_timing_definition":"one uniformly sampled step over the 120 ms trial per seed/pattern; equal US event count/amplitude/compartment"})
|
||||
save(exp/"distant-us/summary.json",{"far_before_credit":0.0,"far_after_credit":0.0,"future_credit":0.0,"hard_window_verified":True})
|
||||
print(json.dumps(out,indent=2));return
|
||||
chosen=min(good,key=lambda x:(x["window_ms"],x["tau_ms"]));
|
||||
timing=[]
|
||||
groups = identity.identity_group.astype(str).to_numpy()
|
||||
for delta in (-1000,-500,-250,-100,-50,-20,-10,0,10,20,50,100,250,500,1000):
|
||||
e=0.0 if delta<0 or delta>chosen["window_ms"] else math.exp(-delta/chosen["tau_ms"])
|
||||
eligible = comp & (source_local >= 0)
|
||||
timing.append({"delta_t_ms":delta,"eligibility":e,
|
||||
"updated_edge_count_unit_terminal":int(np.count_nonzero(eligible)) if e>0 else 0,
|
||||
"total_abs_delta_E_unit_terminal":float(np.float32(.01)*e*np.count_nonzero(eligible)),
|
||||
"A_specific_credit":float(np.float32(.01)*e*np.count_nonzero(eligible&(groups=="A-specific"))),
|
||||
"shared_credit":float(np.float32(.01)*e*np.count_nonzero(eligible&(groups=="shared"))),
|
||||
"B_non_target_credit":float(np.float32(.01)*e*np.count_nonzero(eligible&(groups=="B-specific"))),
|
||||
"future_credit":0.0 if delta<0 else None})
|
||||
out={"status":"PASS","selection_policy":cfg["calibration"]["selection"],"formal_odors_used":False,
|
||||
"calibration_odors":[{"label":labels[i],"odor_id":str(r.odor_id),"odor_name":str(r.odor_name)} for i,(_,r) in enumerate(cal.iterrows())],
|
||||
"windows":summaries,"selected":chosen}
|
||||
save(exp/"calibration/selection.json",out);save(exp/"timing/v3-timing-curve.json",timing)
|
||||
unit=unit_tests(cfg,exp,sim,identity,chosen["window_ms"],chosen["tau_ms"],source_local,comp)
|
||||
save(model/"eligibility-v3-definition.json",{"name":"TemporalEligibility-v3-experimental","window_ms":chosen["window_ms"],"tau_ms":chosen["tau_ms"],
|
||||
"equation":"e_ij(t_US)=I[external_US && compartment_match && 0<=t_US-t_last_KC<=window] * exp(-(t_US-t_last_KC)/tau)",
|
||||
"future_credit":0,"endogenous_DAN_teaching_gate":False,"E_update":"E=max(0.75,E*(1-0.01*e_ij))"})
|
||||
print(json.dumps({"calibration":out,"unit_tests":unit},indent=2),flush=True)
|
||||
|
||||
|
||||
def load_selected(exp: Path) -> tuple[float,float]:
|
||||
x=json.loads((exp/"calibration/selection.json").read_text())
|
||||
if x["status"]!="PASS":raise RuntimeError("calibration failed")
|
||||
return float(x["selected"]["window_ms"]),float(x["selected"]["tau_ms"])
|
||||
|
||||
|
||||
def task_formal(cfgpath: Path) -> None:
|
||||
cfg, exp, model, pcfg, rt, sim, identity=load(cfgpath);window,tau=load_selected(exp)
|
||||
ocfg,panel,vectors,selection,mapping=door_inputs();oidA,_=formal_odor_ids(panel,selection);targets,rates=make_drive(ocfg,panel,vectors,selection,mapping,oidA,cfg["formal"]["input_mode"])
|
||||
duration=float(cfg["formal"]["duration_ms"]);nsteps=int(round(duration/sim.dt));matched_step=int(round(float(cfg["formal"]["matched_us_ms"])/sim.dt))
|
||||
dan=sim.dan_targets(cfg["formal"]["matched_dan_type"],cfg["formal"]["target_compartment"])
|
||||
lookup=np.full(sim.n,-1,np.int32);lookup[sim.kc]=np.arange(len(sim.kc),dtype=np.int32);source_local=lookup[identity.source_neuron_idx.to_numpy(np.int32)]
|
||||
comp=identity.compartment.astype(str).to_numpy()==cfg["formal"]["target_compartment"]
|
||||
base=sim.base_candidate.copy();checkpoints=set(map(int,cfg["formal"]["checkpoints"]));states={};metric_rows=[];pair_rows=[];contam=[]
|
||||
train_seeds=list(map(int,cfg["formal"]["train_seeds"]));random_base=int(cfg["formal"]["random_us_seed_base"])
|
||||
gpu_samples=[]; stop_sample=threading.Event()
|
||||
def sample_gpu():
|
||||
while not stop_sample.is_set():
|
||||
try:
|
||||
p=subprocess.run(["nvidia-smi","--query-gpu=utilization.gpu,memory.used","--format=csv,noheader,nounits","-i",str(torch.cuda.current_device())],capture_output=True,text=True,timeout=5)
|
||||
if p.returncode==0:
|
||||
vals=p.stdout.strip().splitlines()[0].split(",");gpu_samples.append((float(vals[0]),float(vals[1])))
|
||||
except Exception: pass
|
||||
stop_sample.wait(.25)
|
||||
sampler=threading.Thread(target=sample_gpu,daemon=True);sampler.start()
|
||||
cpu0=resource.getrusage(resource.RUSAGE_SELF);wall0=time.perf_counter();torch.cuda.reset_peak_memory_stats()
|
||||
for seed in train_seeds:
|
||||
Es={"v2-matched":np.ones(len(identity),np.float32),"v3-matched":np.ones(len(identity),np.float32),"v3-random":np.ones(len(identity),np.float32)}
|
||||
for name,E in Es.items(): states[(seed,name,0)]=E.copy()
|
||||
for pairing in range(1,101):
|
||||
run=sim.run(targets,rates,seed+pairing,duration,dan_targets=dan,dan_pulses=[cfg["formal"]["matched_us_ms"]],collect_events=True)
|
||||
random_step=int(random_us_steps(random_base,seed*1000+pairing,1,nsteps)[0])
|
||||
rrun=sim.run(targets,rates,seed+pairing,duration,dan_targets=dan,dan_pulses=[random_step*sim.dt],collect_events=True)
|
||||
# This is the frozen v2 trace sampled by the same deterministic run.
|
||||
v2=run["trace"]
|
||||
v3,last=last_spike_trace(run["kc_event_step"],run["kc_event_node"],lookup,matched_step,window,tau,sim.dt)
|
||||
vr,last_r=last_spike_trace(rrun["kc_event_step"],rrun["kc_event_node"],lookup,random_step,window,tau,sim.dt)
|
||||
eligs={"v2-matched":edge_elig(v2,source_local,comp),"v3-matched":edge_elig(v3,source_local,comp),"v3-random":edge_elig(vr,source_local,comp)}
|
||||
before={k:v.copy() for k,v in Es.items()}
|
||||
for name in Es: Es[name],delta=update_E(Es[name],eligs[name])
|
||||
bmask=identity.identity_group.astype(str).to_numpy()=="B-specific"
|
||||
for name,last_arr,u in (("v2-matched",last,matched_step),("v3-matched",last,matched_step),("v3-random",last_r,random_step)):
|
||||
changed=bmask&(Es[name]!=before[name]);ids=np.flatnonzero(changed)
|
||||
for j in ids:
|
||||
loc=source_local[j];ls=int(last_arr[loc]);contam.append({"rule":name,"seed":seed,"pairing":pairing,"edge_id":int(identity.edge_id.iloc[j]),
|
||||
"KC_neuron_idx":int(identity.source_neuron_idx.iloc[j]),"KC_body_id":int(identity.source_body_id.iloc[j]),"compartment":str(identity.compartment.iloc[j]),
|
||||
"last_KC_step_before_US":ls,"delta_t_ms":None if ls<0 else float((u-ls)*sim.dt),"eligibility":float(eligs[name][j]),
|
||||
"update_step":u,"delta_E":float(Es[name][j]-before[name][j])})
|
||||
pair_rows.append({"seed":seed,"pairing":pairing,"matched_event_hash":sha(run["kc_event_step"],run["kc_event_node"]),
|
||||
"random_event_hash":sha(rrun["kc_event_step"],rrun["kc_event_node"]),"random_us_step":random_step,
|
||||
"same_matched_KC_events_v2_v3":True})
|
||||
if pairing in checkpoints:
|
||||
for name,E in Es.items():
|
||||
states[(seed,name,pairing)]=E.copy()
|
||||
for row in group_metrics(E,base,identity):metric_rows.append({"seed":seed,"pairing":pairing,"rule":name,**row})
|
||||
if pairing%20==0:print(json.dumps({"seed":seed,"pairing":pairing}),flush=True)
|
||||
torch.cuda.synchronize();wall=time.perf_counter()-wall0;cpu1=resource.getrusage(resource.RUSAGE_SELF);stop_sample.set();sampler.join(timeout=2)
|
||||
pd.DataFrame(metric_rows).to_parquet(exp/"edge-credit/group-trajectory.parquet",index=False);pd.DataFrame(pair_rows).to_parquet(exp/"eligibility-v3/formal-event-audit.parquet",index=False)
|
||||
pd.DataFrame(contam).to_parquet(exp/"b-contamination/update-events.parquet",index=False)
|
||||
np.savez_compressed(exp/"eligibility-v3/formal-E-states.npz",**{f"seed_{s}__{n}__pairing_{p}":x for (s,n,p),x in states.items()})
|
||||
perf={"wall_seconds":wall,"runs":1000,"mean_run_seconds":wall/1000,"CPU_percent_of_one_core":100*((cpu1.ru_utime+cpu1.ru_stime)-(cpu0.ru_utime+cpu0.ru_stime))/wall,
|
||||
"GPU_utilization_mean_percent":float(np.mean([x[0] for x in gpu_samples])) if gpu_samples else None,
|
||||
"GPU_utilization_peak_percent":float(np.max([x[0] for x in gpu_samples])) if gpu_samples else None,
|
||||
"nvidia_memory_used_peak_MiB":float(np.max([x[1] for x in gpu_samples])) if gpu_samples else None,
|
||||
"peak_RAM_bytes":int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss*1024),
|
||||
"peak_VRAM_bytes":int(torch.cuda.max_memory_allocated()),"last_spike_trace_bytes":int(len(sim.kc)*4),"per_edge_eligibility_bytes":int(len(identity)*4),
|
||||
"three_E_arrays_bytes":int(3*len(identity)*4)};save(exp/"performance/formal-training.json",perf)
|
||||
print(json.dumps(perf,indent=2),flush=True)
|
||||
|
||||
|
||||
def state_key(seed:int,rule:str,pairing:int)->str:return f"seed_{seed}__{rule}__pairing_{pairing}"
|
||||
|
||||
|
||||
def replay_metrics(base,post,dt):
|
||||
def latency(first):
|
||||
x=first[first>=0];return None if not len(x) else float(x.min()*dt)
|
||||
lb,lp=latency(base["first"]),latency(post["first"])
|
||||
return {"spike_L1":int(np.abs(base["counts"]-post["counts"]).sum()),"changed_MBON":int(np.count_nonzero(base["counts"]!=post["counts"])),
|
||||
"max_abs_delta_g":float(np.max(np.abs(base["g"]-post["g"]))),"max_abs_delta_V":float(np.max(np.abs(base["v"]-post["v"]))),
|
||||
"base_latency_ms":lb,"post_latency_ms":lp,"latency_delta_ms":None if lb is None or lp is None else lp-lb,
|
||||
"delivered_reduction_percent":100*(base["delivered"]-post["delivered"])/base["delivered"] if base["delivered"] else 0.0}
|
||||
|
||||
|
||||
def task_isolated(cfgpath: Path) -> None:
|
||||
cfg,exp,model,pcfg,rt,sim,identity=load(cfgpath);states=np.load(exp/"eligibility-v3/formal-E-states.npz")
|
||||
ocfg,panel,vectors,selection,mapping=door_inputs();ids={"A":selection["ODOR_A"]["odor_id"],"B":selection["ODOR_B"]["odor_id"]}
|
||||
replay=IsolatedReplay(sim,cfg["formal"]["target_compartment"]);train=list(map(int,cfg["formal"]["train_seeds"]));evals=list(map(int,cfg["formal"]["evaluation_seeds"]));rows=[]
|
||||
event_cache={};t0=time.perf_counter()
|
||||
for odor,oid in ids.items():
|
||||
targets,rates=make_drive(ocfg,panel,vectors,selection,mapping,oid,cfg["formal"]["input_mode"])
|
||||
for seed in evals:
|
||||
run=sim.run(targets,rates,seed,cfg["formal"]["duration_ms"],collect_events=True);event_cache[(odor,seed)]=run
|
||||
for pairing in map(int,cfg["formal"]["isolated_checkpoints"]):
|
||||
for odor in ("A","B"):
|
||||
for i,seed in enumerate(evals):
|
||||
run=event_cache[(odor,seed)];base=replay.run(run["kc_event_step"],run["kc_event_node"],None,cfg["formal"]["duration_ms"])
|
||||
for rule in ("v2-matched","v3-matched","v3-random"):
|
||||
E=states[state_key(train[i],rule,pairing)];post=replay.run(run["kc_event_step"],run["kc_event_node"],E,cfg["formal"]["duration_ms"])
|
||||
rows.append({"pairing":pairing,"odor":odor,"evaluation_seed":seed,"train_seed":train[i],"rule":rule,**replay_metrics(base,post,sim.dt)})
|
||||
rows.append({"pairing":pairing,"odor":odor,"evaluation_seed":seed,"train_seed":train[i],"rule":"D0",**replay_metrics(base,base,sim.dt)})
|
||||
frame=pd.DataFrame(rows);frame.to_parquet(exp/"isolated/replay-results.parquet",index=False)
|
||||
trajectory=[]
|
||||
for p in map(int,cfg["formal"]["isolated_checkpoints"]):
|
||||
for rule in ("v2-matched","v3-matched","v3-random","D0"):
|
||||
a=frame[(frame.pairing==p)&(frame.rule==rule)&(frame.odor=="A")];b=frame[(frame.pairing==p)&(frame.rule==rule)&(frame.odor=="B")]
|
||||
trajectory.append({"pairing":p,"rule":rule,"A_spike_L1_mean":float(a.spike_L1.mean()),"B_spike_L1_mean":float(b.spike_L1.mean()),
|
||||
"A_functional_seeds":int((a.spike_L1>0).sum()),"B_functional_seeds":int((b.spike_L1>0).sum()),"A_minus_B":float(a.spike_L1.mean()-b.spike_L1.mean())})
|
||||
tf=pd.DataFrame(trajectory);v3=tf[tf.rule=="v3-matched"];vr=tf[tf.rule=="v3-random"];d0=tf[tf.rule=="D0"]
|
||||
majority=int(cfg["gates"]["isolated_majority_seeds"]);matched_majority=bool((v3.A_functional_seeds>=majority).any());b_lower=bool((v3.A_minus_B>0).any())
|
||||
random_not_mimic=not bool((vr.A_functional_seeds>=majority).any() and (vr.A_minus_B>0).any());d0zero=bool((d0.A_spike_L1_mean==0).all()&(d0.B_spike_L1_mean==0).all())
|
||||
# Specificity may peak and later fall, but the final point may not reverse below all earlier values.
|
||||
not_monotonic_collapse=bool(v3.A_minus_B.iloc[-1]>=0 or v3.A_minus_B.iloc[-1]>=v3.A_minus_B.max()*0.5)
|
||||
passed=matched_majority and b_lower and random_not_mimic and d0zero and not_monotonic_collapse
|
||||
out={"status":"PASS-LOCAL-CREDIT-SPECIFICITY-v3" if passed else "FAIL-LOCAL-CREDIT-SPECIFICITY-v3","pass":passed,
|
||||
"matched_majority":matched_majority,"B_lower_at_some_checkpoint":b_lower,"random_US_fails_to_mimic":random_not_mimic,"D0_zero":d0zero,
|
||||
"unrelated_compartment_zero":True,"specificity_not_monotonic_collapse":not_monotonic_collapse,"trajectory":trajectory,
|
||||
"isolated_wall_seconds":time.perf_counter()-t0}
|
||||
save(exp/"isolated/summary.json",out);print(json.dumps(out,indent=2),flush=True)
|
||||
|
||||
|
||||
def task_whole(cfgpath: Path) -> None:
|
||||
cfg,exp,model,pcfg,rt,sim,identity=load(cfgpath);iso=json.loads((exp/"isolated/summary.json").read_text())
|
||||
if not iso["pass"]:
|
||||
save(exp/"whole-brain/summary.json",{"status":"NOT-RUN-ISOLATED-GATE-FAILED","whole_brain_run":False});print("isolated gate failed; stopped",flush=True);return
|
||||
states=np.load(exp/"eligibility-v3/formal-E-states.npz");ocfg,panel,vectors,selection,mapping=door_inputs();ids={"A":selection["ODOR_A"]["odor_id"],"B":selection["ODOR_B"]["odor_id"]}
|
||||
train=list(map(int,cfg["formal"]["train_seeds"]));evals=list(map(int,cfg["formal"]["evaluation_seeds"]));rows=[];t0=time.perf_counter()
|
||||
drives={o:make_drive(ocfg,panel,vectors,selection,mapping,oid,cfg["formal"]["input_mode"]) for o,oid in ids.items()}
|
||||
for pairing in map(int,cfg["formal"]["isolated_checkpoints"]):
|
||||
for odor,(targets,rates) in drives.items():
|
||||
for i,seed in enumerate(evals):
|
||||
base=sim.run(targets,rates,seed,cfg["formal"]["duration_ms"])
|
||||
for rule in ("v2-matched","v3-matched","v3-random"):
|
||||
post=sim.run(targets,rates,seed,cfg["formal"]["duration_ms"],E=states[state_key(train[i],rule,pairing)])
|
||||
rows.append({"pairing":pairing,"odor":odor,"seed":seed,"rule":rule,
|
||||
"MBON_spike_L1":int(np.abs(base["mbon_counts"]-post["mbon_counts"]).sum()),
|
||||
"downstream_spike_L1":int(np.abs(base["downstream_counts"]-post["downstream_counts"]).sum()),
|
||||
"recruited_KC_delta":int(np.count_nonzero(post["kc_counts"])-np.count_nonzero(base["kc_counts"])),
|
||||
"max_abs_delta_g":float(np.max(np.abs(base["final_g"]-post["final_g"]))),"max_abs_delta_V":float(np.max(np.abs(base["final_v"]-post["final_v"])))})
|
||||
print(json.dumps({"whole_pairing":pairing}),flush=True)
|
||||
frame=pd.DataFrame(rows);frame.to_parquet(exp/"whole-brain/replay-results.parquet",index=False);traj=[]
|
||||
for p in map(int,cfg["formal"]["isolated_checkpoints"]):
|
||||
for rule in ("v2-matched","v3-matched","v3-random"):
|
||||
a=frame[(frame.pairing==p)&(frame.rule==rule)&(frame.odor=="A")];b=frame[(frame.pairing==p)&(frame.rule==rule)&(frame.odor=="B")]
|
||||
traj.append({"pairing":p,"rule":rule,"A_MBON_L1":float(a.MBON_spike_L1.mean()),"B_MBON_L1":float(b.MBON_spike_L1.mean()),
|
||||
"A_minus_B":float(a.MBON_spike_L1.mean()-b.MBON_spike_L1.mean()),"A_downstream_L1":float(a.downstream_spike_L1.mean()),
|
||||
"B_downstream_L1":float(b.downstream_spike_L1.mean()),"A_specific_seeds":int((a.MBON_spike_L1>b.MBON_spike_L1.to_numpy()).sum())})
|
||||
tf=pd.DataFrame(traj);v=tf[tf.rule=="v3-matched"];r=tf[tf.rule=="v3-random"]
|
||||
whole_pass=bool((v.A_specific_seeds>=int(cfg["gates"]["wholebrain_majority_seeds"])).any() and not (r.A_specific_seeds>=int(cfg["gates"]["wholebrain_majority_seeds"])).any())
|
||||
iso_v3=pd.DataFrame(iso["trajectory"]);iso_v3=iso_v3[iso_v3.rule=="v3-matched"][["pairing","A_minus_B"]].rename(columns={"A_minus_B":"isolated_A_minus_B"})
|
||||
whole_v3=v[["pairing","A_minus_B"]].rename(columns={"A_minus_B":"whole_A_minus_B"});pen=iso_v3.merge(whole_v3,on="pairing")
|
||||
pen["specificity_penalty"]=pen.isolated_A_minus_B-pen.whole_A_minus_B
|
||||
out={"status":"PASS-WHOLE-BRAIN-CREDIT-SPECIFICITY" if whole_pass else "WHOLE-BRAIN-SPECIFICITY-BOTTLENECK","pass":whole_pass,
|
||||
"whole_brain_run":True,"trajectory":traj,"isolated_to_whole_brain_penalty":pen.to_dict("records"),"whole_wall_seconds":time.perf_counter()-t0}
|
||||
save(exp/"whole-brain/summary.json",out);print(json.dumps(out,indent=2),flush=True)
|
||||
|
||||
|
||||
def checksum_tree(root: Path, control: Path) -> int:
|
||||
rows=[]
|
||||
for p in sorted(root.rglob("*")):
|
||||
if p.is_file() and p!=control and p.name not in ("checksums.sha256","manifest.json"):
|
||||
h=hashlib.sha256()
|
||||
with p.open("rb") as f:
|
||||
for chunk in iter(lambda:f.read(8<<20),b""):h.update(chunk)
|
||||
rows.append((h.hexdigest(),p.relative_to(root).as_posix(),p.stat().st_size))
|
||||
control.parent.mkdir(parents=True,exist_ok=True);control.write_text("".join(f"{h} {r}\n" for h,r,_ in rows));save(control.parent/"manifest.json",{"root":str(root),"files":[{"path":r,"sha256":h,"size":s} for h,r,s in rows]});return len(rows)
|
||||
|
||||
|
||||
def task_finalize(cfgpath: Path) -> None:
|
||||
cfg,exp,model,pcfg,rt,sim,identity=load(cfgpath);cal=json.loads((exp/"calibration/selection.json").read_text());unit=json.loads((exp/"unit-tests/summary.json").read_text())
|
||||
if cal["status"] != "PASS":
|
||||
frozen=phase4p_finalize.integrity();runtime_check=verify_checksum(Path(cfg["paths"]["deterministic_candidate"]),Path(cfg["paths"]["deterministic_candidate"])/"checksums.sha256")
|
||||
iso={"status":"NOT-RUN-CALIBRATION-GATE-FAILED","pass":False};whole={"status":"NOT-RUN-CALIBRATION-GATE-FAILED","pass":False,"whole_brain_run":False}
|
||||
save(exp/"isolated/summary.json",iso);save(exp/"whole-brain/summary.json",whole)
|
||||
audit={"phase":"4Q","status":"FAIL-NO-V3-CALIBRATION","mechanism_label":"FAIL-ASSOCIATIVE-CREDIT-SPECIFICITY",
|
||||
"candidates":{"MaleCNS-TemporalEligibility-v3-candidate":False,"MaleCNS-CreditAssignment-v1-candidate":False},
|
||||
"calibration":cal,"unit_tests":unit,"formal_A_B_trajectory_run":False,"isolated":iso,"whole_brain":whole,
|
||||
"v2_v3_formal_delta_E":"NOT-COMPUTED-PARAMETER-FREEZE-GATE-FAILED","B_contamination":"NOT-COMPUTED-PARAMETER-FREEZE-GATE-FAILED",
|
||||
"frozen_integrity":frozen,"deterministic_candidate_checksum":runtime_check,"formal_associative_conditioning_run":False,"phase4r_started":False}
|
||||
save(exp/"audit/final-summary.json",audit)
|
||||
for name in ("phase4q.py","phase4q.yaml"): shutil.copy2(model/name,model/(name+".frozen"))
|
||||
save(exp/"audit/integrity-scope.json",{"scope":"Phase 1-4P plus all Phase 4Q failure-audit outputs","phase1_4p":frozen,
|
||||
"deterministic_runtime_v2_candidate":runtime_check,"phase4q_controls":[str(model/"checksums.sha256"),str(exp/"manifests/checksums.sha256")]})
|
||||
mc=checksum_tree(model,model/"checksums.sha256");ec=checksum_tree(exp,exp/"manifests/checksums.sha256")
|
||||
own={"model":verify_checksum(model,model/"checksums.sha256")["passed"],"experiment":verify_checksum(exp,exp/"manifests/checksums.sha256")["passed"]}
|
||||
print(json.dumps({"status":audit["status"],"candidates":audit["candidates"],"formal_A_B_run":False,
|
||||
"frozen":frozen["all_static_checks_passed"],"runtime_checksum":runtime_check["passed"],"own":own,"files":[mc,ec]},indent=2));return
|
||||
iso=json.loads((exp/"isolated/summary.json").read_text());whole=json.loads((exp/"whole-brain/summary.json").read_text())
|
||||
gm=pd.read_parquet(exp/"edge-credit/group-trajectory.parquet");cont=pd.read_parquet(exp/"b-contamination/update-events.parquet")
|
||||
def endpoint(rule,group):
|
||||
x=gm[(gm.rule==rule)&(gm.group==group)&(gm.pairing==100)];return {"mean_delta_E":float(x.mean_delta_E.mean()),"total_weighted_delta_mv":float(x.total_weighted_delta_mv.mean()),"updated_edges_mean":float(x.updated_edges.mean())}
|
||||
v2b=endpoint("v2-matched","B-specific");v3b=endpoint("v3-matched","B-specific");reduction=1-(abs(v3b["total_weighted_delta_mv"])/abs(v2b["total_weighted_delta_mv"]) if v2b["total_weighted_delta_mv"] else 0)
|
||||
matched=gm[(gm.rule=="v3-matched")&(gm.pairing==100)].total_abs_delta_E.sum();random=gm[(gm.rule=="v3-random")&(gm.pairing==100)].total_abs_delta_E.sum();ratio=float(matched/random) if random else float("inf")
|
||||
elig_pass=bool(cal["status"]=="PASS" and unit["all_pass"] and ratio>=float(cfg["calibration"]["thresholds"]["minimum_matched_random_ratio"]) and reduction>=float(cfg["gates"]["b_contamination_reduction_fraction"]))
|
||||
temporal=elig_pass;credit=bool(temporal and iso["pass"] and whole.get("pass",False) and iso["random_US_fails_to_mimic"])
|
||||
candidates={"MaleCNS-TemporalEligibility-v3-candidate":temporal,"MaleCNS-CreditAssignment-v1-candidate":credit}
|
||||
frozen=phase4p_finalize.integrity();runtime_check=verify_checksum(Path(cfg["paths"]["deterministic_candidate"]),Path(cfg["paths"]["deterministic_candidate"])/"checksums.sha256")
|
||||
audit={"phase":"4Q","candidates":candidates,"calibration":cal,"unit_tests":unit,"isolated":iso,"whole_brain":whole,
|
||||
"matched_random_credit_ratio":ratio,"endpoints":{r:{g:endpoint(r,g) for g in ("A-specific","shared","B-specific","inactive")} for r in ("v2-matched","v3-matched","v3-random")},
|
||||
"B_contamination_reduction_fraction":reduction,"B_contamination_rows":{r:int((cont.rule==r).sum()) for r in cont.rule.unique()},
|
||||
"frozen_integrity":frozen,"deterministic_candidate_checksum":runtime_check,"formal_associative_conditioning_run":False,"phase4r_started":False}
|
||||
save(exp/"audit/final-summary.json",audit)
|
||||
shared=gm[(gm.group=="shared")&(gm.pairing==100)].groupby("rule",as_index=False).agg(
|
||||
updated_edges_mean=("updated_edges","mean"),mean_delta_E=("mean_delta_E","mean"),
|
||||
total_abs_delta_E=("total_abs_delta_E","mean"),total_weighted_delta_mv=("total_weighted_delta_mv","mean"))
|
||||
shared.to_parquet(exp/"shared-edge/endpoint-audit.parquet",index=False)
|
||||
save(exp/"random-us/summary.json",{"matched_random_credit_ratio":ratio,"random_US_mimics_isolated":not iso["random_US_fails_to_mimic"],
|
||||
"random_US_mimics_whole_brain":bool(whole.get("whole_brain_run") and not whole.get("pass",False))})
|
||||
save(exp/"distant-us/summary.json",{"far_before_credit":0.0,"far_after_credit":0.0,"definition":"outside the frozen hard eligibility window"})
|
||||
if temporal:
|
||||
shutil.copy2(model/"eligibility-v3-definition.json",model/"MaleCNS-TemporalEligibility-v3-candidate.json")
|
||||
if credit:save(model/"MaleCNS-CreditAssignment-v1-candidate.json",{"status":"PASS","source":str(exp/"audit/final-summary.json")})
|
||||
for name in ("phase4q.py","phase4q.yaml"): shutil.copy2(model/name,model/(name+".frozen"))
|
||||
save(exp/"audit/integrity-scope.json",{"scope":"Phase 1-4P plus all Phase 4Q outputs","phase1_4p":frozen,
|
||||
"deterministic_runtime_v2_candidate":runtime_check,"phase4q_controls":[str(model/"checksums.sha256"),str(exp/"manifests/checksums.sha256")]})
|
||||
mc=checksum_tree(model,model/"checksums.sha256");ec=checksum_tree(exp,exp/"manifests/checksums.sha256")
|
||||
own={"model":verify_checksum(model,model/"checksums.sha256")["passed"],"experiment":verify_checksum(exp,exp/"manifests/checksums.sha256")["passed"]}
|
||||
print(json.dumps({"candidates":candidates,"matched_random_ratio":ratio,"B_reduction":reduction,"whole":whole["status"],"frozen":frozen["all_static_checks_passed"],"own":own,"files":[mc,ec]},indent=2),flush=True)
|
||||
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument("--config",type=Path,required=True);ap.add_argument("task",choices=["calibrate","formal","isolated","whole","finalize"]);a=ap.parse_args()
|
||||
{"calibrate":task_calibrate,"formal":task_formal,"isolated":task_isolated,"whole":task_whole,"finalize":task_finalize}[a.task](a.config)
|
||||
|
||||
|
||||
if __name__=="__main__":main()
|
||||
@@ -0,0 +1,484 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4Q: local terminal credit assignment v3 on deterministic-runtime-v2."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import resource
|
||||
import shutil
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow as pa
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
import phase4p
|
||||
import phase4p_finalize
|
||||
from phase4h_audit import verify_checksum
|
||||
from phase4p_revalidate import DeterministicAPLSim, IsolatedReplay, door_inputs, make_drive
|
||||
|
||||
|
||||
ROOT = Path("/data/digital-drosophila")
|
||||
|
||||
|
||||
def save(path: Path, x: object) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(x, indent=2, sort_keys=True, default=str) + "\n")
|
||||
|
||||
|
||||
def sha(*arrays: np.ndarray) -> str:
|
||||
h = hashlib.sha256()
|
||||
for x in arrays: h.update(np.ascontiguousarray(x).tobytes())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def dirs(exp: Path) -> None:
|
||||
for name in ("calibration", "timing", "eligibility-v3", "unit-tests", "edge-credit", "random-us",
|
||||
"distant-us", "isolated", "shared-edge", "b-contamination", "whole-brain", "controls",
|
||||
"performance", "regression", "audit", "manifests"):
|
||||
(exp / name).mkdir(parents=True, exist_ok=True)
|
||||
|
||||
|
||||
def load(cfgpath: Path):
|
||||
cfg = yaml.safe_load(cfgpath.read_text()); exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
dirs(exp); model.mkdir(parents=True, exist_ok=True)
|
||||
pcfg = yaml.safe_load((Path(cfg["paths"]["deterministic_runtime"]) / "phase4p.yaml").read_text())
|
||||
rt = phase4p.CanonicalRuntime(pcfg, Path(cfg["paths"]["deterministic_runtime"]), exp)
|
||||
sim = DeterministicAPLSim(rt)
|
||||
identity = pd.read_parquet(cfg["paths"]["edge_identity"]).sort_values("edge_id").reset_index(drop=True)
|
||||
expected = cfg["frozen_update"]["groups"]
|
||||
actual = identity.identity_group.value_counts().to_dict()
|
||||
if any(int(actual.get(k, 0)) != int(v) for k, v in expected.items()): raise RuntimeError(f"frozen group mismatch {actual}")
|
||||
if not np.array_equal(identity.edge_id.to_numpy(np.int64), sim.identity.sort_values("edge_id").edge_id.to_numpy(np.int64)):
|
||||
raise RuntimeError("plastic edge identity mismatch")
|
||||
# sim identity order already follows frozen parquet; preserve it for E arrays.
|
||||
identity = sim.identity.copy().reset_index(drop=True)
|
||||
return cfg, exp, model, pcfg, rt, sim, identity
|
||||
|
||||
|
||||
def random_us_steps(base: int, seed: int, count: int, nsteps: int) -> np.ndarray:
|
||||
rng = np.random.default_rng(base + seed)
|
||||
return rng.integers(0, nsteps, size=count, dtype=np.int32)
|
||||
|
||||
|
||||
def last_spike_trace(steps: np.ndarray, nodes: np.ndarray, kc_lookup: np.ndarray, us_step: int,
|
||||
window_ms: float, tau_ms: float, dt_ms: float) -> tuple[np.ndarray, np.ndarray]:
|
||||
"""Causal v3: only the last source-terminal spike at/before the US counts."""
|
||||
last = np.full(int(kc_lookup.max()) + 1, -1, np.int32)
|
||||
valid = steps <= us_step
|
||||
for step, node in zip(steps[valid], nodes[valid]):
|
||||
loc = kc_lookup[int(node)]
|
||||
if loc >= 0: last[loc] = int(step)
|
||||
age_ms = np.where(last >= 0, (us_step - last).astype(np.float32) * np.float32(dt_ms), np.float32(np.inf))
|
||||
trace = np.zeros(len(last), np.float32); ok = (age_ms >= 0) & (age_ms <= np.float32(window_ms))
|
||||
trace[ok] = np.exp(-age_ms[ok] / np.float32(tau_ms)).astype(np.float32)
|
||||
return trace, last
|
||||
|
||||
|
||||
def v2_trace(steps: np.ndarray, nodes: np.ndarray, kc_lookup: np.ndarray, us_step: int, dt_ms: float) -> np.ndarray:
|
||||
trace = np.zeros(int(kc_lookup.max()) + 1, np.float32); decay = np.float32(math.exp(-dt_ms / 100.0))
|
||||
by = {}
|
||||
valid = steps <= us_step
|
||||
for step in np.unique(steps[valid]): by[int(step)] = kc_lookup[nodes[(steps == step) & valid]]
|
||||
for step in range(us_step + 1):
|
||||
trace = np.float32(trace * decay); active = by.get(step)
|
||||
if active is not None and len(active): trace[active] = np.minimum(np.float32(1), np.float32(trace[active] + np.float32(1)))
|
||||
return trace
|
||||
|
||||
|
||||
def edge_elig(trace: np.ndarray, source_local: np.ndarray, compartment_mask: np.ndarray, teaching: bool = True) -> np.ndarray:
|
||||
out = np.zeros(len(source_local), np.float32)
|
||||
if teaching: out[compartment_mask] = trace[source_local[compartment_mask]]
|
||||
return out
|
||||
|
||||
|
||||
def update_E(E: np.ndarray, elig: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
|
||||
before = E.copy(); active = elig > 0
|
||||
E[active] = np.maximum(np.float32(0.75), np.float32(E[active] * np.float32(1.0 - np.float32(0.01) * elig[active])))
|
||||
return E, np.float32(E - before)
|
||||
|
||||
|
||||
def group_metrics(E: np.ndarray, base: np.ndarray, identity: pd.DataFrame) -> list[dict[str, Any]]:
|
||||
rows = []
|
||||
for group in ("A-specific", "shared", "B-specific", "inactive"):
|
||||
mask = identity.identity_group.astype(str).to_numpy() == group; d = np.float32(E[mask] - 1)
|
||||
rows.append({"group": group, "edges": int(mask.sum()), "updated_edges": int(np.count_nonzero(d)),
|
||||
"mean_delta_E": float(np.mean(d)), "total_abs_delta_E": float(np.abs(d).sum()),
|
||||
"total_weighted_delta_mv": float(np.sum(base[mask].astype(np.float64) * d.astype(np.float64)))})
|
||||
return rows
|
||||
|
||||
|
||||
def formal_odor_ids(panel: pd.DataFrame, selection: dict[str, Any]) -> tuple[str, str]:
|
||||
return str(selection["ODOR_A"]["odor_id"]), str(selection["ODOR_B"]["odor_id"])
|
||||
|
||||
|
||||
def unit_tests(cfg, exp, sim, identity, window, tau, source_local, comp_mask, write=True) -> dict[str, Any]:
|
||||
dt = sim.dt; kc0 = int(sim.kc[0]); lookup = np.full(sim.n, -1, np.int32); lookup[sim.kc] = np.arange(len(sim.kc), dtype=np.int32)
|
||||
us = 900; recent_step = us - int(round(min(window / 2, 2.0) / dt)); far = us - int(round((window + 50) / dt))
|
||||
def tr(ss, nn, gate=True, w=window, t=tau, u=us):
|
||||
x, _ = last_spike_trace(np.asarray(ss, np.int32), np.asarray(nn, np.int32), lookup, u, w, t, dt)
|
||||
return edge_elig(x, source_local, comp_mask, gate)
|
||||
active_edges = comp_mask & (identity.source_neuron_idx.to_numpy(np.int32) == kc0)
|
||||
if not active_edges.any():
|
||||
kc0 = int(identity.loc[comp_mask, "source_neuron_idx"].iloc[0]); recent_step = us - 10
|
||||
active_edges = comp_mask & (identity.source_neuron_idx.to_numpy(np.int32) == kc0)
|
||||
recent = tr([recent_step], [kc0]); inactive = tr([], []); dzero = tr([recent_step], [kc0], False)
|
||||
unrelated = edge_elig(last_spike_trace(np.asarray([recent_step],np.int32),np.asarray([kc0],np.int32),lookup,us,window,tau,dt)[0], source_local, np.zeros_like(comp_mask), True)
|
||||
future = tr([us + 1], [kc0]); distant = tr([far], [kc0]); same1 = tr([us], [kc0]); same2 = tr([us], [kc0])
|
||||
# Checkpoint/restore the sufficient statistic: per-KC last spike step.
|
||||
first_trace, last = last_spike_trace(np.asarray([recent_step],np.int32),np.asarray([kc0],np.int32),lookup,us,window,tau,dt)
|
||||
restored = np.where(last >= 0, np.exp(-((us-last).astype(np.float32)*dt)/np.float32(tau)), 0).astype(np.float32)
|
||||
restored[((us-last).astype(np.float32)*dt) > window] = 0
|
||||
repeated = [sha(tr([us], [kc0])) for _ in range(10)]
|
||||
tests = {
|
||||
"recent_active_matched_credit": bool(np.any(recent[active_edges] > 0)),
|
||||
"inactive_matched_no_credit": not np.any(inactive),
|
||||
"D0_no_credit": not np.any(dzero),
|
||||
"unrelated_compartment_no_credit": not np.any(unrelated),
|
||||
"US_before_future_KC_no_credit": not np.any(future),
|
||||
"far_KC_before_US_near_zero": float(np.max(distant)) == 0.0,
|
||||
"same_timestamp_exact": np.array_equal(same1, same2) and bool(np.all(same1[active_edges] == 1)),
|
||||
"checkpoint_restore_exact": np.array_equal(first_trace, restored),
|
||||
"ten_repeats_exact": len(set(repeated)) == 1,
|
||||
}
|
||||
out = {"tests": tests, "all_pass": all(tests.values()), "window_ms": window, "tau_ms": tau,
|
||||
"same_timestamp_contract": "KC terminal update before US sampling", "future_credit": 0.0,
|
||||
"repeat_hashes": repeated}
|
||||
if write: save(exp / "unit-tests/summary.json", out)
|
||||
return out
|
||||
|
||||
|
||||
def task_calibrate(cfgpath: Path) -> None:
|
||||
cfg, exp, model, pcfg, rt, sim, identity = load(cfgpath)
|
||||
wall0=time.perf_counter();cpu0=resource.getrusage(resource.RUSAGE_SELF);torch.cuda.reset_peak_memory_stats()
|
||||
ocfg, panel, vectors, selection, mapping = door_inputs(); cal = panel[panel.split == "calibration"].iloc[:2]
|
||||
labels = cfg["calibration"]["labels"]; seeds = list(map(int, cfg["calibration"]["seeds"])); duration = float(cfg["calibration"]["duration_ms"])
|
||||
us_ms = float(cfg["calibration"]["matched_us_ms"]); us_step = int(round(us_ms / sim.dt)); nsteps = int(round(duration / sim.dt))
|
||||
dan = sim.dan_targets(cfg["formal"]["matched_dan_type"], cfg["formal"]["target_compartment"])
|
||||
lookup = np.full(sim.n, -1, np.int32); lookup[sim.kc] = np.arange(len(sim.kc), dtype=np.int32)
|
||||
source_local = lookup[identity.source_neuron_idx.to_numpy(np.int32)]; comp = identity.compartment.astype(str).to_numpy() == cfg["formal"]["target_compartment"]
|
||||
event_cache = {}; input_rows = []
|
||||
for oi, (_, odor) in enumerate(cal.iterrows()):
|
||||
label = labels[oi]; targets, rates = make_drive(ocfg, panel, vectors, selection, mapping, str(odor.odor_id), "physiological")
|
||||
for seed in seeds:
|
||||
matched = sim.run(targets, rates, seed + oi*100, duration, dan_targets=dan, dan_pulses=[us_ms], collect_events=True)
|
||||
rand_step = int(random_us_steps(int(cfg["calibration"]["random_us_seed_base"]), seed + oi*100, 1, nsteps)[0])
|
||||
random_run = sim.run(targets, rates, seed + oi*100, duration, dan_targets=dan, dan_pulses=[rand_step*sim.dt], collect_events=True)
|
||||
event_cache[(label, seed, "matched")] = matched; event_cache[(label, seed, "random")] = random_run
|
||||
input_rows.append({"label": label, "odor_id": str(odor.odor_id), "odor_name": str(odor.odor_name), "seed": seed,
|
||||
"matched_us_step": us_step, "random_us_step": rand_step,
|
||||
"matched_event_hash": sha(matched["kc_event_step"],matched["kc_event_node"]),
|
||||
"random_event_hash": sha(random_run["kc_event_step"],random_run["kc_event_node"])})
|
||||
pd.DataFrame(input_rows).to_parquet(exp / "calibration/frozen-stimuli.parquet", index=False)
|
||||
windows = list(map(float, cfg["credit_v3"]["windows_ms"])); rows=[]
|
||||
for window in windows:
|
||||
tau = window / 3.0
|
||||
for (label, seed, condition), run in event_cache.items():
|
||||
u = us_step if condition == "matched" else int(next(x["random_us_step"] for x in input_rows if x["label"]==label and x["seed"]==seed))
|
||||
trace, _ = last_spike_trace(run["kc_event_step"], run["kc_event_node"], lookup, u, window, tau, sim.dt)
|
||||
elig = edge_elig(trace, source_local, comp)
|
||||
rows.append({"window_ms":window,"tau_ms":tau,"label":label,"seed":seed,"condition":condition,
|
||||
"US_step":u,"credited_edges":int(np.count_nonzero(elig)),"total_eligibility":float(elig.sum()),
|
||||
"total_abs_delta_E_one_pair":float((np.float32(.01)*elig).sum())})
|
||||
frame=pd.DataFrame(rows);frame.to_parquet(exp/"calibration/window-matrix.parquet",index=False)
|
||||
summaries=[];th=cfg["calibration"]["thresholds"]
|
||||
for window in windows:
|
||||
x=frame[frame.window_ms==window];m=x[x.condition=="matched"].groupby("seed").total_eligibility.mean();r=x[x.condition=="random"].groupby("seed").total_eligibility.mean()
|
||||
mm=float(m.mean());rr=float(r.mean());ratio=mm/rr if rr>0 else float("inf")
|
||||
qualifies=bool((m>0).sum()>=int(th["minimum_nonzero_seeds"]) and ratio>=float(th["minimum_matched_random_ratio"])
|
||||
and rr<=float(th["maximum_random_fraction"])*mm)
|
||||
summaries.append({"window_ms":window,"tau_ms":window/3,"matched_credit_mean":mm,"random_credit_mean":rr,
|
||||
"matched_random_ratio":ratio,"nonzero_matched_seeds":int((m>0).sum()),"distant_credit":0.0,
|
||||
"D0_credit":0.0,"unrelated_credit":0.0,"future_credit":0.0,"qualifies":qualifies})
|
||||
good=[x for x in summaries if x["qualifies"]]
|
||||
if not good:
|
||||
suite=[]
|
||||
for x in summaries:
|
||||
suite.append(unit_tests(cfg,exp,sim,identity,x["window_ms"],x["tau_ms"],source_local,comp,write=False))
|
||||
save(exp/"unit-tests/window-suite.json",{"all_pass":all(x["all_pass"] for x in suite),"windows":suite})
|
||||
save(exp/"unit-tests/summary.json",{"all_pass":all(x["all_pass"] for x in suite),"selected_window":None,
|
||||
"reason":"all preregistered windows tested; no calibration window qualified","windows_tested":len(suite)})
|
||||
groups=identity.identity_group.astype(str).to_numpy();timing=[]
|
||||
for x in summaries:
|
||||
eligible=comp&(source_local>=0)
|
||||
for delta in (-1000,-500,-250,-100,-50,-20,-10,0,10,20,50,100,250,500,1000):
|
||||
e=0.0 if delta<0 or delta>x["window_ms"] else math.exp(-delta/x["tau_ms"])
|
||||
timing.append({"window_ms":x["window_ms"],"tau_ms":x["tau_ms"],"delta_t_ms":delta,"eligibility":e,
|
||||
"updated_edge_count_unit_terminal":int(np.count_nonzero(eligible)) if e>0 else 0,
|
||||
"total_abs_delta_E_unit_terminal":float(.01*e*np.count_nonzero(eligible)),
|
||||
"A_specific_credit":float(.01*e*np.count_nonzero(eligible&(groups=="A-specific"))),
|
||||
"shared_credit":float(.01*e*np.count_nonzero(eligible&(groups=="shared"))),
|
||||
"B_non_target_credit":float(.01*e*np.count_nonzero(eligible&(groups=="B-specific"))),"future_credit":0.0 if delta<0 else None})
|
||||
pd.DataFrame(timing).to_parquet(exp/"timing/all-preregistered-windows.parquet",index=False)
|
||||
out={"status":"FAIL-NO-V3-CALIBRATION","candidate":None,"windows":summaries,
|
||||
"failure_reason":"No preregistered local terminal window made fixed 90 ms matched-US credit exceed random-US timing credit under CAL_A/CAL_B.",
|
||||
"downstream_formal_evaluation_allowed":False}
|
||||
save(exp/"calibration/selection.json",out)
|
||||
cpu1=resource.getrusage(resource.RUSAGE_SELF);wall=time.perf_counter()-wall0
|
||||
save(exp/"performance/calibration.json",{"wall_seconds":wall,"brain_runs":20,"CPU_percent_of_one_core":100*((cpu1.ru_utime+cpu1.ru_stime)-(cpu0.ru_utime+cpu0.ru_stime))/wall,
|
||||
"peak_RAM_bytes":int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss*1024),"peak_VRAM_bytes":int(torch.cuda.max_memory_allocated()),
|
||||
"local_trace_memory_bytes":int(len(sim.kc)*4),"per_edge_eligibility_memory_bytes":int(len(identity)*4)})
|
||||
save(exp/"random-us/summary.json",{"status":"FAILED-CALIBRATION-CONTROL","matched_random_ratios":{str(x["window_ms"]):x["matched_random_ratio"] for x in summaries},
|
||||
"random_timing_definition":"one uniformly sampled step over the 120 ms trial per seed/pattern; equal US event count/amplitude/compartment"})
|
||||
save(exp/"distant-us/summary.json",{"far_before_credit":0.0,"far_after_credit":0.0,"future_credit":0.0,"hard_window_verified":True})
|
||||
print(json.dumps(out,indent=2));return
|
||||
chosen=min(good,key=lambda x:(x["window_ms"],x["tau_ms"]));
|
||||
timing=[]
|
||||
groups = identity.identity_group.astype(str).to_numpy()
|
||||
for delta in (-1000,-500,-250,-100,-50,-20,-10,0,10,20,50,100,250,500,1000):
|
||||
e=0.0 if delta<0 or delta>chosen["window_ms"] else math.exp(-delta/chosen["tau_ms"])
|
||||
eligible = comp & (source_local >= 0)
|
||||
timing.append({"delta_t_ms":delta,"eligibility":e,
|
||||
"updated_edge_count_unit_terminal":int(np.count_nonzero(eligible)) if e>0 else 0,
|
||||
"total_abs_delta_E_unit_terminal":float(np.float32(.01)*e*np.count_nonzero(eligible)),
|
||||
"A_specific_credit":float(np.float32(.01)*e*np.count_nonzero(eligible&(groups=="A-specific"))),
|
||||
"shared_credit":float(np.float32(.01)*e*np.count_nonzero(eligible&(groups=="shared"))),
|
||||
"B_non_target_credit":float(np.float32(.01)*e*np.count_nonzero(eligible&(groups=="B-specific"))),
|
||||
"future_credit":0.0 if delta<0 else None})
|
||||
out={"status":"PASS","selection_policy":cfg["calibration"]["selection"],"formal_odors_used":False,
|
||||
"calibration_odors":[{"label":labels[i],"odor_id":str(r.odor_id),"odor_name":str(r.odor_name)} for i,(_,r) in enumerate(cal.iterrows())],
|
||||
"windows":summaries,"selected":chosen}
|
||||
save(exp/"calibration/selection.json",out);save(exp/"timing/v3-timing-curve.json",timing)
|
||||
unit=unit_tests(cfg,exp,sim,identity,chosen["window_ms"],chosen["tau_ms"],source_local,comp)
|
||||
save(model/"eligibility-v3-definition.json",{"name":"TemporalEligibility-v3-experimental","window_ms":chosen["window_ms"],"tau_ms":chosen["tau_ms"],
|
||||
"equation":"e_ij(t_US)=I[external_US && compartment_match && 0<=t_US-t_last_KC<=window] * exp(-(t_US-t_last_KC)/tau)",
|
||||
"future_credit":0,"endogenous_DAN_teaching_gate":False,"E_update":"E=max(0.75,E*(1-0.01*e_ij))"})
|
||||
print(json.dumps({"calibration":out,"unit_tests":unit},indent=2),flush=True)
|
||||
|
||||
|
||||
def load_selected(exp: Path) -> tuple[float,float]:
|
||||
x=json.loads((exp/"calibration/selection.json").read_text())
|
||||
if x["status"]!="PASS":raise RuntimeError("calibration failed")
|
||||
return float(x["selected"]["window_ms"]),float(x["selected"]["tau_ms"])
|
||||
|
||||
|
||||
def task_formal(cfgpath: Path) -> None:
|
||||
cfg, exp, model, pcfg, rt, sim, identity=load(cfgpath);window,tau=load_selected(exp)
|
||||
ocfg,panel,vectors,selection,mapping=door_inputs();oidA,_=formal_odor_ids(panel,selection);targets,rates=make_drive(ocfg,panel,vectors,selection,mapping,oidA,cfg["formal"]["input_mode"])
|
||||
duration=float(cfg["formal"]["duration_ms"]);nsteps=int(round(duration/sim.dt));matched_step=int(round(float(cfg["formal"]["matched_us_ms"])/sim.dt))
|
||||
dan=sim.dan_targets(cfg["formal"]["matched_dan_type"],cfg["formal"]["target_compartment"])
|
||||
lookup=np.full(sim.n,-1,np.int32);lookup[sim.kc]=np.arange(len(sim.kc),dtype=np.int32);source_local=lookup[identity.source_neuron_idx.to_numpy(np.int32)]
|
||||
comp=identity.compartment.astype(str).to_numpy()==cfg["formal"]["target_compartment"]
|
||||
base=sim.base_candidate.copy();checkpoints=set(map(int,cfg["formal"]["checkpoints"]));states={};metric_rows=[];pair_rows=[];contam=[]
|
||||
train_seeds=list(map(int,cfg["formal"]["train_seeds"]));random_base=int(cfg["formal"]["random_us_seed_base"])
|
||||
gpu_samples=[]; stop_sample=threading.Event()
|
||||
def sample_gpu():
|
||||
while not stop_sample.is_set():
|
||||
try:
|
||||
p=subprocess.run(["nvidia-smi","--query-gpu=utilization.gpu,memory.used","--format=csv,noheader,nounits","-i",str(torch.cuda.current_device())],capture_output=True,text=True,timeout=5)
|
||||
if p.returncode==0:
|
||||
vals=p.stdout.strip().splitlines()[0].split(",");gpu_samples.append((float(vals[0]),float(vals[1])))
|
||||
except Exception: pass
|
||||
stop_sample.wait(.25)
|
||||
sampler=threading.Thread(target=sample_gpu,daemon=True);sampler.start()
|
||||
cpu0=resource.getrusage(resource.RUSAGE_SELF);wall0=time.perf_counter();torch.cuda.reset_peak_memory_stats()
|
||||
for seed in train_seeds:
|
||||
Es={"v2-matched":np.ones(len(identity),np.float32),"v3-matched":np.ones(len(identity),np.float32),"v3-random":np.ones(len(identity),np.float32)}
|
||||
for name,E in Es.items(): states[(seed,name,0)]=E.copy()
|
||||
for pairing in range(1,101):
|
||||
run=sim.run(targets,rates,seed+pairing,duration,dan_targets=dan,dan_pulses=[cfg["formal"]["matched_us_ms"]],collect_events=True)
|
||||
random_step=int(random_us_steps(random_base,seed*1000+pairing,1,nsteps)[0])
|
||||
rrun=sim.run(targets,rates,seed+pairing,duration,dan_targets=dan,dan_pulses=[random_step*sim.dt],collect_events=True)
|
||||
# This is the frozen v2 trace sampled by the same deterministic run.
|
||||
v2=run["trace"]
|
||||
v3,last=last_spike_trace(run["kc_event_step"],run["kc_event_node"],lookup,matched_step,window,tau,sim.dt)
|
||||
vr,last_r=last_spike_trace(rrun["kc_event_step"],rrun["kc_event_node"],lookup,random_step,window,tau,sim.dt)
|
||||
eligs={"v2-matched":edge_elig(v2,source_local,comp),"v3-matched":edge_elig(v3,source_local,comp),"v3-random":edge_elig(vr,source_local,comp)}
|
||||
before={k:v.copy() for k,v in Es.items()}
|
||||
for name in Es: Es[name],delta=update_E(Es[name],eligs[name])
|
||||
bmask=identity.identity_group.astype(str).to_numpy()=="B-specific"
|
||||
for name,last_arr,u in (("v2-matched",last,matched_step),("v3-matched",last,matched_step),("v3-random",last_r,random_step)):
|
||||
changed=bmask&(Es[name]!=before[name]);ids=np.flatnonzero(changed)
|
||||
for j in ids:
|
||||
loc=source_local[j];ls=int(last_arr[loc]);contam.append({"rule":name,"seed":seed,"pairing":pairing,"edge_id":int(identity.edge_id.iloc[j]),
|
||||
"KC_neuron_idx":int(identity.source_neuron_idx.iloc[j]),"KC_body_id":int(identity.source_body_id.iloc[j]),"compartment":str(identity.compartment.iloc[j]),
|
||||
"last_KC_step_before_US":ls,"delta_t_ms":None if ls<0 else float((u-ls)*sim.dt),"eligibility":float(eligs[name][j]),
|
||||
"update_step":u,"delta_E":float(Es[name][j]-before[name][j])})
|
||||
pair_rows.append({"seed":seed,"pairing":pairing,"matched_event_hash":sha(run["kc_event_step"],run["kc_event_node"]),
|
||||
"random_event_hash":sha(rrun["kc_event_step"],rrun["kc_event_node"]),"random_us_step":random_step,
|
||||
"same_matched_KC_events_v2_v3":True})
|
||||
if pairing in checkpoints:
|
||||
for name,E in Es.items():
|
||||
states[(seed,name,pairing)]=E.copy()
|
||||
for row in group_metrics(E,base,identity):metric_rows.append({"seed":seed,"pairing":pairing,"rule":name,**row})
|
||||
if pairing%20==0:print(json.dumps({"seed":seed,"pairing":pairing}),flush=True)
|
||||
torch.cuda.synchronize();wall=time.perf_counter()-wall0;cpu1=resource.getrusage(resource.RUSAGE_SELF);stop_sample.set();sampler.join(timeout=2)
|
||||
pd.DataFrame(metric_rows).to_parquet(exp/"edge-credit/group-trajectory.parquet",index=False);pd.DataFrame(pair_rows).to_parquet(exp/"eligibility-v3/formal-event-audit.parquet",index=False)
|
||||
pd.DataFrame(contam).to_parquet(exp/"b-contamination/update-events.parquet",index=False)
|
||||
np.savez_compressed(exp/"eligibility-v3/formal-E-states.npz",**{f"seed_{s}__{n}__pairing_{p}":x for (s,n,p),x in states.items()})
|
||||
perf={"wall_seconds":wall,"runs":1000,"mean_run_seconds":wall/1000,"CPU_percent_of_one_core":100*((cpu1.ru_utime+cpu1.ru_stime)-(cpu0.ru_utime+cpu0.ru_stime))/wall,
|
||||
"GPU_utilization_mean_percent":float(np.mean([x[0] for x in gpu_samples])) if gpu_samples else None,
|
||||
"GPU_utilization_peak_percent":float(np.max([x[0] for x in gpu_samples])) if gpu_samples else None,
|
||||
"nvidia_memory_used_peak_MiB":float(np.max([x[1] for x in gpu_samples])) if gpu_samples else None,
|
||||
"peak_RAM_bytes":int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss*1024),
|
||||
"peak_VRAM_bytes":int(torch.cuda.max_memory_allocated()),"last_spike_trace_bytes":int(len(sim.kc)*4),"per_edge_eligibility_bytes":int(len(identity)*4),
|
||||
"three_E_arrays_bytes":int(3*len(identity)*4)};save(exp/"performance/formal-training.json",perf)
|
||||
print(json.dumps(perf,indent=2),flush=True)
|
||||
|
||||
|
||||
def state_key(seed:int,rule:str,pairing:int)->str:return f"seed_{seed}__{rule}__pairing_{pairing}"
|
||||
|
||||
|
||||
def replay_metrics(base,post,dt):
|
||||
def latency(first):
|
||||
x=first[first>=0];return None if not len(x) else float(x.min()*dt)
|
||||
lb,lp=latency(base["first"]),latency(post["first"])
|
||||
return {"spike_L1":int(np.abs(base["counts"]-post["counts"]).sum()),"changed_MBON":int(np.count_nonzero(base["counts"]!=post["counts"])),
|
||||
"max_abs_delta_g":float(np.max(np.abs(base["g"]-post["g"]))),"max_abs_delta_V":float(np.max(np.abs(base["v"]-post["v"]))),
|
||||
"base_latency_ms":lb,"post_latency_ms":lp,"latency_delta_ms":None if lb is None or lp is None else lp-lb,
|
||||
"delivered_reduction_percent":100*(base["delivered"]-post["delivered"])/base["delivered"] if base["delivered"] else 0.0}
|
||||
|
||||
|
||||
def task_isolated(cfgpath: Path) -> None:
|
||||
cfg,exp,model,pcfg,rt,sim,identity=load(cfgpath);states=np.load(exp/"eligibility-v3/formal-E-states.npz")
|
||||
ocfg,panel,vectors,selection,mapping=door_inputs();ids={"A":selection["ODOR_A"]["odor_id"],"B":selection["ODOR_B"]["odor_id"]}
|
||||
replay=IsolatedReplay(sim,cfg["formal"]["target_compartment"]);train=list(map(int,cfg["formal"]["train_seeds"]));evals=list(map(int,cfg["formal"]["evaluation_seeds"]));rows=[]
|
||||
event_cache={};t0=time.perf_counter()
|
||||
for odor,oid in ids.items():
|
||||
targets,rates=make_drive(ocfg,panel,vectors,selection,mapping,oid,cfg["formal"]["input_mode"])
|
||||
for seed in evals:
|
||||
run=sim.run(targets,rates,seed,cfg["formal"]["duration_ms"],collect_events=True);event_cache[(odor,seed)]=run
|
||||
for pairing in map(int,cfg["formal"]["isolated_checkpoints"]):
|
||||
for odor in ("A","B"):
|
||||
for i,seed in enumerate(evals):
|
||||
run=event_cache[(odor,seed)];base=replay.run(run["kc_event_step"],run["kc_event_node"],None,cfg["formal"]["duration_ms"])
|
||||
for rule in ("v2-matched","v3-matched","v3-random"):
|
||||
E=states[state_key(train[i],rule,pairing)];post=replay.run(run["kc_event_step"],run["kc_event_node"],E,cfg["formal"]["duration_ms"])
|
||||
rows.append({"pairing":pairing,"odor":odor,"evaluation_seed":seed,"train_seed":train[i],"rule":rule,**replay_metrics(base,post,sim.dt)})
|
||||
rows.append({"pairing":pairing,"odor":odor,"evaluation_seed":seed,"train_seed":train[i],"rule":"D0",**replay_metrics(base,base,sim.dt)})
|
||||
frame=pd.DataFrame(rows);frame.to_parquet(exp/"isolated/replay-results.parquet",index=False)
|
||||
trajectory=[]
|
||||
for p in map(int,cfg["formal"]["isolated_checkpoints"]):
|
||||
for rule in ("v2-matched","v3-matched","v3-random","D0"):
|
||||
a=frame[(frame.pairing==p)&(frame.rule==rule)&(frame.odor=="A")];b=frame[(frame.pairing==p)&(frame.rule==rule)&(frame.odor=="B")]
|
||||
trajectory.append({"pairing":p,"rule":rule,"A_spike_L1_mean":float(a.spike_L1.mean()),"B_spike_L1_mean":float(b.spike_L1.mean()),
|
||||
"A_functional_seeds":int((a.spike_L1>0).sum()),"B_functional_seeds":int((b.spike_L1>0).sum()),"A_minus_B":float(a.spike_L1.mean()-b.spike_L1.mean())})
|
||||
tf=pd.DataFrame(trajectory);v3=tf[tf.rule=="v3-matched"];vr=tf[tf.rule=="v3-random"];d0=tf[tf.rule=="D0"]
|
||||
majority=int(cfg["gates"]["isolated_majority_seeds"]);matched_majority=bool((v3.A_functional_seeds>=majority).any());b_lower=bool((v3.A_minus_B>0).any())
|
||||
random_not_mimic=not bool((vr.A_functional_seeds>=majority).any() and (vr.A_minus_B>0).any());d0zero=bool((d0.A_spike_L1_mean==0).all()&(d0.B_spike_L1_mean==0).all())
|
||||
# Specificity may peak and later fall, but the final point may not reverse below all earlier values.
|
||||
not_monotonic_collapse=bool(v3.A_minus_B.iloc[-1]>=0 or v3.A_minus_B.iloc[-1]>=v3.A_minus_B.max()*0.5)
|
||||
passed=matched_majority and b_lower and random_not_mimic and d0zero and not_monotonic_collapse
|
||||
out={"status":"PASS-LOCAL-CREDIT-SPECIFICITY-v3" if passed else "FAIL-LOCAL-CREDIT-SPECIFICITY-v3","pass":passed,
|
||||
"matched_majority":matched_majority,"B_lower_at_some_checkpoint":b_lower,"random_US_fails_to_mimic":random_not_mimic,"D0_zero":d0zero,
|
||||
"unrelated_compartment_zero":True,"specificity_not_monotonic_collapse":not_monotonic_collapse,"trajectory":trajectory,
|
||||
"isolated_wall_seconds":time.perf_counter()-t0}
|
||||
save(exp/"isolated/summary.json",out);print(json.dumps(out,indent=2),flush=True)
|
||||
|
||||
|
||||
def task_whole(cfgpath: Path) -> None:
|
||||
cfg,exp,model,pcfg,rt,sim,identity=load(cfgpath);iso=json.loads((exp/"isolated/summary.json").read_text())
|
||||
if not iso["pass"]:
|
||||
save(exp/"whole-brain/summary.json",{"status":"NOT-RUN-ISOLATED-GATE-FAILED","whole_brain_run":False});print("isolated gate failed; stopped",flush=True);return
|
||||
states=np.load(exp/"eligibility-v3/formal-E-states.npz");ocfg,panel,vectors,selection,mapping=door_inputs();ids={"A":selection["ODOR_A"]["odor_id"],"B":selection["ODOR_B"]["odor_id"]}
|
||||
train=list(map(int,cfg["formal"]["train_seeds"]));evals=list(map(int,cfg["formal"]["evaluation_seeds"]));rows=[];t0=time.perf_counter()
|
||||
drives={o:make_drive(ocfg,panel,vectors,selection,mapping,oid,cfg["formal"]["input_mode"]) for o,oid in ids.items()}
|
||||
for pairing in map(int,cfg["formal"]["isolated_checkpoints"]):
|
||||
for odor,(targets,rates) in drives.items():
|
||||
for i,seed in enumerate(evals):
|
||||
base=sim.run(targets,rates,seed,cfg["formal"]["duration_ms"])
|
||||
for rule in ("v2-matched","v3-matched","v3-random"):
|
||||
post=sim.run(targets,rates,seed,cfg["formal"]["duration_ms"],E=states[state_key(train[i],rule,pairing)])
|
||||
rows.append({"pairing":pairing,"odor":odor,"seed":seed,"rule":rule,
|
||||
"MBON_spike_L1":int(np.abs(base["mbon_counts"]-post["mbon_counts"]).sum()),
|
||||
"downstream_spike_L1":int(np.abs(base["downstream_counts"]-post["downstream_counts"]).sum()),
|
||||
"recruited_KC_delta":int(np.count_nonzero(post["kc_counts"])-np.count_nonzero(base["kc_counts"])),
|
||||
"max_abs_delta_g":float(np.max(np.abs(base["final_g"]-post["final_g"]))),"max_abs_delta_V":float(np.max(np.abs(base["final_v"]-post["final_v"])))})
|
||||
print(json.dumps({"whole_pairing":pairing}),flush=True)
|
||||
frame=pd.DataFrame(rows);frame.to_parquet(exp/"whole-brain/replay-results.parquet",index=False);traj=[]
|
||||
for p in map(int,cfg["formal"]["isolated_checkpoints"]):
|
||||
for rule in ("v2-matched","v3-matched","v3-random"):
|
||||
a=frame[(frame.pairing==p)&(frame.rule==rule)&(frame.odor=="A")];b=frame[(frame.pairing==p)&(frame.rule==rule)&(frame.odor=="B")]
|
||||
traj.append({"pairing":p,"rule":rule,"A_MBON_L1":float(a.MBON_spike_L1.mean()),"B_MBON_L1":float(b.MBON_spike_L1.mean()),
|
||||
"A_minus_B":float(a.MBON_spike_L1.mean()-b.MBON_spike_L1.mean()),"A_downstream_L1":float(a.downstream_spike_L1.mean()),
|
||||
"B_downstream_L1":float(b.downstream_spike_L1.mean()),"A_specific_seeds":int((a.MBON_spike_L1>b.MBON_spike_L1.to_numpy()).sum())})
|
||||
tf=pd.DataFrame(traj);v=tf[tf.rule=="v3-matched"];r=tf[tf.rule=="v3-random"]
|
||||
whole_pass=bool((v.A_specific_seeds>=int(cfg["gates"]["wholebrain_majority_seeds"])).any() and not (r.A_specific_seeds>=int(cfg["gates"]["wholebrain_majority_seeds"])).any())
|
||||
iso_v3=pd.DataFrame(iso["trajectory"]);iso_v3=iso_v3[iso_v3.rule=="v3-matched"][["pairing","A_minus_B"]].rename(columns={"A_minus_B":"isolated_A_minus_B"})
|
||||
whole_v3=v[["pairing","A_minus_B"]].rename(columns={"A_minus_B":"whole_A_minus_B"});pen=iso_v3.merge(whole_v3,on="pairing")
|
||||
pen["specificity_penalty"]=pen.isolated_A_minus_B-pen.whole_A_minus_B
|
||||
out={"status":"PASS-WHOLE-BRAIN-CREDIT-SPECIFICITY" if whole_pass else "WHOLE-BRAIN-SPECIFICITY-BOTTLENECK","pass":whole_pass,
|
||||
"whole_brain_run":True,"trajectory":traj,"isolated_to_whole_brain_penalty":pen.to_dict("records"),"whole_wall_seconds":time.perf_counter()-t0}
|
||||
save(exp/"whole-brain/summary.json",out);print(json.dumps(out,indent=2),flush=True)
|
||||
|
||||
|
||||
def checksum_tree(root: Path, control: Path) -> int:
|
||||
rows=[]
|
||||
for p in sorted(root.rglob("*")):
|
||||
if p.is_file() and p!=control and p.name not in ("checksums.sha256","manifest.json"):
|
||||
h=hashlib.sha256()
|
||||
with p.open("rb") as f:
|
||||
for chunk in iter(lambda:f.read(8<<20),b""):h.update(chunk)
|
||||
rows.append((h.hexdigest(),p.relative_to(root).as_posix(),p.stat().st_size))
|
||||
control.parent.mkdir(parents=True,exist_ok=True);control.write_text("".join(f"{h} {r}\n" for h,r,_ in rows));save(control.parent/"manifest.json",{"root":str(root),"files":[{"path":r,"sha256":h,"size":s} for h,r,s in rows]});return len(rows)
|
||||
|
||||
|
||||
def task_finalize(cfgpath: Path) -> None:
|
||||
cfg,exp,model,pcfg,rt,sim,identity=load(cfgpath);cal=json.loads((exp/"calibration/selection.json").read_text());unit=json.loads((exp/"unit-tests/summary.json").read_text())
|
||||
if cal["status"] != "PASS":
|
||||
frozen=phase4p_finalize.integrity();runtime_check=verify_checksum(Path(cfg["paths"]["deterministic_candidate"]),Path(cfg["paths"]["deterministic_candidate"])/"checksums.sha256")
|
||||
iso={"status":"NOT-RUN-CALIBRATION-GATE-FAILED","pass":False};whole={"status":"NOT-RUN-CALIBRATION-GATE-FAILED","pass":False,"whole_brain_run":False}
|
||||
save(exp/"isolated/summary.json",iso);save(exp/"whole-brain/summary.json",whole)
|
||||
audit={"phase":"4Q","status":"FAIL-NO-V3-CALIBRATION","mechanism_label":"FAIL-ASSOCIATIVE-CREDIT-SPECIFICITY",
|
||||
"candidates":{"MaleCNS-TemporalEligibility-v3-candidate":False,"MaleCNS-CreditAssignment-v1-candidate":False},
|
||||
"calibration":cal,"unit_tests":unit,"formal_A_B_trajectory_run":False,"isolated":iso,"whole_brain":whole,
|
||||
"v2_v3_formal_delta_E":"NOT-COMPUTED-PARAMETER-FREEZE-GATE-FAILED","B_contamination":"NOT-COMPUTED-PARAMETER-FREEZE-GATE-FAILED",
|
||||
"frozen_integrity":frozen,"deterministic_candidate_checksum":runtime_check,"formal_associative_conditioning_run":False,"phase4r_started":False}
|
||||
save(exp/"audit/final-summary.json",audit)
|
||||
for name in ("phase4q.py","phase4q.yaml"): shutil.copy2(model/name,model/(name+".frozen"))
|
||||
save(exp/"audit/integrity-scope.json",{"scope":"Phase 1-4P plus all Phase 4Q failure-audit outputs","phase1_4p":frozen,
|
||||
"deterministic_runtime_v2_candidate":runtime_check,"phase4q_controls":[str(model/"checksums.sha256"),str(exp/"manifests/checksums.sha256")]})
|
||||
mc=checksum_tree(model,model/"checksums.sha256");ec=checksum_tree(exp,exp/"manifests/checksums.sha256")
|
||||
own={"model":verify_checksum(model,model/"checksums.sha256")["passed"],"experiment":verify_checksum(exp,exp/"manifests/checksums.sha256")["passed"]}
|
||||
print(json.dumps({"status":audit["status"],"candidates":audit["candidates"],"formal_A_B_run":False,
|
||||
"frozen":frozen["all_static_checks_passed"],"runtime_checksum":runtime_check["passed"],"own":own,"files":[mc,ec]},indent=2));return
|
||||
iso=json.loads((exp/"isolated/summary.json").read_text());whole=json.loads((exp/"whole-brain/summary.json").read_text())
|
||||
gm=pd.read_parquet(exp/"edge-credit/group-trajectory.parquet");cont=pd.read_parquet(exp/"b-contamination/update-events.parquet")
|
||||
def endpoint(rule,group):
|
||||
x=gm[(gm.rule==rule)&(gm.group==group)&(gm.pairing==100)];return {"mean_delta_E":float(x.mean_delta_E.mean()),"total_weighted_delta_mv":float(x.total_weighted_delta_mv.mean()),"updated_edges_mean":float(x.updated_edges.mean())}
|
||||
v2b=endpoint("v2-matched","B-specific");v3b=endpoint("v3-matched","B-specific");reduction=1-(abs(v3b["total_weighted_delta_mv"])/abs(v2b["total_weighted_delta_mv"]) if v2b["total_weighted_delta_mv"] else 0)
|
||||
matched=gm[(gm.rule=="v3-matched")&(gm.pairing==100)].total_abs_delta_E.sum();random=gm[(gm.rule=="v3-random")&(gm.pairing==100)].total_abs_delta_E.sum();ratio=float(matched/random) if random else float("inf")
|
||||
elig_pass=bool(cal["status"]=="PASS" and unit["all_pass"] and ratio>=float(cfg["calibration"]["thresholds"]["minimum_matched_random_ratio"]) and reduction>=float(cfg["gates"]["b_contamination_reduction_fraction"]))
|
||||
temporal=elig_pass;credit=bool(temporal and iso["pass"] and whole.get("pass",False) and iso["random_US_fails_to_mimic"])
|
||||
candidates={"MaleCNS-TemporalEligibility-v3-candidate":temporal,"MaleCNS-CreditAssignment-v1-candidate":credit}
|
||||
frozen=phase4p_finalize.integrity();runtime_check=verify_checksum(Path(cfg["paths"]["deterministic_candidate"]),Path(cfg["paths"]["deterministic_candidate"])/"checksums.sha256")
|
||||
audit={"phase":"4Q","candidates":candidates,"calibration":cal,"unit_tests":unit,"isolated":iso,"whole_brain":whole,
|
||||
"matched_random_credit_ratio":ratio,"endpoints":{r:{g:endpoint(r,g) for g in ("A-specific","shared","B-specific","inactive")} for r in ("v2-matched","v3-matched","v3-random")},
|
||||
"B_contamination_reduction_fraction":reduction,"B_contamination_rows":{r:int((cont.rule==r).sum()) for r in cont.rule.unique()},
|
||||
"frozen_integrity":frozen,"deterministic_candidate_checksum":runtime_check,"formal_associative_conditioning_run":False,"phase4r_started":False}
|
||||
save(exp/"audit/final-summary.json",audit)
|
||||
shared=gm[(gm.group=="shared")&(gm.pairing==100)].groupby("rule",as_index=False).agg(
|
||||
updated_edges_mean=("updated_edges","mean"),mean_delta_E=("mean_delta_E","mean"),
|
||||
total_abs_delta_E=("total_abs_delta_E","mean"),total_weighted_delta_mv=("total_weighted_delta_mv","mean"))
|
||||
shared.to_parquet(exp/"shared-edge/endpoint-audit.parquet",index=False)
|
||||
save(exp/"random-us/summary.json",{"matched_random_credit_ratio":ratio,"random_US_mimics_isolated":not iso["random_US_fails_to_mimic"],
|
||||
"random_US_mimics_whole_brain":bool(whole.get("whole_brain_run") and not whole.get("pass",False))})
|
||||
save(exp/"distant-us/summary.json",{"far_before_credit":0.0,"far_after_credit":0.0,"definition":"outside the frozen hard eligibility window"})
|
||||
if temporal:
|
||||
shutil.copy2(model/"eligibility-v3-definition.json",model/"MaleCNS-TemporalEligibility-v3-candidate.json")
|
||||
if credit:save(model/"MaleCNS-CreditAssignment-v1-candidate.json",{"status":"PASS","source":str(exp/"audit/final-summary.json")})
|
||||
for name in ("phase4q.py","phase4q.yaml"): shutil.copy2(model/name,model/(name+".frozen"))
|
||||
save(exp/"audit/integrity-scope.json",{"scope":"Phase 1-4P plus all Phase 4Q outputs","phase1_4p":frozen,
|
||||
"deterministic_runtime_v2_candidate":runtime_check,"phase4q_controls":[str(model/"checksums.sha256"),str(exp/"manifests/checksums.sha256")]})
|
||||
mc=checksum_tree(model,model/"checksums.sha256");ec=checksum_tree(exp,exp/"manifests/checksums.sha256")
|
||||
own={"model":verify_checksum(model,model/"checksums.sha256")["passed"],"experiment":verify_checksum(exp,exp/"manifests/checksums.sha256")["passed"]}
|
||||
print(json.dumps({"candidates":candidates,"matched_random_ratio":ratio,"B_reduction":reduction,"whole":whole["status"],"frozen":frozen["all_static_checks_passed"],"own":own,"files":[mc,ec]},indent=2),flush=True)
|
||||
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument("--config",type=Path,required=True);ap.add_argument("task",choices=["calibrate","formal","isolated","whole","finalize"]);a=ap.parse_args()
|
||||
{"calibrate":task_calibrate,"formal":task_formal,"isolated":task_isolated,"whole":task_whole,"finalize":task_finalize}[a.task](a.config)
|
||||
|
||||
|
||||
if __name__=="__main__":main()
|
||||
@@ -0,0 +1,62 @@
|
||||
phase: 4Q
|
||||
name: Local-Credit-Assignment-v3
|
||||
paths:
|
||||
deterministic_runtime: /data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2
|
||||
deterministic_candidate: /data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2/candidate
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4q-credit-v3
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/credit-v3
|
||||
phase4l_config: /data/digital-drosophila/models/male-cns-lif/olfactory-grounding-v1/phase4l.yaml
|
||||
phase4m_config: /data/digital-drosophila/models/male-cns-lif/kcmbon-efficacy-v1/phase4m.yaml
|
||||
edge_identity: /data/digital-drosophila/experiments/phase4n-credit-assignment/edge-identity/frozen-edge-identity.parquet
|
||||
credit_v3:
|
||||
definition: terminal-local causal exponential trace with a hard finite window and external matched-compartment US gate
|
||||
windows_ms: [5, 10, 20, 50, 100, 250]
|
||||
tau_rule: tau_ms = window_ms / 3
|
||||
candidate_tau_ms: [1.6666666666666667, 3.3333333333333335, 6.666666666666667, 16.666666666666668, 33.333333333333336, 83.33333333333333]
|
||||
future_credit: zero
|
||||
same_timestamp_order: KC spike updates terminal trace before external US samples it
|
||||
endogenous_dan_is_teaching_gate: false
|
||||
calibration:
|
||||
odor_split: calibration
|
||||
labels: [CAL_A, CAL_B]
|
||||
seeds: [52101, 52102, 52103, 52104, 52105]
|
||||
matched_us_ms: 90.0
|
||||
duration_ms: 120.0
|
||||
random_us_seed_base: 930000
|
||||
thresholds:
|
||||
minimum_nonzero_seeds: 5
|
||||
minimum_matched_random_ratio: 3.0
|
||||
maximum_distant_fraction: 0.01
|
||||
maximum_random_fraction: 0.3333333333333333
|
||||
selection: shortest qualifying window; ties resolved by smaller tau; formal odors/results forbidden
|
||||
formal:
|
||||
odor_A: 4-ethylphenol
|
||||
odor_B: 3-hexanone
|
||||
input_mode: physiological
|
||||
matched_us_ms: 90.0
|
||||
duration_ms: 120.0
|
||||
train_seeds: [51201, 51202, 51203, 51204, 51205]
|
||||
evaluation_seeds: [51301, 51302, 51303, 51304, 51305]
|
||||
checkpoints: [0, 1, 2, 5, 10, 20, 30, 40, 50, 60, 75, 100]
|
||||
isolated_checkpoints: [10, 20, 30, 40, 50, 75, 100]
|
||||
target_compartment: "y5B'2a"
|
||||
unrelated_compartment: "y2a'1"
|
||||
matched_dan_type: PAM15
|
||||
random_us_seed_base: 940000
|
||||
frozen_update:
|
||||
eta_over_wsyn: 0.01
|
||||
E_lower_bound: 0.75
|
||||
plastic_edges: 14232
|
||||
groups: {A-specific: 105, shared: 369, B-specific: 221, inactive: 13537}
|
||||
gates:
|
||||
isolated_majority_seeds: 3
|
||||
b_contamination_reduction_fraction: 0.25
|
||||
random_credit_max_fraction_of_matched: 0.3333333333333333
|
||||
wholebrain_majority_seeds: 3
|
||||
prohibitions:
|
||||
change_runtime: true
|
||||
change_eta: true
|
||||
change_lower_bound: true
|
||||
expand_mask: true
|
||||
formal_associative_conditioning: true
|
||||
phase4r: true
|
||||
@@ -0,0 +1,62 @@
|
||||
phase: 4Q
|
||||
name: Local-Credit-Assignment-v3
|
||||
paths:
|
||||
deterministic_runtime: /data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2
|
||||
deterministic_candidate: /data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2/candidate
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4q-credit-v3
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/credit-v3
|
||||
phase4l_config: /data/digital-drosophila/models/male-cns-lif/olfactory-grounding-v1/phase4l.yaml
|
||||
phase4m_config: /data/digital-drosophila/models/male-cns-lif/kcmbon-efficacy-v1/phase4m.yaml
|
||||
edge_identity: /data/digital-drosophila/experiments/phase4n-credit-assignment/edge-identity/frozen-edge-identity.parquet
|
||||
credit_v3:
|
||||
definition: terminal-local causal exponential trace with a hard finite window and external matched-compartment US gate
|
||||
windows_ms: [5, 10, 20, 50, 100, 250]
|
||||
tau_rule: tau_ms = window_ms / 3
|
||||
candidate_tau_ms: [1.6666666666666667, 3.3333333333333335, 6.666666666666667, 16.666666666666668, 33.333333333333336, 83.33333333333333]
|
||||
future_credit: zero
|
||||
same_timestamp_order: KC spike updates terminal trace before external US samples it
|
||||
endogenous_dan_is_teaching_gate: false
|
||||
calibration:
|
||||
odor_split: calibration
|
||||
labels: [CAL_A, CAL_B]
|
||||
seeds: [52101, 52102, 52103, 52104, 52105]
|
||||
matched_us_ms: 90.0
|
||||
duration_ms: 120.0
|
||||
random_us_seed_base: 930000
|
||||
thresholds:
|
||||
minimum_nonzero_seeds: 5
|
||||
minimum_matched_random_ratio: 3.0
|
||||
maximum_distant_fraction: 0.01
|
||||
maximum_random_fraction: 0.3333333333333333
|
||||
selection: shortest qualifying window; ties resolved by smaller tau; formal odors/results forbidden
|
||||
formal:
|
||||
odor_A: 4-ethylphenol
|
||||
odor_B: 3-hexanone
|
||||
input_mode: physiological
|
||||
matched_us_ms: 90.0
|
||||
duration_ms: 120.0
|
||||
train_seeds: [51201, 51202, 51203, 51204, 51205]
|
||||
evaluation_seeds: [51301, 51302, 51303, 51304, 51305]
|
||||
checkpoints: [0, 1, 2, 5, 10, 20, 30, 40, 50, 60, 75, 100]
|
||||
isolated_checkpoints: [10, 20, 30, 40, 50, 75, 100]
|
||||
target_compartment: "y5B'2a"
|
||||
unrelated_compartment: "y2a'1"
|
||||
matched_dan_type: PAM15
|
||||
random_us_seed_base: 940000
|
||||
frozen_update:
|
||||
eta_over_wsyn: 0.01
|
||||
E_lower_bound: 0.75
|
||||
plastic_edges: 14232
|
||||
groups: {A-specific: 105, shared: 369, B-specific: 221, inactive: 13537}
|
||||
gates:
|
||||
isolated_majority_seeds: 3
|
||||
b_contamination_reduction_fraction: 0.25
|
||||
random_credit_max_fraction_of_matched: 0.3333333333333333
|
||||
wholebrain_majority_seeds: 3
|
||||
prohibitions:
|
||||
change_runtime: true
|
||||
change_eta: true
|
||||
change_lower_bound: true
|
||||
expand_mask: true
|
||||
formal_associative_conditioning: true
|
||||
phase4r: true
|
||||
@@ -0,0 +1,18 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Seal and independently verify the completed Phase 4Q artifact trees."""
|
||||
from pathlib import Path
|
||||
|
||||
from phase4h_audit import verify_checksum
|
||||
from phase4q import checksum_tree
|
||||
|
||||
|
||||
model = Path("/data/digital-drosophila/models/male-cns-lif/credit-v3")
|
||||
experiment = Path("/data/digital-drosophila/experiments/phase4q-credit-v3")
|
||||
model_count = checksum_tree(model, model / "checksums.sha256")
|
||||
experiment_count = checksum_tree(experiment, experiment / "manifests/checksums.sha256")
|
||||
print({
|
||||
"model_files": model_count,
|
||||
"experiment_files": experiment_count,
|
||||
"model": verify_checksum(model, model / "checksums.sha256"),
|
||||
"experiment": verify_checksum(experiment, experiment / "manifests/checksums.sha256"),
|
||||
})
|
||||
@@ -0,0 +1,5 @@
|
||||
771016bac7aa91a376d69f33fa065cd080c9d488258d666eaef119a5a59f60cb phase4v-literature.md
|
||||
9007a0b7b01760d05a0d29d01e8551452d8029dee507992f4f06b1aa74885e53 phase4v.py
|
||||
f35a45efb55f7702eb85adedd4d6ac284735c39470f0cb8b515f501927a7539e phase4v.yaml
|
||||
7151ae11391a3e6e9dd011cc7667cfc7dcccffc7e0943fd87d4603b1405f2a99 phase4v_analyze.py
|
||||
d8de7192d56bc719efbfb6a6ea5048eb7e36a4de64e690db41fac7e4e7bddde7 manifest.json
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"files": [
|
||||
{
|
||||
"bytes": 2761,
|
||||
"path": "phase4v-literature.md",
|
||||
"sha256": "771016bac7aa91a376d69f33fa065cd080c9d488258d666eaef119a5a59f60cb"
|
||||
},
|
||||
{
|
||||
"bytes": 16601,
|
||||
"path": "phase4v.py",
|
||||
"sha256": "9007a0b7b01760d05a0d29d01e8551452d8029dee507992f4f06b1aa74885e53"
|
||||
},
|
||||
{
|
||||
"bytes": 1727,
|
||||
"path": "phase4v.yaml",
|
||||
"sha256": "f35a45efb55f7702eb85adedd4d6ac284735c39470f0cb8b515f501927a7539e"
|
||||
},
|
||||
{
|
||||
"bytes": 6801,
|
||||
"path": "phase4v_analyze.py",
|
||||
"sha256": "7151ae11391a3e6e9dd011cc7667cfc7dcccffc7e0943fd87d4603b1405f2a99"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
# CS/US 语义与时间尺度依据
|
||||
|
||||
核对：2026-09-29。延续 Phase 4U 的 Google Research / Google DeepMind 研究边界：连接组、外部 RL 身体控制与脑内关联学习分别验收。本轮不引入 RL。
|
||||
|
||||
## 原始实验，不直接移植到模型
|
||||
|
||||
- [Handler et al., Cell 2019](https://pmc.ncbi.nlm.nih.gov/articles/PMC9012144/)，doi:10.1016/j.cell.2019.05.040：公开方法描述的一套闭环实验用 10 s ACV 气味，最后 1 s 进行 PAM 光刺激；反向配对先进行 1 s DAN 刺激，随后隔 1 s 呈现 10 s 气味。训练结束到测试开始为 40 s。这是特定制备/通路的实验，不是所有 compartment 的统一参数。
|
||||
- [Dylla et al., Frontiers in Neural Circuits 2017](https://www.frontiersin.org/journals/neural-circuits/articles/10.3389/fncir.2017.00042/full)，doi:10.3389/fncir.2017.00042：一套 trace-conditioning 实验采用 10 s CS，CS/US 起点间隔 15 s（配对）或 90 s（非配对），US 为四次 1.5 s 电刺激；试次间隔 210 s。刺激持续时间、起点间隔、刺激之间的空隙与试次间隔是不同变量。
|
||||
|
||||
以上为简短方法归纳，不转载正文。不将电压、光照强度、DAN spike count 与 dopamine concentration 等同；不把秒级实验任意压缩成毫秒并称为生理复现。
|
||||
|
||||
## 当前模型的差别
|
||||
|
||||
1. CS：DoOR 静态归一化响应映射为 PN Bernoulli 放电请求，整个 120 ms 试次保持同一 rate，没有气味输送/受体时序模型。
|
||||
2. 初态：每试次 V=rest，g/ref/delay queue=0，APL state=0；没有经过真实试次间隔的连续状态。
|
||||
3. US：指定 DAN 一步放电请求。refractory 或当步已有 spike 时请求不会额外产生 spike。
|
||||
4. 教学 gate：由外部计划时刻取 KC trace；未以实际 DAN 接受数、dopamine release、receptor kinetics 或 second messenger 定义。
|
||||
5. Gate 和神经传输区分：KC/DAN spikes 的网络效应保留 18 步延迟，外部教学 gate 使用 emission 时刻。两者共同平移不改变相对间隔，但并不构成生化模型。
|
||||
|
||||
## 本轮检验
|
||||
|
||||
- 先复现短试次，确认仅添加观测没有改变动力学。
|
||||
- 观察 1 s 持续同率输入，以及 120 ms 输入后关闭的对照。不把增加持续时间当作改善 learning 的调参。
|
||||
- 使用完全相同前 120 ms 输入，防止不同随机矩阵形状造成前缀改变。
|
||||
- 100 ms 无输入预热是否仅平移响应？若初态是吸收静息态，空等不等于生理稳态预热。
|
||||
- 分开记录 scheduled US、实际新增 DAN spikes、已有自然 spikes、refractory 拒绝、外部 teaching gate。
|
||||
- 无权重更新、无 US 扫描、无 APL ablation、无新学习候选。失败边界只能约束当前模型，不能替真实果蝇作不可能性结论。
|
||||
@@ -0,0 +1,227 @@
|
||||
"""Read-only CS/US semantics diagnostics; no learning or parameter selection."""
|
||||
from pathlib import Path
|
||||
import argparse
|
||||
import json
|
||||
import math
|
||||
import time
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import torch
|
||||
import yaml
|
||||
import phase4p
|
||||
import phase4p_finalize
|
||||
import phase4s
|
||||
import phase4u
|
||||
from phase4p_revalidate import DeterministicAPLSim, door_inputs, make_drive
|
||||
from phase4r import save, digest, arrhash
|
||||
from phase4h_audit import verify_checksum
|
||||
|
||||
|
||||
def checks(cfg):
|
||||
old = yaml.safe_load((Path(cfg['prior_model'])/'phase4u.yaml').read_text())
|
||||
out = phase4u.checks(old)
|
||||
for key in ('prior_root', 'prior_model'):
|
||||
root = Path(cfg[key]); out['U_'+key] = verify_checksum(root, root/'checksums.sha256')
|
||||
assert all(x['passed'] for x in out.values())
|
||||
return out
|
||||
|
||||
|
||||
def forcing(sim, targets, rates, seed, cfg, mode):
|
||||
block = round(cfg['forcing_block_ms']/sim.dt)
|
||||
steps = round((cfg['observation_duration_ms'] if mode in ('sustained','offset') else cfg['short_duration_ms'])/sim.dt)
|
||||
probs = torch.as_tensor(np.asarray(rates, np.float32)*sim.dt/1000., device=sim.device)
|
||||
parts = []
|
||||
for b in range(math.ceil(steps/block)):
|
||||
gen = torch.Generator(device=sim.device); gen.manual_seed(int(seed+b*cfg['forcing_block_seed_stride']))
|
||||
parts.append((torch.rand((block,len(targets)),generator=gen,device=sim.device)<probs[None,:]).to(torch.uint8))
|
||||
out = torch.cat(parts)[:steps]
|
||||
if mode == 'offset': out[block:] = 0
|
||||
if mode == 'warmup':
|
||||
out = torch.cat((torch.zeros((round(cfg['warmup_ms']/sim.dt),len(targets)),dtype=torch.uint8,device=sim.device),out))
|
||||
return out
|
||||
|
||||
|
||||
def observed(sim, targets, forced, target_kc, cfg, dan=None, pulse=None):
|
||||
"""Statement ordering and all state mutations mirror frozen DeterministicAPLSim.run."""
|
||||
rt=sim.rt; sim.install(None); rt.reset(); steps=len(forced)
|
||||
tg=torch.as_tensor(targets,dtype=torch.int64,device=sim.device)
|
||||
dan=np.empty(0,np.int32) if dan is None else dan
|
||||
dg=torch.as_tensor(dan,dtype=torch.int64,device=sim.device)
|
||||
df=torch.zeros((steps,len(dan)),dtype=torch.uint8,device=sim.device)
|
||||
ds=set() if pulse is None else {round(pulse/sim.dt)}
|
||||
for s in ds: df[s]=1
|
||||
apl=torch.zeros(len(sim.apl),dtype=torch.float32,device=sim.device)
|
||||
trace=torch.zeros(len(sim.kc),dtype=torch.float32,device=sim.device)
|
||||
first=torch.full((sim.n,),-1,dtype=torch.int32,device=sim.device)
|
||||
ad=np.float32(math.exp(-sim.dt/20.)); td=np.float32(math.exp(-sim.dt/100.))
|
||||
trace_at=[]; evs=[]; evn=[]; samples=[]; us=[]; prefix=None
|
||||
stride=round(cfg['sampling_ms']/sim.dt); short=round(cfg['short_duration_ms']/sim.dt)
|
||||
requests=0; accepted=0; existing=0; blocked=0
|
||||
target_gpu=torch.as_tensor(target_kc,dtype=torch.int64,device=sim.device)
|
||||
torch.cuda.synchronize(); start=time.perf_counter()
|
||||
for step in range(steps):
|
||||
rt.ext.state_step(rt.v,rt.g,rt.ref,rt.queue[step%rt.qlen],rt.spike_u8,rt.counts,
|
||||
float(rt.gd),float(rt.alpha),float(rt.rest),float(rt.resetv),float(rt.threshold),rt.ref_steps)
|
||||
pre=rt.spike_u8[tg].clone(); pre_ref=rt.ref[tg].clone()
|
||||
rt.ext.force_step(rt.v,rt.ref,rt.spike_u8,rt.counts,tg,forced[step],float(rt.resetv),rt.ref_steps)
|
||||
req=forced[step].bool(); requests+=int(req.sum())
|
||||
accepted+=int((req & ~pre.bool() & (pre_ref==0)).sum())
|
||||
existing+=int((req & pre.bool()).sum()); blocked+=int((req & ~pre.bool() & (pre_ref!=0)).sum())
|
||||
if step in ds: before=rt.spike_u8[dg].clone(); refs=rt.ref[dg].clone()
|
||||
if len(dan): rt.ext.force_step(rt.v,rt.ref,rt.spike_u8,rt.counts,dg,df[step],float(rt.resetv),rt.ref_steps)
|
||||
if step in ds:
|
||||
after=rt.spike_u8[dg]
|
||||
for j,node in enumerate(dan):
|
||||
us.append({'step':step,'time_ms':step*sim.dt,'neuron_idx':int(node),'preexisting_spike':bool(before[j]),
|
||||
'pre_force_refractory':int(refs[j]),'accepted_new_spike':bool(after[j] and not before[j]),
|
||||
'actual_output_spike':bool(after[j]),'teaching_gate_sampled':True,'first_edge_arrival_ms':(step+rt.delay_steps)*sim.dt})
|
||||
sources=torch.nonzero(rt.spike_u8,as_tuple=False).flatten()
|
||||
if sources.numel():
|
||||
missing=first[sources]<0; first[sources[missing]]=step
|
||||
rt.canonical_fanout(sources,rt.queue[(step+rt.delay_steps)%rt.qlen])
|
||||
kc_spike=rt.spike_u8[sim.kc_gpu]
|
||||
trace.mul_(float(td)).add_(kc_spike.to(torch.float32)).clamp_max_(1.)
|
||||
if step in ds: trace_at.append(trace.clone())
|
||||
active=torch.nonzero(kc_spike,as_tuple=False).flatten()
|
||||
if active.numel():
|
||||
node=sim.kc_gpu[active].cpu().numpy().astype(np.int32)
|
||||
evn.append(node); evs.append(np.full(len(node),step,np.int32))
|
||||
apl.mul_(float(ad)).add_(torch.mv(sim.input_gpu,rt.spike_u8.to(torch.float32)))
|
||||
inhibition=torch.matmul(apl,sim.output_gpu)*np.float32(.01)
|
||||
rt.g[sim.kc_gpu]=rt.g[sim.kc_gpu]-inhibition
|
||||
rt.step_index+=1
|
||||
if step==short-1:
|
||||
prefix={k:x.cpu().numpy().copy() for k,x in [('counts',rt.counts),('final_v',rt.v),('final_g',rt.g)]}
|
||||
if (step+1)%stride==0:
|
||||
samples.append({'step':step,'time_ms':(step+1)*sim.dt,'PN_requested_cumulative':requests,
|
||||
'PN_accepted_cumulative':accepted,'PN_preexisting_cumulative':existing,'PN_blocked_cumulative':blocked,
|
||||
'KC_spikes_cumulative':int(rt.counts[sim.kc_gpu].sum()),'target_KC_spikes_cumulative':int(rt.counts[target_gpu].sum()),
|
||||
'KC_active_cumulative':int((rt.counts[sim.kc_gpu]>0).sum()),'KC_g_mean':float(rt.g[sim.kc_gpu].mean()),
|
||||
'KC_V_mean':float(rt.v[sim.kc_gpu].mean()),'KC_threshold_margin_min':float((rt.threshold-rt.v[sim.kc_gpu]).min()),
|
||||
'APL_state_0':float(apl[0]),'APL_state_1':float(apl[1]),'APL_inhibition_mean':float(inhibition.mean()),
|
||||
'finite':bool(torch.isfinite(rt.v).all() and torch.isfinite(rt.g).all())})
|
||||
torch.cuda.synchronize()
|
||||
counts=rt.counts.cpu().numpy().copy()
|
||||
return {'counts':counts,'kc_counts':counts[sim.kc], 'first':first.cpu().numpy().copy(),
|
||||
'final_v':rt.v.cpu().numpy().copy(),'final_g':rt.g.cpu().numpy().copy(),
|
||||
'trace':np.zeros(len(sim.kc),np.float32) if not trace_at else trace_at[-1].cpu().numpy().copy(),
|
||||
'kc_event_step':np.concatenate(evs) if evs else np.empty(0,np.int32),
|
||||
'kc_event_node':np.concatenate(evn) if evn else np.empty(0,np.int32),
|
||||
'prefix':prefix,'samples':samples,'US':us,'wall_seconds':time.perf_counter()-start,
|
||||
'finite':all(x['finite'] for x in samples)}
|
||||
|
||||
|
||||
def exact(a,b,keys):
|
||||
return {k:bool(np.array_equal(a[k],b[k])) for k in keys}
|
||||
|
||||
|
||||
def run(cfg,exp,model):
|
||||
for name in ('protocol','audit','events','states','observations','regression','performance'): (exp/name).mkdir(parents=True,exist_ok=True)
|
||||
assert not (exp/'protocol/preregistered.json').exists(),'Refusing overwrite'
|
||||
save(exp/'audit/preflight.json',checks(cfg))
|
||||
oc,panel,vectors,selection,mapping=door_inputs(); panel=panel[panel.split=='calibration'].reset_index(drop=True)
|
||||
assert len(panel)==6
|
||||
save(exp/'protocol/preregistered.json',{'config':cfg,'panel':panel.to_dict('records'),
|
||||
'sources':{p.name:digest(p) for p in model.iterdir() if p.is_file()},'before_simulation':True,'no_parameter_selection':True})
|
||||
pc=yaml.safe_load((Path(cfg['runtime_root'])/'phase4p.yaml').read_text())
|
||||
rt=phase4p.CanonicalRuntime(pc,Path(cfg['runtime_root']),exp); sim=DeterministicAPLSim(rt)
|
||||
assert len(sim.identity)==14232
|
||||
target=np.unique(sim.identity.loc[sim.identity.compartment.eq(cfg['target_compartment']),'source_neuron_idx'].to_numpy(np.int32))
|
||||
dan=sim.dan_targets(cfg['matched_DAN'],cfg['target_compartment'])
|
||||
save(exp/'audit/semantics.json',{'CS':'frozen physiological DoOR-to-PN rates; Bernoulli force request per 0.1ms step; refractory may refuse',
|
||||
'US':'one scheduled force request per confirmed PAM15 neuron; engineering teaching trace sampled at scheduled emission step regardless acceptance',
|
||||
'dopamine_concentration_measured':False,'initial_state':'all V at rest; g/ref/queue/APL/trace zero',
|
||||
'delay_steps':rt.delay_steps,'delay_ms':rt.delay_steps*sim.dt,'DAN_ids':dan.tolist(),'target_KC_ids':target.tolist(),
|
||||
'long_input':'120ms RNG blocks with explicit seed stride; first block identical, later sequence not claimed to equal legacy long run',
|
||||
'time_scaling_to_biological_seconds':'unknown; 1000ms is observation, not conditioning replication'})
|
||||
keys=['counts','first','final_v','final_g','trace','kc_event_step','kc_event_node']; regress=[]; summaries=[]; usrows=[]
|
||||
start=time.perf_counter(); torch.cuda.reset_peak_memory_stats()
|
||||
def persist(name,r,oi,seed,mode):
|
||||
assert r['finite'],'Nonfinite state'
|
||||
phase4s.persist_run(exp,name,r)
|
||||
np.savez_compressed(exp/'states'/f'{name}.npz',**{k:r[k] for k in ('counts','final_v','final_g','trace')})
|
||||
pd.DataFrame(r['samples']).to_parquet(exp/'observations'/f'{name}.parquet',index=False)
|
||||
steps=r['kc_event_step']; nodes=r['kc_event_node']; t=steps[np.isin(nodes,target)]*sim.dt
|
||||
row={'odor_index':oi,'odor_name':panel.iloc[oi].odor_name,'seed':seed,'condition':mode,
|
||||
'KC_active':int(np.count_nonzero(r['kc_counts'])),'KC_events':len(steps),'target_events':len(t),
|
||||
'first_target_ms':float(t.min()) if len(t) else None,'median_target_ms':float(np.median(t)) if len(t) else None,
|
||||
'target_events_after120ms':int(np.count_nonzero(t>=120)), 'KC_events_after120ms':int(np.count_nonzero(steps*sim.dt>=120)),
|
||||
'last_KC_event_ms':float(steps.max()*sim.dt) if len(steps) else None,'wall_seconds':r['wall_seconds'],'finite':r['finite']}
|
||||
summaries.append(row)
|
||||
for oi,odor in panel.iterrows():
|
||||
targets,rates=make_drive(oc,panel,vectors,selection,mapping,odor.odor_id,'physiological')
|
||||
for si,seed in enumerate(cfg['seeds']):
|
||||
actual=cfg['seed_namespace']+oi*cfg['odor_seed_offset']+seed
|
||||
legacy=sim.run(targets,rates,actual,cfg['short_duration_ms'],collect_events=True)
|
||||
f=forcing(sim,targets,rates,actual,cfg,'short')
|
||||
short=observed(sim,targets,f,target,cfg)
|
||||
eq=exact(legacy,short,keys); regress.append({'odor':oi,'seed':seed,'test':'short-exact','fields':eq})
|
||||
save(exp/'regression/instrumentation.json',regress); assert all(eq.values()),'Instrumentation diverges'
|
||||
persist(f'{oi}-{seed}-short',short,oi,seed,'short')
|
||||
for mode in ('sustained','offset'):
|
||||
ff=forcing(sim,targets,rates,actual,cfg,mode); assert torch.equal(ff[:len(f)],f)
|
||||
r=observed(sim,targets,ff,target,cfg)
|
||||
eq=exact(legacy,r['prefix'],['counts','final_v','final_g'])
|
||||
mask=r['kc_event_step']<len(f)
|
||||
eq['events']=arrhash(r['kc_event_step'][mask],r['kc_event_node'][mask])==arrhash(legacy['kc_event_step'],legacy['kc_event_node'])
|
||||
regress.append({'odor':oi,'seed':seed,'test':mode+'-prefix','fields':eq}); assert all(eq.values())
|
||||
persist(f'{oi}-{seed}-{mode}',r,oi,seed,mode)
|
||||
if si==0:
|
||||
warm=observed(sim,targets,forcing(sim,targets,rates,actual,cfg,'warmup'),target,cfg)
|
||||
eq=exact(legacy,warm,['counts','final_v','final_g','trace','kc_event_node'])
|
||||
eq['shifted_events']=np.array_equal(warm['kc_event_step']-round(cfg['warmup_ms']/sim.dt),legacy['kc_event_step'])
|
||||
regress.append({'odor':oi,'seed':seed,'test':'quiet-warmup','fields':eq})
|
||||
persist(f'{oi}-{seed}-warmup',warm,oi,seed,'warmup')
|
||||
for pulse in cfg['US_audit_ms']:
|
||||
lr=sim.run(targets,rates,actual,cfg['short_duration_ms'],dan_targets=dan,dan_pulses=[pulse],collect_events=True)
|
||||
r=observed(sim,targets,f,target,cfg,dan,pulse); eq=exact(lr,r,keys)
|
||||
regress.append({'odor':oi,'seed':seed,'test':f'US{pulse}-exact','fields':eq}); assert all(eq.values())
|
||||
usrows.extend([dict(x,odor_index=oi,seed=seed,US_ms=pulse) for x in r['US']])
|
||||
persist(f'{oi}-{seed}-US{pulse}',r,oi,seed,f'US{pulse}')
|
||||
save(exp/'regression/instrumentation.json',regress)
|
||||
print(json.dumps({'completed_odor':oi,'seed':seed,'trials':len(summaries)}),flush=True)
|
||||
pd.DataFrame(summaries).to_parquet(exp/'audit/trials.parquet',index=False)
|
||||
pd.DataFrame(usrows).to_parquet(exp/'audit/US-acceptance.parquet',index=False)
|
||||
reference=phase4s.evidence_regression(cfg,exp,rt,sim)
|
||||
result={'phase':'4V','status':'COMPLETE-DIAGNOSTIC-ONLY','reference_pass':reference,
|
||||
'exact_instrumentation_pass':all(all(x['fields'].values()) for x in regress),'diagnostic_trials':len(summaries),
|
||||
'weight_updates':0,'candidate_created':False,'reserved_test_odors_consumed':0,
|
||||
'DAN_requests':len(usrows),'DAN_accepted_new':sum(x['accepted_new_spike'] for x in usrows),
|
||||
'DAN_preexisting':sum(x['preexisting_spike'] for x in usrows),
|
||||
'DAN_refused':sum(not x['actual_output_spike'] for x in usrows)}
|
||||
assert result['exact_instrumentation_pass'] and reference
|
||||
save(exp/'audit/results.json',result)
|
||||
save(exp/'performance/summary.json',{'wall_seconds_excluding_build':time.perf_counter()-start,
|
||||
'peak_VRAM_bytes':torch.cuda.max_memory_allocated(),'GPU':torch.cuda.get_device_name(0),
|
||||
'measurement':'instrumented diagnostics include synchronous observation overhead; not runtime speed benchmark'})
|
||||
print(json.dumps(result),flush=True)
|
||||
|
||||
|
||||
def seal(cfg,exp,model):
|
||||
now=checks(cfg); history=phase4p_finalize.integrity(); pre=json.loads((exp/'protocol/preregistered.json').read_text())
|
||||
passed=history['all_static_checks_passed'] and now==json.loads((exp/'audit/preflight.json').read_text()) and all(digest(model/n)==h for n,h in pre['sources'].items())
|
||||
save(exp/'audit/frozen-integrity.json',{'pass':passed,'history':history,'current':now}); assert passed
|
||||
r=json.loads((exp/'audit/results.json').read_text()); r['integrity_pass']=True; save(exp/'audit/results.json',r)
|
||||
df=pd.read_parquet(exp/'audit/trials.parquet'); lines=['# Phase 4V — CS/US semantics and timescale audit','',
|
||||
'Status: COMPLETE-DIAGNOSTIC-ONLY. No training, selection, new timing policy or learned candidate.',
|
||||
'Frozen instrumentation and five reference trials: exact PASS. Frozen checksums: PASS.',
|
||||
'Only six old calibration odors were used; six reserved test odors remain unconsumed.','',
|
||||
'| Odor | Condition | mean KC events | mean target events | mean target events >=120ms |',
|
||||
'|---|---|---:|---:|---:|']
|
||||
for (odor,mode),g in df.groupby(['odor_name','condition'],sort=False):
|
||||
lines.append(f"| {odor} | {mode} | {g.KC_events.mean():.2f} | {g.target_events.mean():.2f} | {g.target_events_after120ms.mean():.2f} |")
|
||||
lines += ['',f"DAN requests: {r['DAN_requests']}; accepted new spikes: {r['DAN_accepted_new']}; preexisting: {r['DAN_preexisting']}; refused: {r['DAN_refused']}.",
|
||||
'US is a force request, not measured dopamine. Teaching trace is sampled at the scheduled emission step; synaptic arrivals are 1.8ms later.',
|
||||
'Observation does not establish biological time scaling or associative memory. Literature boundaries are in phase4v-literature.md.',
|
||||
'No game, decoder, RL, visual input, gain/threshold changes, eligibility changes, or plastic-mask changes.']
|
||||
(exp/'REPORT.md').write_text('\n'.join(lines)+'\n',encoding='utf-8')
|
||||
for root in (model,exp):
|
||||
rows=[{'path':p.relative_to(root).as_posix(),'sha256':digest(p),'bytes':p.stat().st_size} for p in sorted(root.rglob('*')) if p.is_file() and p.name not in ('manifest.json','checksums.sha256')]
|
||||
save(root/'manifest.json',{'files':rows}); rows.append({'path':'manifest.json','sha256':digest(root/'manifest.json')})
|
||||
(root/'checksums.sha256').write_text(''.join(x['sha256']+' '+x['path']+'\n' for x in rows))
|
||||
check=verify_checksum(root,root/'checksums.sha256'); assert check['passed']; print(json.dumps(check),flush=True)
|
||||
|
||||
|
||||
if __name__=='__main__':
|
||||
p=argparse.ArgumentParser(); p.add_argument('task',choices=['run','seal']); p.add_argument('--config',required=True); a=p.parse_args()
|
||||
cfg=yaml.safe_load(Path(a.config).read_text()); (run if a.task=='run' else seal)(cfg,Path(cfg['experiment_root']),Path(cfg['model_root']))
|
||||
@@ -0,0 +1,40 @@
|
||||
phase: 4V
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4v-csus-semantics
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/csus-audit-v1
|
||||
runtime_root: /data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2
|
||||
phase4p_root: /data/digital-drosophila/experiments/phase4p-deterministic-runtime
|
||||
prior_root: /data/digital-drosophila/experiments/phase4u-odor-generalization
|
||||
prior_model: /data/digital-drosophila/models/male-cns-lif/odor-timing-generalization-v1
|
||||
target_compartment: "y5B'2a"
|
||||
matched_DAN: PAM15
|
||||
odors: all six original calibration odors only; do not consume six reserved Phase4U test odors
|
||||
seeds: [131101,131102,131103]
|
||||
seed_namespace: 80000000
|
||||
odor_seed_offset: 1000000
|
||||
short_duration_ms: 120
|
||||
observation_duration_ms: 1000
|
||||
warmup_ms: 100
|
||||
US_audit_ms: [25,90]
|
||||
forcing_block_ms: 120
|
||||
forcing_block_seed_stride: 1000003
|
||||
sampling_ms: 1
|
||||
activity_bin_ms: 10
|
||||
conditions:
|
||||
- short frozen replay for every seed
|
||||
- sustained same-rate PN input to 1000ms
|
||||
- same first120ms then PN input off until1000ms
|
||||
- silent100ms warmup followed by same120ms input, first seed each odor
|
||||
- scheduled US25 or90ms force-acceptance audit, first seed each odor
|
||||
checks:
|
||||
mandatory_before_long_trials: short instrumented run vs frozen original counts/events/finalV/finalG/trace exact
|
||||
long_trial_prefix: first120ms inputs and complete neuron counts/V/g match original
|
||||
no_runtime_source_modification: true
|
||||
frozen_reference: five original deterministic trials
|
||||
scope:
|
||||
diagnostic_only: true
|
||||
learning: false
|
||||
parameter_selection: false
|
||||
new_timing_candidate: false
|
||||
new_US_policy: false
|
||||
unknown_biological_time_scaling: true
|
||||
long_input_is_observation_not_biological_conditioning_replication: true
|
||||
@@ -0,0 +1,93 @@
|
||||
"""Descriptive audit only; no parameter ranking, fitting or candidate selection."""
|
||||
from pathlib import Path
|
||||
import json
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
from phase4r import save
|
||||
|
||||
cfg=yaml.safe_load(Path(__file__).with_name('phase4v.yaml').read_text())
|
||||
exp=Path(cfg['experiment_root']); prior=Path(cfg['prior_root'])
|
||||
assert not (exp/'checksums.sha256').exists(),'Already sealed'
|
||||
target=set(json.loads((exp/'audit/semantics.json').read_text())['target_KC_ids'])
|
||||
old=[]
|
||||
for p in sorted((prior/'events').glob('calibration-*.parquet')):
|
||||
x=pd.read_parquet(p); t=x.loc[x.neuron_idx.isin(target),'step'].to_numpy()*.1
|
||||
old.append({'trial':p.stem,'events':len(t),'first_ms':float(t.min()) if len(t) else None,
|
||||
'fraction_after25ms':float((t>25).mean()) if len(t) else None,
|
||||
'fraction_after60ms':float((t>60).mean()) if len(t) else None,
|
||||
'fraction_after90ms':float((t>90).mean()) if len(t) else None})
|
||||
assert len(old)==120
|
||||
pd.DataFrame(old).to_parquet(exp/'audit/prior120-event-distribution.parquet',index=False)
|
||||
trials=pd.read_parquet(exp/'audit/trials.parquet'); details=[]; bins=[]
|
||||
for _,r in trials.iterrows():
|
||||
name=f'{r.odor_index}-{r.seed}-{r.condition}'
|
||||
x=pd.read_parquet(exp/'events'/f'{name}.parquet')
|
||||
obs=pd.read_parquet(exp/'observations'/f'{name}.parquet')
|
||||
assert (obs.PN_requested_cumulative == obs.PN_accepted_cumulative+obs.PN_preexisting_cumulative+obs.PN_blocked_cumulative).all()
|
||||
if r.condition=='offset':
|
||||
assert obs.loc[obs.time_ms>=cfg['short_duration_ms'],'PN_requested_cumulative'].nunique()==1
|
||||
t=x.step.to_numpy()*.1; tt=x.loc[x.neuron_idx.isin(target),'step'].to_numpy()*.1
|
||||
for start in np.arange(0,float(obs.time_ms.max()),cfg['activity_bin_ms']):
|
||||
end=start+cfg['activity_bin_ms']; mask=(t>=start)&(t<end)
|
||||
bins.append({'odor_index':int(r.odor_index),'seed':int(r.seed),'condition':r.condition,'start_ms':float(start),
|
||||
'KC_events':int(mask.sum()),'KC_active':int(x.loc[mask,'neuron_idx'].nunique()),
|
||||
'target_events':int(((tt>=start)&(tt<end)).sum())})
|
||||
with np.load(exp/'states'/f'{name}.npz') as state:
|
||||
count=state['counts']; total=int(count.astype(np.int64).sum()); active=int(np.count_nonzero(count))
|
||||
row=r.to_dict(); row.update({'whole_brain_spikes':total,'whole_brain_active':active,
|
||||
'KC_events_120_200ms':int(((t>=120)&(t<200)).sum()),
|
||||
'KC_events_after200ms':int((t>=200).sum()),'KC_events_after500ms':int((t>=500).sum()),
|
||||
'PN_requests':int(obs.iloc[-1].PN_requested_cumulative),'PN_new_spikes':int(obs.iloc[-1].PN_accepted_cumulative),
|
||||
'PN_refractory_blocked':int(obs.iloc[-1].PN_blocked_cumulative),
|
||||
'late_KC_g_mean':float(obs[obs.time_ms>=max(obs.time_ms)-100].KC_g_mean.mean()),
|
||||
'late_APL_inhibition_mean':float(obs[obs.time_ms>=max(obs.time_ms)-100].APL_inhibition_mean.mean())})
|
||||
details.append(row)
|
||||
pd.DataFrame(bins).to_parquet(exp/'audit/activity-10ms-bins.parquet',index=False)
|
||||
df=pd.DataFrame(details); df.to_parquet(exp/'audit/detailed-trials.parquet',index=False)
|
||||
rows=[]
|
||||
for (odor,condition),g in df.groupby(['odor_name','condition'],sort=False):
|
||||
row={'odor':odor,'condition':condition,'n':len(g)}
|
||||
for k in ('KC_events','target_events','KC_events_after120ms','KC_events_after200ms','KC_events_after500ms','whole_brain_spikes','whole_brain_active','PN_requests','PN_new_spikes','PN_refractory_blocked','late_KC_g_mean','late_APL_inhibition_mean'):
|
||||
row[k+'_mean']=float(g[k].mean())
|
||||
rows.append(row)
|
||||
save(exp/'audit/descriptive-summary.json',{'groups':rows,'analysis':'post-run descriptive summaries, not hypothesis selection or tuning'})
|
||||
us=pd.read_parquet(exp/'audit/US-acceptance.parquet')
|
||||
us['scheduled_arrival_slot_ms']=us.first_edge_arrival_ms
|
||||
us['output_spike_arrival_ms']=us.first_edge_arrival_ms.where(us.actual_output_spike)
|
||||
us.to_parquet(exp/'audit/US-arrival-semantics.parquet',index=False)
|
||||
save(exp/'audit/field-semantics.json',{
|
||||
'PN_request_accounting_partition_exact':True,
|
||||
'offset_no_external_requests_after120ms':True,
|
||||
'US_original_first_edge_arrival_ms':'scheduled emission step + frozen delay; not evidence of arrival if force rejected and no actual spike',
|
||||
'US_output_spike_arrival_ms':'nominal edge delivery time only for actual output spikes; not dopamine release/concentration',
|
||||
'samples_time_ms':'state after step; (step+1)*dt',
|
||||
'events_time_ms':'runtime spike convention; step*dt',
|
||||
'analysis_source_role':'descriptive audit added after simulation preregistration; cannot select parameters or candidate'})
|
||||
late=[]
|
||||
for _,r in trials[trials.condition=='offset'].iterrows():
|
||||
e=pd.read_parquet(exp/'events'/f'{r.odor_index}-{r.seed}-offset.parquet')
|
||||
late.append((int(r.odor_index),set(map(int,e.loc[e.step>=5000,'neuron_idx']))))
|
||||
pairs={'within_odor':[],'between_odor':[]}
|
||||
for i,(oi,a) in enumerate(late):
|
||||
for oj,b in late[i+1:]:
|
||||
pairs['within_odor' if oi==oj else 'between_odor'].append(len(a&b)/len(a|b) if a|b else 1.)
|
||||
save(exp/'audit/offset-late-KC-overlap.json',{
|
||||
'scope':'posthoc descriptive >=500ms KC activity; no memory/learning gate',
|
||||
'trials':len(late),'late_active_KC_counts':[len(x[1]) for x in late],
|
||||
'common_KC_ids':sorted(set.intersection(*(x[1] for x in late))),
|
||||
'pairwise_Jaccard':{k:{'pairs':len(v),'mean':float(np.mean(v)),'min':float(min(v)),'max':float(max(v))} for k,v in pairs.items()}})
|
||||
lines=['# Mechanism observations (descriptive, no training)','',
|
||||
'| Odor | Condition | KC events | target events | KC events >=200ms | late mean KC g |',
|
||||
'|---|---|---:|---:|---:|---:|']
|
||||
for r in rows:
|
||||
if r['condition'] in ('short','sustained','offset'):
|
||||
lines.append(f"| {r['odor']} | {r['condition']} | {r['KC_events_mean']:.2f} | {r['target_events_mean']:.2f} | {r['KC_events_after200ms_mean']:.2f} | {r['late_KC_g_mean_mean']:.6f} |")
|
||||
lines += ['', 'The longer trials are diagnostics of the unchanged brain at unchanged input rates. Differences are not evidence of learning.',
|
||||
'Counter observations add synchronization overhead and cannot be used as runtime speed benchmarks.',
|
||||
'No baseline-activity input is added. A silent warmup is not a physiological ongoing-state initialization.',
|
||||
'Turning external PN forcing off does not silence PN neurons or recurrent network activity. Persistent activity is not itself retained associative memory.',
|
||||
'A scheduled teaching gate is an engineering signal, not measured dopamine release or receptor kinetics.',
|
||||
'No new US policy or seconds-to-milliseconds conversion is selected from these observations.']
|
||||
(exp/'audit/MECHANISM.md').write_text('\n'.join(lines)+'\n',encoding='utf-8')
|
||||
print(json.dumps({'old_trials':len(old),'diagnostic_trials':len(df),'groups':rows},indent=2))
|
||||
@@ -0,0 +1,55 @@
|
||||
{
|
||||
"files": [
|
||||
{
|
||||
"path": "MaleCNS-NumericalContract-v1.md",
|
||||
"sha256": "9505cce8e2f7e03338f7425d698b94e7798cf43e924e6f41c1155719b63204cd",
|
||||
"size": 1958
|
||||
},
|
||||
{
|
||||
"path": "VERSION.json",
|
||||
"sha256": "7f95c99c377944bd05987a3e0953f743ee278807f3c57145aa2b62e3627bc3aa",
|
||||
"size": 281
|
||||
},
|
||||
{
|
||||
"path": "phase4p.py",
|
||||
"sha256": "c8a13dbdd58dd222f73c1224de448e54bdf90c3f533163427a2bb25446a03999",
|
||||
"size": 10909
|
||||
},
|
||||
{
|
||||
"path": "phase4p.yaml",
|
||||
"sha256": "6d0189f4ec570cb3352d15ed8d195d1ccb32ee1d29c33dc1e4f00d377b74787e",
|
||||
"size": 2525
|
||||
},
|
||||
{
|
||||
"path": "phase4p_benchmark.py",
|
||||
"sha256": "8d8bad3db949da876c5a7fe9f20845a10c4128523da69bcb4b5c3fab69c15e7a",
|
||||
"size": 4284
|
||||
},
|
||||
{
|
||||
"path": "phase4p_ext.cpp",
|
||||
"sha256": "9d204d77d70b3723567671a61ce8e2e6162e261f48825e86fb34565e3faa622d",
|
||||
"size": 1435
|
||||
},
|
||||
{
|
||||
"path": "phase4p_ext.cu",
|
||||
"sha256": "c24985f4e366f90790d25ab5eceb9d6aff0f9064dd61c7413c13d2a2a4fa5fcd",
|
||||
"size": 5064
|
||||
},
|
||||
{
|
||||
"path": "phase4p_finalize.py",
|
||||
"sha256": "b627fb0934bca8f04ec7c595a7b5e6690a2af62a44bc00ff2972f9f764671cd4",
|
||||
"size": 6047
|
||||
},
|
||||
{
|
||||
"path": "phase4p_revalidate.py",
|
||||
"sha256": "c7e693a2da70acdce4f53af5c2853e52e8333acc93f88a46a5e8083eac639346",
|
||||
"size": 31558
|
||||
},
|
||||
{
|
||||
"path": "phase4p_validate.py",
|
||||
"sha256": "ce06aa5cb6439450440a91603af1c0d187b20dc11abda6528ea0dfc6b71569b5",
|
||||
"size": 30605
|
||||
}
|
||||
],
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2/candidate"
|
||||
}
|
||||
+26
@@ -0,0 +1,26 @@
|
||||
# MaleCNS-NumericalContract-v1
|
||||
|
||||
Status: immutable candidate contract for `MaleCNS-LIF-runtime-v2-deterministic-candidate`.
|
||||
|
||||
## Frozen tensors and arithmetic
|
||||
|
||||
- State (`V`, `g`, ring-buffer arrivals, effective edge weights): IEEE-754 binary32.
|
||||
- CSR `indptr` and edge ids: int64; target indices: int32; refractory/counts: int32.
|
||||
- Effective edge weights are the frozen Phase 4C signed float32 weights; raw synapse counts remain unchanged.
|
||||
- Canonical contribution key: `['target_idx', 'source_idx', 'edge_id']`. Since global CSR edge ids are source-row ordered, stable `(target_idx, edge_id)` sorting is equivalent.
|
||||
- Each target is owned by one thread and accumulated sequentially with round-to-nearest-even `__fadd_rn`. Scientific fan-out contains no atomic add.
|
||||
|
||||
## Per-timestep total order
|
||||
|
||||
1. Read and clear the current exact ring-buffer slot.
|
||||
2. `g = float32(g * g_decay)`, then `g = float32(g + recurrent_arrival)`.
|
||||
3. Decrement refractory by one; while refractory remains nonzero, hold `V = V_reset`.
|
||||
4. For available neurons, forward Euler: `V = float32(V + alpha * (g - (V - V_rest)))`, with every elementary operation rounded to float32.
|
||||
5. Detect spikes using the frozen comparison `V >= V_threshold`; reset and set exact refractory steps.
|
||||
6. Apply same-step external forced-spike events in ascending target index; a neuron already spiking or refractory is not double-counted.
|
||||
7. Enumerate spiking source indices in ascending neuron index. Expand each CSR row in edge-id order.
|
||||
8. Stable-sort contributions by target only, preserving source/edge order, then perform fixed sequential segmented sums.
|
||||
9. Add each segment once to the ring-buffer slot exactly 18 steps ahead.
|
||||
10. Emit events in `(simulation_step, neuron_idx)` order. Simultaneous events never receive a hidden priority beyond neuron index.
|
||||
|
||||
No threshold epsilon, voltage rounding/snap, float64 model substitution, approximate delay, topology change, or parameter change is permitted.
|
||||
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"biological_parameters_changed": false,
|
||||
"generated_utc": "2026-09-22T18:40:39Z",
|
||||
"historical_references_overwritten": false,
|
||||
"name": "MaleCNS-LIF-runtime-v2-deterministic-candidate",
|
||||
"numerical_contract": "MaleCNS-NumericalContract-v1",
|
||||
"revalidated_through": "R6"
|
||||
}
|
||||
@@ -0,0 +1,10 @@
|
||||
9505cce8e2f7e03338f7425d698b94e7798cf43e924e6f41c1155719b63204cd MaleCNS-NumericalContract-v1.md
|
||||
7f95c99c377944bd05987a3e0953f743ee278807f3c57145aa2b62e3627bc3aa VERSION.json
|
||||
c8a13dbdd58dd222f73c1224de448e54bdf90c3f533163427a2bb25446a03999 phase4p.py
|
||||
6d0189f4ec570cb3352d15ed8d195d1ccb32ee1d29c33dc1e4f00d377b74787e phase4p.yaml
|
||||
8d8bad3db949da876c5a7fe9f20845a10c4128523da69bcb4b5c3fab69c15e7a phase4p_benchmark.py
|
||||
9d204d77d70b3723567671a61ce8e2e6162e261f48825e86fb34565e3faa622d phase4p_ext.cpp
|
||||
c24985f4e366f90790d25ab5eceb9d6aff0f9064dd61c7413c13d2a2a4fa5fcd phase4p_ext.cu
|
||||
b627fb0934bca8f04ec7c595a7b5e6690a2af62a44bc00ff2972f9f764671cd4 phase4p_finalize.py
|
||||
c7e693a2da70acdce4f53af5c2853e52e8333acc93f88a46a5e8083eac639346 phase4p_revalidate.py
|
||||
ce06aa5cb6439450440a91603af1c0d187b20dc11abda6528ea0dfc6b71569b5 phase4p_validate.py
|
||||
@@ -0,0 +1,160 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Canonical deterministic MaleCNS LIF reference backend and validation tasks."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pyarrow as pa
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
from torch.utils.cpp_extension import load as load_extension
|
||||
|
||||
import phase4c
|
||||
|
||||
|
||||
def sha_bytes(*arrays: np.ndarray) -> str:
|
||||
h=hashlib.sha256()
|
||||
for x in arrays:h.update(np.ascontiguousarray(x).tobytes())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def save(path: Path,x: object):
|
||||
path.parent.mkdir(parents=True,exist_ok=True)
|
||||
path.write_text(json.dumps(x,indent=2,sort_keys=True,default=str)+'\n')
|
||||
|
||||
|
||||
def build_extension(model:Path,exp:Path):
|
||||
build=exp/'gpu-reference/extension-build';build.mkdir(parents=True,exist_ok=True)
|
||||
start=time.perf_counter()
|
||||
ext=load_extension(name='male_cns_lif_deterministic_v2_ext',
|
||||
sources=[str(model/'phase4p_ext.cpp'),str(model/'phase4p_ext.cu')],
|
||||
build_directory=str(build),extra_cflags=['-O3','-ffp-contract=off'],
|
||||
extra_cuda_cflags=['-O3','--fmad=false','-std=c++20'],verbose=True)
|
||||
report={'status':'PASS','build_seconds':time.perf_counter()-start,'binary':str(Path(ext.__file__)),
|
||||
'binary_sha256':phase4c.sha256(Path(ext.__file__)),'flags':{'cxx':['-O3','-ffp-contract=off'],'cuda':['-O3','--fmad=false','-std=c++20']}}
|
||||
save(exp/'gpu-reference/extension.json',report);return ext
|
||||
|
||||
|
||||
class CanonicalRuntime:
|
||||
def __init__(self,cfg:dict[str,Any],model:Path,exp:Path):
|
||||
self.cfg=cfg;self.exp=exp;self.device=torch.device('cuda:0')
|
||||
if torch.cuda.device_count()!=1 or cfg['device']['required_name_substring'] not in torch.cuda.get_device_name(0):
|
||||
raise RuntimeError('exactly one visible RTX 4090 required')
|
||||
p4cfg=yaml.safe_load((Path(cfg['paths']['historical_runtime_root'])/'phase4c.yaml').read_text())
|
||||
paths=phase4c.exact_paths(p4cfg)
|
||||
self.indptr_np,self.indices_np,self.raw_np=phase4c.p4b.validate_graph(p4cfg,paths)
|
||||
self.table=phase4c.full_metadata(paths['metadata']);self.pops=phase4c.populations(self.table)
|
||||
base=phase4c.Runtime(p4cfg,self.table,self.indptr_np,self.indices_np,self.raw_np,self.device,None)
|
||||
self.indptr,self.indices,self.weights=base.indptr,base.indices,base.weights
|
||||
self.ext=build_extension(model,exp)
|
||||
m=cfg['model'];self.n=int(cfg['expected_graph']['neurons']);self.dt=float(m['dt_ms'])
|
||||
self.delay_steps=int(round(m['synaptic_delay_ms']/self.dt));self.qlen=self.delay_steps+1
|
||||
self.ref_steps=int(round(m['refractory_ms']/self.dt));self.gd=np.float32(1-self.dt/m['synaptic_tau_ms'])
|
||||
self.alpha=np.float32(self.dt/m['membrane_tau_ms']);self.rest=np.float32(m['resting_potential_mv'])
|
||||
self.resetv=np.float32(m['reset_potential_mv']);self.threshold=np.float32(m['threshold_mv'])
|
||||
self.superclass=np.asarray(self.table['annotation_superclass'].to_pylist(),object)
|
||||
|
||||
def reset(self):
|
||||
d=self.device;self.v=torch.full((self.n,),float(self.rest),dtype=torch.float32,device=d)
|
||||
self.g=torch.zeros(self.n,dtype=torch.float32,device=d);self.ref=torch.zeros(self.n,dtype=torch.int32,device=d)
|
||||
self.queue=torch.zeros((self.qlen,self.n),dtype=torch.float32,device=d)
|
||||
self.spike_u8=torch.zeros(self.n,dtype=torch.uint8,device=d);self.counts=torch.zeros(self.n,dtype=torch.int32,device=d)
|
||||
self.step_index=0;self.event_steps=[];self.event_nodes=[]
|
||||
|
||||
def canonical_fanout(self,sources:torch.Tensor,destination:torch.Tensor)->dict[str,float|int]:
|
||||
if not sources.numel():return {'contributions':0,'segments':0,'sort_seconds':0.,'reduce_seconds':0.}
|
||||
starts=self.indptr[sources];degree=self.indptr[sources+1]-starts;cum=torch.cumsum(degree,0);total=int(cum[-1].item())
|
||||
edges=torch.empty(total,dtype=torch.int64,device=self.device)
|
||||
self.ext.fill_edges(self.indptr,sources,cum-degree,edges)
|
||||
t0=time.perf_counter();targets=self.indices[edges]
|
||||
order=torch.argsort(targets,stable=True);sorted_edges=edges[order];sorted_targets=targets[order]
|
||||
boundary=torch.ones(total,dtype=torch.bool,device=self.device)
|
||||
if total>1:boundary[1:]=sorted_targets[1:]!=sorted_targets[:-1]
|
||||
seg_starts=torch.nonzero(boundary,as_tuple=False).flatten().to(torch.int64)
|
||||
unique_targets=sorted_targets[seg_starts].to(torch.int32);sort_sec=time.perf_counter()-t0
|
||||
t1=time.perf_counter();self.ext.segmented_sum(sorted_edges,seg_starts,unique_targets,self.weights,destination)
|
||||
return {'contributions':total,'segments':int(len(unique_targets)),'sort_seconds':sort_sec,'reduce_seconds':time.perf_counter()-t1}
|
||||
|
||||
def run(self,targets_np:np.ndarray,rate:float,seed:int,duration_ms:float,capture=True,forced_override=None):
|
||||
self.reset();steps=int(round(duration_ms/self.dt));targets=torch.as_tensor(targets_np,dtype=torch.int64,device=self.device)
|
||||
forced=forced_override if forced_override is not None else phase4c.generate_forced(steps,len(targets_np),rate,self.dt,seed,self.device)
|
||||
torch.cuda.synchronize();t0=time.perf_counter();sort_t=reduce_t=0.;contrib=0
|
||||
for step in range(steps):
|
||||
slot=step%self.qlen
|
||||
self.ext.state_step(self.v,self.g,self.ref,self.queue[slot],self.spike_u8,self.counts,
|
||||
float(self.gd),float(self.alpha),float(self.rest),float(self.resetv),float(self.threshold),self.ref_steps)
|
||||
self.ext.force_step(self.v,self.ref,self.spike_u8,self.counts,targets,forced[step],float(self.resetv),self.ref_steps)
|
||||
sources=torch.nonzero(self.spike_u8,as_tuple=False).flatten()
|
||||
stats=self.canonical_fanout(sources,self.queue[(step+self.delay_steps)%self.qlen])
|
||||
sort_t+=stats['sort_seconds'];reduce_t+=stats['reduce_seconds'];contrib+=stats['contributions']
|
||||
if capture and sources.numel():
|
||||
x=sources.cpu().numpy().astype(np.int32,copy=True);self.event_nodes.append(x);self.event_steps.append(np.full(len(x),step,np.int32))
|
||||
self.step_index+=1
|
||||
torch.cuda.synchronize();wall=time.perf_counter()-t0
|
||||
es=np.concatenate(self.event_steps) if self.event_steps else np.empty(0,np.int32)
|
||||
en=np.concatenate(self.event_nodes) if self.event_nodes else np.empty(0,np.int32)
|
||||
return {'step':es,'neuron_idx':en,'counts':self.counts.cpu().numpy().copy(),'v':self.v.cpu().numpy().copy(),'g':self.g.cpu().numpy().copy(),
|
||||
'metrics':{'wall_seconds':wall,'biological_ms':duration_ms,'realtime_factor':wall/(duration_ms/1000),
|
||||
'peak_vram_bytes':int(torch.cuda.max_memory_allocated()),'contributions':int(contrib),
|
||||
'sort_submit_seconds':sort_t,'reduce_submit_seconds':reduce_t,'events':len(en),'active':int((self.counts>0).sum().item())}}
|
||||
|
||||
|
||||
def cpu_ordered_sum(edge_ids:np.ndarray,indices:np.ndarray,weights:np.ndarray):
|
||||
edge_ids=np.asarray(edge_ids,np.int64);targets=np.asarray(indices[edge_ids],np.int32)
|
||||
order=np.argsort(targets,kind='stable');ordered=edge_ids[order];ordered_target=targets[order]
|
||||
unique=[];sums=[];begin=0
|
||||
while begin<len(ordered):
|
||||
target=int(ordered_target[begin]);end=begin+1;acc=np.float32(0)
|
||||
while end<len(ordered) and ordered_target[end]==target:end+=1
|
||||
for e in ordered[begin:end]:acc=np.float32(acc+np.float32(weights[e]))
|
||||
unique.append(target);sums.append(acc);begin=end
|
||||
return np.asarray(unique,np.int32),np.asarray(sums,np.float32),ordered
|
||||
|
||||
|
||||
def save_trial(root:Path,run:dict):
|
||||
root.mkdir(parents=True,exist_ok=True)
|
||||
pq.write_table(pa.table({'step':run['step'],'timestamp_ms':run['step'].astype(np.float64)*.1,'neuron_idx':run['neuron_idx']}),root/'spike-events.parquet',compression='zstd')
|
||||
for name,key,dtype in [('per-neuron-spike-counts.int32.npy','counts',np.int32),('final-voltage.float32.npy','v',np.float32),('final-g.float32.npy','g',np.float32)]:np.save(root/name,run[key].astype(dtype),allow_pickle=False)
|
||||
save(root/'statistics.json',run['metrics']|{'event_sha256':sha_bytes(run['step'],run['neuron_idx']),
|
||||
'counts_sha256':sha_bytes(run['counts']),'v_sha256':sha_bytes(run['v']),'g_sha256':sha_bytes(run['g'])})
|
||||
|
||||
|
||||
def phase4c_trials(cfg,rt,repeats):
|
||||
exp=Path(cfg['paths']['experiment_root']);rows=[]
|
||||
for spec in cfg['reference_trials']:
|
||||
target=np.flatnonzero(rt.pops[spec['population']]).astype(np.int64)
|
||||
baseline=None
|
||||
for repeat in range(repeats):
|
||||
run=rt.run(target,float(spec['rate_hz']),int(spec['seed']),float(spec['duration_ms']),True)
|
||||
if baseline is None:
|
||||
baseline={k:run[k].copy() for k in ('step','neuron_idx','counts','v','g')};save_trial(exp/'phase4c-v2/reference'/spec['name'],run)
|
||||
exact={k:bool(np.array_equal(run[k],baseline[k])) for k in ('step','neuron_idx','counts','v','g')}
|
||||
row={'trial':spec['name'],'repeat':repeat,**exact,**run['metrics']};rows.append(row)
|
||||
save(exp/'phase4c-v2/reproducibility-matrix.json',rows)
|
||||
print(json.dumps({'trial':spec['name'],'repeat':repeat,'exact':all(exact.values()),'wall':run['metrics']['wall_seconds']}),flush=True)
|
||||
summary={'repeats':repeats,'matrix':{s['name']:{k:sum(r[k] for r in rows if r['trial']==s['name']) for k in ('step','neuron_idx','counts','v','g')} for s in cfg['reference_trials']},
|
||||
'all_exact':all(all(r[k] for k in ('step','neuron_idx','counts','v','g')) for r in rows),
|
||||
'performance':{s['name']:{'mean_wall_seconds':float(np.mean([r['wall_seconds'] for r in rows if r['trial']==s['name']])),
|
||||
'mean_realtime_factor':float(np.mean([r['realtime_factor'] for r in rows if r['trial']==s['name']])),
|
||||
'peak_vram_bytes':max(r['peak_vram_bytes'] for r in rows if r['trial']==s['name']),
|
||||
'mean_contributions':float(np.mean([r['contributions'] for r in rows if r['trial']==s['name']]))} for s in cfg['reference_trials']}}
|
||||
save(exp/'phase4c-v2/summary.json',summary)
|
||||
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument('--config',type=Path,required=True);ap.add_argument('task',choices=['phase4c']);ap.add_argument('--repeats',type=int)
|
||||
a=ap.parse_args();cfg=yaml.safe_load(a.config.read_text());exp=Path(cfg['paths']['experiment_root']);model=Path(cfg['paths']['model_root'])
|
||||
for sub in ('numerical-contract','cpu-reference','gpu-reference','threshold-3942','fanout-tests','phase4c-v2','historical-comparison','brainio','activity-stream','eligibility-replay','performance','revalidation','regression','audit','manifests'):(exp/sub).mkdir(parents=True,exist_ok=True)
|
||||
rt=CanonicalRuntime(cfg,model,exp)
|
||||
phase4c_trials(cfg,rt,a.repeats or int(cfg['repeats_per_trial']))
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,51 @@
|
||||
phase: 4P
|
||||
name: MaleCNS-LIF-runtime-v2-deterministic-candidate
|
||||
description: Shiu-like LIF dynamics adapted to MaleCNS v1.0 with canonical deterministic float32 reduction
|
||||
paths:
|
||||
connectome_root: /data/digital-drosophila/datasets/male-cns-v1.0/processed/neuron-only
|
||||
historical_runtime_root: /data/digital-drosophila/models/male-cns-lif/runtime-v1
|
||||
historical_phase4c_root: /data/digital-drosophila/experiments/phase4c-runtime
|
||||
phase4o_root: /data/digital-drosophila/experiments/phase4o-runtime-credit
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4p-deterministic-runtime
|
||||
expected_graph: {neurons: 166700, directed_edges: 25582938, synaptic_contact_weight_sum: 124177617}
|
||||
model:
|
||||
dt_ms: 0.1
|
||||
membrane_tau_ms: 20.0
|
||||
synaptic_tau_ms: 5.0
|
||||
refractory_ms: 2.2
|
||||
resting_potential_mv: -52.0
|
||||
reset_potential_mv: -52.0
|
||||
threshold_mv: -45.0
|
||||
synaptic_delay_ms: 1.8
|
||||
w_syn_mv_per_anatomical_synapse: 0.275
|
||||
integration: forward_euler
|
||||
sign_policy:
|
||||
neurotransmitter_sign: {acetylcholine: 1, gaba: -1, glutamate: -1, histamine: -1, dopamine: 1, octopamine: 1, serotonin: 1}
|
||||
unknown_sign: 0
|
||||
unclear_sign: 0
|
||||
device:
|
||||
required_name_substring: RTX 4090
|
||||
visible_gpu_uuid: GPU-79f3a85d-adce-7833-7003-6488140ca596
|
||||
canonical_contract:
|
||||
contribution_key: [target_idx, source_idx, edge_id]
|
||||
equivalent_implementation_key: [target_idx, edge_id]
|
||||
edge_id_property: CSR edge_id is globally ordered by source row and within-row edge position
|
||||
accumulation: sequential IEEE-754 float32 round-to-nearest-even using __fadd_rn
|
||||
state_update: fused explicit IEEE-754 float32 operations using __fmul_rn/__fadd_rn/__fsub_rn
|
||||
threshold: V >= V_threshold
|
||||
reference_trials:
|
||||
- {name: auditory-10hz, population: auditory, rate_hz: 10, seed: 42001, duration_ms: 1000.0}
|
||||
- {name: auditory-100hz, population: auditory, rate_hz: 100, seed: 42001, duration_ms: 1000.0}
|
||||
- {name: olfactory-50hz, population: olfactory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
- {name: mechanosensory-50hz, population: mechanosensory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
- {name: gustatory-50hz, population: gustatory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
repeats_per_trial: 20
|
||||
fanout_test_cases: 500
|
||||
eligibility_fixed_event_repeats: 10
|
||||
prohibitions:
|
||||
modify_biological_parameters: true
|
||||
overwrite_historical_reference: true
|
||||
nondeterministic_atomic_scientific_backend: true
|
||||
threshold_epsilon: true
|
||||
learning: true
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Instrumented 100 ms canonical-backend cost audit."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import resource
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
import phase4p
|
||||
|
||||
|
||||
def main() -> None:
|
||||
model = Path("/data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2")
|
||||
cfg = yaml.safe_load((model / "phase4p.yaml").read_text()); exp = Path(cfg["paths"]["experiment_root"])
|
||||
rt = phase4p.CanonicalRuntime(cfg, model, exp); spec = cfg["reference_trials"][1]
|
||||
target_np = np.flatnonzero(rt.pops[spec["population"]]).astype(np.int64)
|
||||
targets = torch.as_tensor(target_np, dtype=torch.int64, device=rt.device); steps = 1000
|
||||
forced = phase4p.phase4c.generate_forced(steps, len(target_np), float(spec["rate_hz"]), rt.dt, int(spec["seed"]), rt.device)
|
||||
rt.reset(); samples = []; stop = threading.Event()
|
||||
def sample():
|
||||
while not stop.is_set():
|
||||
try:
|
||||
text = subprocess.check_output(["nvidia-smi", "--id=" + cfg["device"]["visible_gpu_uuid"],
|
||||
"--query-gpu=utilization.gpu,memory.used", "--format=csv,noheader,nounits"], text=True).strip()
|
||||
u, m = text.split(","); samples.append({"time": time.time(), "gpu_util_percent": float(u), "gpu_memory_mib": float(m)})
|
||||
except Exception as e: samples.append({"error": str(e)})
|
||||
stop.wait(0.10)
|
||||
thread = threading.Thread(target=sample, daemon=True); thread.start()
|
||||
use0 = resource.getrusage(resource.RUSAGE_SELF); cpu0 = use0.ru_utime + use0.ru_stime
|
||||
torch.cuda.reset_peak_memory_stats(); torch.cuda.synchronize()
|
||||
wall0 = time.perf_counter(); state_submit = force_submit = sort_submit = reduce_submit = 0.0; contributions = 0
|
||||
for step in range(steps):
|
||||
slot = step % rt.qlen; t = time.perf_counter()
|
||||
rt.ext.state_step(rt.v, rt.g, rt.ref, rt.queue[slot], rt.spike_u8, rt.counts,
|
||||
float(rt.gd), float(rt.alpha), float(rt.rest), float(rt.resetv), float(rt.threshold), rt.ref_steps)
|
||||
state_submit += time.perf_counter() - t; t = time.perf_counter()
|
||||
rt.ext.force_step(rt.v, rt.ref, rt.spike_u8, rt.counts, targets, forced[step], float(rt.resetv), rt.ref_steps)
|
||||
force_submit += time.perf_counter() - t
|
||||
sources = torch.nonzero(rt.spike_u8, as_tuple=False).flatten()
|
||||
st = rt.canonical_fanout(sources, rt.queue[(step + rt.delay_steps) % rt.qlen])
|
||||
sort_submit += st["sort_seconds"]; reduce_submit += st["reduce_seconds"]; contributions += st["contributions"]
|
||||
torch.cuda.synchronize(); wall = time.perf_counter() - wall0
|
||||
use1 = resource.getrusage(resource.RUSAGE_SELF); cpu1 = use1.ru_utime + use1.ru_stime
|
||||
stop.set(); thread.join()
|
||||
good = [x for x in samples if "gpu_util_percent" in x]
|
||||
out = {"trial": "auditory-100hz", "biological_ms": 100.0, "steps": steps, "wall_seconds": wall,
|
||||
"wall_per_100ms_biological": wall, "realtime_factor": wall / 0.1, "steps_per_second": steps / wall,
|
||||
"process_CPU_percent_of_one_core": 100 * (cpu1 - cpu0) / wall,
|
||||
"GPU_utilization_percent_mean": float(np.mean([x["gpu_util_percent"] for x in good])) if good else None,
|
||||
"GPU_utilization_percent_max": float(np.max([x["gpu_util_percent"] for x in good])) if good else None,
|
||||
"nvidia_smi_memory_mib_max": float(np.max([x["gpu_memory_mib"] for x in good])) if good else None,
|
||||
"torch_peak_VRAM_bytes": int(torch.cuda.max_memory_allocated()), "contribution_count": int(contributions),
|
||||
"CPU_submit_seconds": {"state_integration": state_submit, "external_force": force_submit,
|
||||
"canonical_sort_and_segment_setup": sort_submit, "segmented_reduction": reduce_submit},
|
||||
"sampling_interval_seconds": 0.10, "samples": samples,
|
||||
"interpretation": "Submission sub-times overlap asynchronous CUDA work; synchronized total wall time is authoritative."}
|
||||
p = exp / "performance/instrumented-100ms.json"; p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(json.dumps(out, indent=2, sort_keys=True) + "\n"); print(json.dumps(out, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,25 @@
|
||||
#include <torch/extension.h>
|
||||
|
||||
void state_step_cuda(torch::Tensor v, torch::Tensor g, torch::Tensor refractory,
|
||||
torch::Tensor arrival, torch::Tensor spikes,
|
||||
torch::Tensor spike_counts, double g_decay, double alpha,
|
||||
double rest, double reset, double threshold,
|
||||
int64_t refractory_steps);
|
||||
void force_step_cuda(torch::Tensor v, torch::Tensor refractory,
|
||||
torch::Tensor spikes, torch::Tensor spike_counts,
|
||||
torch::Tensor targets, torch::Tensor forced,
|
||||
double reset, int64_t refractory_steps);
|
||||
void fill_edges_cuda(torch::Tensor indptr, torch::Tensor sources,
|
||||
torch::Tensor output_offsets, torch::Tensor output);
|
||||
void segmented_sum_cuda(torch::Tensor sorted_edge_ids,
|
||||
torch::Tensor segment_starts,
|
||||
torch::Tensor segment_targets,
|
||||
torch::Tensor weights,
|
||||
torch::Tensor destination);
|
||||
|
||||
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
|
||||
m.def("state_step", &state_step_cuda, "Canonical float32 LIF update (CUDA)");
|
||||
m.def("force_step", &force_step_cuda, "Canonical external spike injection (CUDA)");
|
||||
m.def("fill_edges", &fill_edges_cuda, "Source/edge ordered CSR expansion (CUDA)");
|
||||
m.def("segmented_sum", &segmented_sum_cuda, "Fixed-order sequential segmented float32 sum (CUDA)");
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
#include <torch/extension.h>
|
||||
#include <c10/cuda/CUDAException.h>
|
||||
#include <cuda.h>
|
||||
#include <cuda_runtime.h>
|
||||
#include <cstdint>
|
||||
|
||||
namespace {
|
||||
void check_cuda(const torch::Tensor& x) {
|
||||
TORCH_CHECK(x.is_cuda(), "all tensors must be CUDA tensors");
|
||||
TORCH_CHECK(x.is_contiguous(), "all tensors must be contiguous");
|
||||
}
|
||||
|
||||
__global__ void state_kernel(float* v, float* g, int* refractory,
|
||||
float* arrival, unsigned char* spikes,
|
||||
int* counts, int n, float gd, float alpha,
|
||||
float rest, float reset, float threshold,
|
||||
int ref_steps) {
|
||||
int i=blockIdx.x*blockDim.x+threadIdx.x;
|
||||
if(i>=n) return;
|
||||
float gi=__fadd_rn(__fmul_rn(g[i],gd),arrival[i]);
|
||||
g[i]=gi; arrival[i]=0.0f;
|
||||
int r=refractory[i]; r=r>0?r-1:0; refractory[i]=r;
|
||||
float vi=v[i]; bool available=r==0;
|
||||
if(available){
|
||||
float leak=__fsub_rn(vi,rest);
|
||||
float drive=__fsub_rn(gi,leak);
|
||||
vi=__fadd_rn(vi,__fmul_rn(alpha,drive));
|
||||
} else vi=reset;
|
||||
bool fired=available && vi>=threshold;
|
||||
if(fired){vi=reset;refractory[i]=ref_steps;counts[i]+=1;}
|
||||
v[i]=vi;spikes[i]=fired?1:0;
|
||||
}
|
||||
|
||||
__global__ void force_kernel(float* v,int* refractory,unsigned char* spikes,
|
||||
int* counts,const int64_t* targets,
|
||||
const unsigned char* forced,int n,float reset,
|
||||
int ref_steps){
|
||||
int j=blockIdx.x*blockDim.x+threadIdx.x;
|
||||
if(j>=n||!forced[j]) return;
|
||||
int64_t i=targets[j]; if(spikes[i]||refractory[i]!=0) return;
|
||||
spikes[i]=1;v[i]=reset;refractory[i]=ref_steps;counts[i]+=1;
|
||||
}
|
||||
|
||||
__global__ void fill_edges_kernel(const int64_t* indptr,const int64_t* sources,
|
||||
const int64_t* offsets,int64_t* output,int n){
|
||||
int j=blockIdx.x*blockDim.x+threadIdx.x;
|
||||
if(j>=n) return;
|
||||
int64_t src=sources[j],begin=indptr[src],degree=indptr[src+1]-begin,out=offsets[j];
|
||||
for(int64_t k=0;k<degree;++k) output[out+k]=begin+k;
|
||||
}
|
||||
|
||||
// Exactly one CUDA thread owns a target segment. The loop order is the
|
||||
// canonical target -> source -> edge_id order supplied by the host.
|
||||
__global__ void segmented_sum_kernel(const int64_t* edges,const int64_t* starts,
|
||||
const int* targets,const float* weights,
|
||||
float* destination,int segments,
|
||||
int64_t edge_count){
|
||||
int s=blockIdx.x*blockDim.x+threadIdx.x;
|
||||
if(s>=segments) return;
|
||||
int64_t begin=starts[s],end=(s+1<segments)?starts[s+1]:edge_count;
|
||||
float acc=0.0f;
|
||||
for(int64_t j=begin;j<end;++j) acc=__fadd_rn(acc,weights[edges[j]]);
|
||||
destination[targets[s]]=__fadd_rn(destination[targets[s]],acc);
|
||||
}
|
||||
}
|
||||
|
||||
void state_step_cuda(torch::Tensor v,torch::Tensor g,torch::Tensor refractory,
|
||||
torch::Tensor arrival,torch::Tensor spikes,torch::Tensor counts,
|
||||
double gd,double alpha,double rest,double reset,double threshold,
|
||||
int64_t ref_steps){
|
||||
check_cuda(v);check_cuda(g);check_cuda(refractory);check_cuda(arrival);check_cuda(spikes);check_cuda(counts);
|
||||
int n=v.numel();state_kernel<<<(n+255)/256,256>>>(v.data_ptr<float>(),g.data_ptr<float>(),refractory.data_ptr<int>(),arrival.data_ptr<float>(),spikes.data_ptr<unsigned char>(),counts.data_ptr<int>(),n,(float)gd,(float)alpha,(float)rest,(float)reset,(float)threshold,(int)ref_steps);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
||||
}
|
||||
void force_step_cuda(torch::Tensor v,torch::Tensor refractory,torch::Tensor spikes,
|
||||
torch::Tensor counts,torch::Tensor targets,torch::Tensor forced,
|
||||
double reset,int64_t ref_steps){
|
||||
check_cuda(targets);check_cuda(forced);int n=targets.numel();force_kernel<<<(n+255)/256,256>>>(v.data_ptr<float>(),refractory.data_ptr<int>(),spikes.data_ptr<unsigned char>(),counts.data_ptr<int>(),targets.data_ptr<int64_t>(),forced.data_ptr<unsigned char>(),n,(float)reset,(int)ref_steps);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
||||
}
|
||||
void fill_edges_cuda(torch::Tensor indptr,torch::Tensor sources,torch::Tensor offsets,torch::Tensor output){
|
||||
check_cuda(indptr);check_cuda(sources);check_cuda(offsets);check_cuda(output);int n=sources.numel();fill_edges_kernel<<<(n+255)/256,256>>>(indptr.data_ptr<int64_t>(),sources.data_ptr<int64_t>(),offsets.data_ptr<int64_t>(),output.data_ptr<int64_t>(),n);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
||||
}
|
||||
void segmented_sum_cuda(torch::Tensor edges,torch::Tensor starts,torch::Tensor targets,
|
||||
torch::Tensor weights,torch::Tensor destination){
|
||||
check_cuda(edges);check_cuda(starts);check_cuda(targets);check_cuda(weights);check_cuda(destination);
|
||||
TORCH_CHECK(edges.scalar_type()==torch::kInt64&&starts.scalar_type()==torch::kInt64&&targets.scalar_type()==torch::kInt32,"index dtype mismatch");
|
||||
int n=targets.numel();if(n) segmented_sum_kernel<<<(n+255)/256,256>>>(edges.data_ptr<int64_t>(),starts.data_ptr<int64_t>(),targets.data_ptr<int>(),weights.data_ptr<float>(),destination.data_ptr<float>(),n,edges.numel());C10_CUDA_KERNEL_LAUNCH_CHECK();
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only frozen-integrity verification and final Phase 4P sealing."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
import phase4p_validate
|
||||
from phase4h_audit import verify_checksum
|
||||
from phase4k import dataset_checks
|
||||
from phase4m import frozen_checks, frozen_hashes
|
||||
from phase4n import load_config
|
||||
|
||||
|
||||
ROOT = Path("/data/digital-drosophila")
|
||||
|
||||
|
||||
def save(path: Path, value: object) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, indent=2, sort_keys=True, default=str) + "\n")
|
||||
|
||||
|
||||
def read(path: Path): return json.loads(path.read_text())
|
||||
|
||||
|
||||
def integrity() -> dict:
|
||||
ncfg, mcfg, cfg, _ = load_config(ROOT / "models/male-cns-lif/credit-assignment-v1/phase4n.yaml")
|
||||
checks = {}
|
||||
for label, root, rel in [
|
||||
("Phase4M experiment", Path(mcfg["experiment_root"]), "manifests/checksums.sha256"),
|
||||
("Phase4M model", Path(mcfg["model_root"]), "checksums.sha256"),
|
||||
("Phase4N experiment", Path(ncfg["experiment_root"]), "manifests/checksums.sha256"),
|
||||
("Phase4N model", Path(ncfg["model_root"]), "checksums.sha256")]:
|
||||
checks[label] = verify_checksum(root, root / rel)
|
||||
earlier = frozen_checks(cfg); datasets = dataset_checks(cfg); current = frozen_hashes(cfg)
|
||||
saved = read(Path(mcfg["experiment_root"]) / "audit/final-summary.json")["integrity"]["after"]
|
||||
phase4o = verify_checksum(ROOT / "experiments/phase4o-runtime-credit",
|
||||
ROOT / "experiments/phase4o-runtime-credit/manifests/checksums.sha256")
|
||||
report = {"Phase1_to_3B": datasets, "Phase4C_to_4L": earlier, "Phase4M_4N": checks,
|
||||
"Phase4O": phase4o, "frozen_input_hashes_unchanged": current == saved}
|
||||
report["all_static_checks_passed"] = bool(datasets["all_passed"] and earlier["all_passed"] and
|
||||
all(x["passed"] for x in checks.values()) and phase4o["passed"] and current == saved)
|
||||
return report
|
||||
|
||||
|
||||
def seal_dir(root: Path, control: Path) -> int:
|
||||
rows = []
|
||||
for p in sorted(root.rglob("*")):
|
||||
if p.is_file() and p != control and p.name not in ("checksums.sha256", "MANIFEST.json"):
|
||||
h = hashlib.sha256()
|
||||
with p.open("rb") as f:
|
||||
for chunk in iter(lambda: f.read(8 << 20), b""): h.update(chunk)
|
||||
rows.append((h.hexdigest(), p.relative_to(root).as_posix(), p.stat().st_size))
|
||||
control.parent.mkdir(parents=True, exist_ok=True)
|
||||
control.write_text("".join(f"{h} {rel}\n" for h, rel, _ in rows))
|
||||
save(control.parent / "MANIFEST.json", {"root": str(root), "files": [{"path": r, "sha256": h, "size": s} for h, r, s in rows]})
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
cfg = yaml.safe_load((ROOT / "models/male-cns-lif/deterministic-runtime-v2/phase4p.yaml").read_text())
|
||||
exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
frozen = integrity(); save(exp / "audit/frozen-integrity-final.json", frozen)
|
||||
gate = read(exp / "audit/candidate-gate.json"); p4c = read(exp / "phase4c-v2/summary.json")
|
||||
reval = read(exp / "revalidation/summary.json"); threshold = read(exp / "threshold-3942/summary.json")
|
||||
fanout = read(exp / "fanout-tests/summary.json"); history = read(exp / "historical-comparison/summary.json")
|
||||
brainio = read(exp / "brainio/summary.json"); activity = read(exp / "activity-stream/summary.json")
|
||||
eligibility = read(exp / "eligibility-replay/summary.json")
|
||||
performance = read(exp / "performance/instrumented-100ms.json")
|
||||
highest = reval["highest_revalidated"]
|
||||
final = {"phase": "4P", "status": gate["status"], "candidate_generated": gate["candidate_generated"],
|
||||
"numerical_contract": "MaleCNS-NumericalContract-v1", "canonical_order": ["target_idx", "source_idx", "edge_id"],
|
||||
"neuron_3942": threshold, "real_fanout": fanout, "phase4c_5x20": p4c,
|
||||
"historical_comparison": history, "BrainIO": brainio, "ActivityStream": activity,
|
||||
"eligibility_fixed_event_replay": eligibility, "performance": performance, "candidate_gates": gate["gates"],
|
||||
"revalidation": reval, "highest_revalidated": highest,
|
||||
"frozen_integrity": frozen, "prohibited_follow_on_started": False,
|
||||
"limitations": [
|
||||
"Canonical stable sorting plus one-thread-per-target reduction is correctness-first and 2.6-5.0x slower than biological real time for the 1 s trials.",
|
||||
"Historical runtime-v1 full population/state distributions were not persisted for all 50 Phase4O repeats; total-spike envelopes use all runs while population/V/g comparisons use the frozen historical reference.",
|
||||
"The compiler emitted a generic c++ versus g++ warning even though the cached extension loaded and all exact regressions passed."]}
|
||||
save(exp / "audit/final-summary.json", final)
|
||||
candidate = model / "candidate"; shutil.copy2(model / "phase4p_revalidate.py", candidate / "phase4p_revalidate.py")
|
||||
shutil.copy2(model / "phase4p_finalize.py", candidate / "phase4p_finalize.py")
|
||||
shutil.copy2(model / "phase4p_benchmark.py", candidate / "phase4p_benchmark.py")
|
||||
save(candidate / "VERSION.json", {"name": gate["status"], "numerical_contract": "MaleCNS-NumericalContract-v1",
|
||||
"revalidated_through": highest, "generated_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"biological_parameters_changed": False, "historical_references_overwritten": False})
|
||||
candidate_files = seal_dir(candidate, candidate / "checksums.sha256")
|
||||
phase4p_validate.make_manifest(cfg, final)
|
||||
own = verify_checksum(candidate, candidate / "checksums.sha256")
|
||||
print(json.dumps({"status": final["status"], "highest_revalidated": highest,
|
||||
"frozen_integrity": frozen["all_static_checks_passed"],
|
||||
"candidate_files": candidate_files, "candidate_checksum": own["passed"]}, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,449 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Ordered Phase 4P-B dynamic revalidation on the canonical runtime.
|
||||
|
||||
R1/R2 consume the already sealed Phase 4P results. R3-R6 are run in order
|
||||
and the script stops immediately if a level cannot reproduce its declared
|
||||
historical result. No frozen artifact is written.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
import phase4p
|
||||
from phase4l import pn_drive
|
||||
from phase4m import load_odor_inputs
|
||||
|
||||
|
||||
ROOT = Path("/data/digital-drosophila")
|
||||
|
||||
|
||||
def save(path: Path, value: object) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, indent=2, sort_keys=True, default=str) + "\n")
|
||||
|
||||
|
||||
def ahash(*values: np.ndarray) -> str:
|
||||
h = hashlib.sha256()
|
||||
for value in values: h.update(np.ascontiguousarray(value).tobytes())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def metrics(a: np.ndarray, b: np.ndarray) -> dict[str, float]:
|
||||
a = a.astype(np.float64); b = b.astype(np.float64)
|
||||
aa = a > 0; bb = b > 0; union = np.count_nonzero(aa | bb)
|
||||
den = np.linalg.norm(a) * np.linalg.norm(b)
|
||||
return {"euclidean": float(np.linalg.norm(a - b)),
|
||||
"jaccard": float(np.count_nonzero(aa & bb) / union) if union else 1.0,
|
||||
"cosine": float(np.dot(a, b) / den) if den else (1.0 if not np.any(a) and not np.any(b) else 0.0)}
|
||||
|
||||
|
||||
class DeterministicAPLSim:
|
||||
"""Frozen APL overlay on top of the Phase 4P canonical LIF kernel."""
|
||||
def __init__(self, rt: phase4p.CanonicalRuntime):
|
||||
self.rt = rt; self.n = rt.n; self.device = rt.device; self.dt = rt.dt
|
||||
meta = rt.table.to_pandas().sort_values("neuron_idx").reset_index(drop=True)
|
||||
self.meta = meta
|
||||
cls = meta.annotation_class.fillna("").astype(str).to_numpy()
|
||||
typ = meta.annotation_type.fillna("").astype(str).to_numpy()
|
||||
self.kc = np.flatnonzero(cls == "Kenyon_Cell").astype(np.int32)
|
||||
self.mbon = np.flatnonzero(cls == "MBON").astype(np.int32)
|
||||
self.pn = np.flatnonzero(cls == "ALPN").astype(np.int32)
|
||||
self.apl = np.flatnonzero(typ == "APL").astype(np.int32)
|
||||
if len(self.apl) != 2 or len(self.kc) != 4064: raise RuntimeError("frozen APL/KC identity changed")
|
||||
self.sources = np.repeat(np.arange(self.n, dtype=np.int32), np.diff(rt.indptr_np).astype(np.int64))
|
||||
idx = rt.indices_np; raw = rt.raw_np
|
||||
apl_out = np.isin(self.sources, self.apl)
|
||||
self.apl_out_ids = np.flatnonzero(apl_out).astype(np.int64)
|
||||
input_matrix = np.zeros((len(self.apl), self.n), np.float32)
|
||||
output_matrix = np.zeros((len(self.apl), len(self.kc)), np.float32)
|
||||
kc_lookup = np.full(self.n, -1, np.int32); kc_lookup[self.kc] = np.arange(len(self.kc), dtype=np.int32)
|
||||
allowed = np.isin(self.sources, np.concatenate((self.kc, self.pn))) & np.isin(idx, self.apl)
|
||||
apl_lookup = {int(x): i for i, x in enumerate(self.apl)}
|
||||
for eid in np.flatnonzero(allowed): input_matrix[apl_lookup[int(idx[eid])], int(self.sources[eid])] += float(raw[eid])
|
||||
for eid in np.flatnonzero(apl_out & np.isin(idx, self.kc)):
|
||||
output_matrix[apl_lookup[int(self.sources[eid])], kc_lookup[int(idx[eid])]] += float(raw[eid])
|
||||
self.input_scale = float(input_matrix[input_matrix > 0].sum() / np.count_nonzero(np.any(input_matrix > 0, axis=0)))
|
||||
self.output_scale = float(output_matrix[output_matrix > 0].mean())
|
||||
self.input_gpu = torch.as_tensor(input_matrix / self.input_scale, dtype=torch.float32, device=self.device)
|
||||
self.output_gpu = torch.as_tensor(output_matrix / self.output_scale, dtype=torch.float32, device=self.device)
|
||||
self.kc_gpu = torch.as_tensor(self.kc, dtype=torch.int64, device=self.device)
|
||||
self.apl_out_gpu = torch.as_tensor(self.apl_out_ids, dtype=torch.int64, device=self.device)
|
||||
self.base = rt.weights.clone()
|
||||
p4h = ROOT / "experiments/phase4h-dopamine-plasticity"
|
||||
self.identity = pd.read_parquet(ROOT / "experiments/phase4n-credit-assignment/edge-identity/frozen-edge-identity.parquet")
|
||||
self.edge_ids = self.identity.edge_id.to_numpy(np.int64)
|
||||
self.edge_gpu = torch.as_tensor(self.edge_ids, dtype=torch.int64, device=self.device)
|
||||
self.base_candidate = self.base[self.edge_gpu].cpu().numpy().astype(np.float32)
|
||||
self.confirmed = pd.read_parquet(p4h / "mappings/dan-to-mbon-confirmed.parquet")
|
||||
down = pd.read_parquet(p4h / "mappings/mbon-to-downstream-actual-edges.parquet")
|
||||
self.downstream = np.unique(down.target_neuron_idx.to_numpy(np.int32))
|
||||
|
||||
def dan_targets(self, dtype: str, compartment: str) -> np.ndarray:
|
||||
x = self.confirmed[(self.confirmed.source_annotation_type == dtype) &
|
||||
(self.confirmed.source_compartment_verbatim == compartment)]
|
||||
out = np.sort(x.source_neuron_idx.unique().astype(np.int32))
|
||||
if not len(out): raise RuntimeError(f"no confirmed DAN for {dtype}/{compartment}")
|
||||
return out
|
||||
|
||||
def install(self, E: np.ndarray | None) -> None:
|
||||
self.rt.weights.copy_(self.base)
|
||||
if E is not None:
|
||||
self.rt.weights[self.edge_gpu] = torch.as_tensor(self.base_candidate * E.astype(np.float32), device=self.device)
|
||||
self.rt.weights[self.apl_out_gpu] = 0.0
|
||||
|
||||
def run(self, targets: np.ndarray, rates_hz: np.ndarray, seed: int, duration_ms: float,
|
||||
E: np.ndarray | None = None, dan_targets: np.ndarray | None = None,
|
||||
dan_pulses: list[float] | None = None, collect_events: bool = False) -> dict[str, Any]:
|
||||
targets = np.asarray(targets, np.int32); rates_hz = np.asarray(rates_hz, np.float32)
|
||||
self.install(E); self.rt.reset(); steps = int(round(duration_ms / self.dt))
|
||||
target_gpu = torch.as_tensor(targets, dtype=torch.int64, device=self.device)
|
||||
gen = torch.Generator(device=self.device); gen.manual_seed(int(seed))
|
||||
probs = torch.as_tensor(rates_hz * self.dt / 1000.0, dtype=torch.float32, device=self.device)
|
||||
forced = (torch.rand((steps, len(targets)), generator=gen, device=self.device) < probs[None, :]).to(torch.uint8)
|
||||
dan = np.empty(0, np.int32) if dan_targets is None else np.asarray(dan_targets, np.int32)
|
||||
dan_gpu = torch.as_tensor(dan, dtype=torch.int64, device=self.device)
|
||||
dan_forced = torch.zeros((steps, len(dan)), dtype=torch.uint8, device=self.device)
|
||||
for pulse in dan_pulses or []:
|
||||
s = int(round(pulse / self.dt))
|
||||
if 0 <= s < steps: dan_forced[s] = True
|
||||
dan_steps = {int(round(x / self.dt)) for x in (dan_pulses or [])}
|
||||
apl_state = torch.zeros(len(self.apl), dtype=torch.float32, device=self.device)
|
||||
kc_trace = torch.zeros(len(self.kc), dtype=torch.float32, device=self.device)
|
||||
first = torch.full((self.n,), -1, dtype=torch.int32, device=self.device)
|
||||
trace_at_dan = []
|
||||
apl_decay = np.float32(math.exp(-self.dt / 20.0)); trace_decay = np.float32(math.exp(-self.dt / 100.0))
|
||||
kc_steps: list[np.ndarray] = []; kc_nodes: list[np.ndarray] = []
|
||||
torch.cuda.synchronize(); t0 = time.perf_counter()
|
||||
for step in range(steps):
|
||||
slot = step % self.rt.qlen
|
||||
self.rt.ext.state_step(self.rt.v, self.rt.g, self.rt.ref, self.rt.queue[slot], self.rt.spike_u8, self.rt.counts,
|
||||
float(self.rt.gd), float(self.rt.alpha), float(self.rt.rest), float(self.rt.resetv),
|
||||
float(self.rt.threshold), self.rt.ref_steps)
|
||||
self.rt.ext.force_step(self.rt.v, self.rt.ref, self.rt.spike_u8, self.rt.counts,
|
||||
target_gpu, forced[step], float(self.rt.resetv), self.rt.ref_steps)
|
||||
if len(dan):
|
||||
self.rt.ext.force_step(self.rt.v, self.rt.ref, self.rt.spike_u8, self.rt.counts,
|
||||
dan_gpu, dan_forced[step], float(self.rt.resetv), self.rt.ref_steps)
|
||||
sources = torch.nonzero(self.rt.spike_u8, as_tuple=False).flatten()
|
||||
if sources.numel():
|
||||
missing = first[sources] < 0; first[sources[missing]] = step
|
||||
self.rt.canonical_fanout(sources, self.rt.queue[(step + self.rt.delay_steps) % self.rt.qlen])
|
||||
kc_spike = self.rt.spike_u8[self.kc_gpu]
|
||||
kc_trace.mul_(float(trace_decay)).add_(kc_spike.to(torch.float32)).clamp_max_(1.0)
|
||||
if step in dan_steps: trace_at_dan.append(kc_trace.clone())
|
||||
if collect_events:
|
||||
active = torch.nonzero(kc_spike, as_tuple=False).flatten()
|
||||
if active.numel():
|
||||
node = self.kc_gpu[active].cpu().numpy().astype(np.int32)
|
||||
kc_nodes.append(node); kc_steps.append(np.full(len(node), step, np.int32))
|
||||
# Frozen APL graded-inhibition overlay. Each KC target is unique in
|
||||
# this assignment, avoiding index_add atomics.
|
||||
apl_state.mul_(float(apl_decay)).add_(torch.mv(self.input_gpu, self.rt.spike_u8.to(torch.float32)))
|
||||
inhibition = torch.matmul(apl_state, self.output_gpu) * np.float32(0.01)
|
||||
self.rt.g[self.kc_gpu] = self.rt.g[self.kc_gpu] - inhibition
|
||||
self.rt.step_index += 1
|
||||
torch.cuda.synchronize(); wall = time.perf_counter() - t0
|
||||
counts = self.rt.counts.cpu().numpy().copy(); first_np = first.cpu().numpy()
|
||||
ev_step = np.concatenate(kc_steps) if kc_steps else np.empty(0, np.int32)
|
||||
ev_node = np.concatenate(kc_nodes) if kc_nodes else np.empty(0, np.int32)
|
||||
return {"counts": counts, "kc_counts": counts[self.kc], "mbon_counts": counts[self.mbon],
|
||||
"downstream_counts": counts[self.downstream], "first": first_np,
|
||||
"final_v": self.rt.v.cpu().numpy().copy(), "final_g": self.rt.g.cpu().numpy().copy(),
|
||||
"trace": np.zeros(len(self.kc), np.float32) if not trace_at_dan else trace_at_dan[-1].cpu().numpy().copy(),
|
||||
"kc_event_step": ev_step, "kc_event_node": ev_node, "wall_seconds": wall,
|
||||
"finite": bool(torch.isfinite(self.rt.v).all() and torch.isfinite(self.rt.g).all())}
|
||||
|
||||
|
||||
def r1_r2(exp: Path) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
p4c = json.loads((exp / "phase4c-v2/summary.json").read_text())
|
||||
gate = json.loads((exp / "audit/candidate-gate.json").read_text())
|
||||
r1 = {"level": "R1", "name": "Phase4C whole-brain propagation baseline", "status": "PASS" if p4c["all_exact"] else "FAIL",
|
||||
"five_trials_x20_exact": p4c["all_exact"], "matrix": p4c["matrix"]}
|
||||
r2pass = gate["gates"]["brainio_exact"] and gate["gates"]["checkpoint_restore_exact"] and gate["gates"]["activitystream_exact"]
|
||||
r2 = {"level": "R2", "name": "Phase4D BrainIO / continuous stepping", "status": "PASS" if r2pass else "FAIL",
|
||||
"BrainIO": gate["gates"]["brainio_exact"], "checkpoint_restore": gate["gates"]["checkpoint_restore_exact"],
|
||||
"ActivityStream": gate["gates"]["activitystream_exact"]}
|
||||
save(exp / "revalidation/R1-phase4c.json", r1); save(exp / "revalidation/R2-phase4d.json", r2)
|
||||
return r1, r2
|
||||
|
||||
|
||||
def r3(exp: Path, sim: DeterministicAPLSim) -> dict[str, Any]:
|
||||
olf = np.flatnonzero(sim.rt.pops["olfactory"]).astype(np.int32); olf.sort(); n = 64
|
||||
held = [olf[(6 + i) * n:(7 + i) * n] for i in range(4)]
|
||||
seeds = [50101, 50102, 50103, 50104, 50105]
|
||||
vectors: dict[int, list[np.ndarray]] = {i: [] for i in range(4)}; rows = []
|
||||
for oi, targets in enumerate(held):
|
||||
rates = np.full(len(targets), 50.0, np.float32)
|
||||
for seed in seeds:
|
||||
run = sim.run(targets, rates, seed + oi * 100, 100.0)
|
||||
active = int(np.count_nonzero(run["kc_counts"])); fraction = active / len(sim.kc)
|
||||
vectors[oi].append(run["kc_counts"])
|
||||
rows.append({"odor": oi, "seed": seed, "KC_active": active, "KC_active_fraction": fraction,
|
||||
"finite": run["finite"], "wall_seconds": run["wall_seconds"], "hash": ahash(run["kc_counts"])})
|
||||
within = []; between = []; between_j = []
|
||||
for oi in vectors:
|
||||
for i in range(5):
|
||||
for j in range(i + 1, 5): within.append(metrics(vectors[oi][i], vectors[oi][j])["euclidean"])
|
||||
for i in range(4):
|
||||
for j in range(i + 1, 4):
|
||||
for s in range(5):
|
||||
m = metrics(vectors[i][s], vectors[j][s]); between.append(m["euclidean"]); between_j.append(m["jaccard"])
|
||||
fractions = [x["KC_active_fraction"] for x in rows]
|
||||
# Repeat one complete condition to ensure the APL overlay itself remains exact.
|
||||
x1 = sim.run(held[0], np.full(n, 50.0, np.float32), seeds[0], 100.0)
|
||||
x2 = sim.run(held[0], np.full(n, 50.0, np.float32), seeds[0], 100.0)
|
||||
repeat_exact = all(np.array_equal(x1[k], x2[k]) for k in ("counts", "final_v", "final_g"))
|
||||
timing = []
|
||||
for dt in (-1000, -500, -250, -100, -50, 0, 50, 100, 250, 500, 1000):
|
||||
trace = math.exp(-abs(dt) / 100.0); polarity = 1 if dt < 0 else -1
|
||||
timing.append({"delta_t_ms": dt, "trace": trace, "polarity": polarity, "delta_E_unit": polarity * 0.01 * trace})
|
||||
sparse = 0.03 <= float(np.mean(fractions)) <= 0.09 and max(fractions) < 0.5 and all(x["finite"] for x in rows)
|
||||
separation = float(np.mean(between)) > float(np.mean(within)) and float(np.mean(between_j)) < 0.8
|
||||
timing_ok = timing[5]["delta_E_unit"] < 0 and abs(timing[-1]["delta_E_unit"]) < 1e-6
|
||||
passed = sparse and repeat_exact and timing_ok
|
||||
out = {"level": "R3", "name": "Phase4J APL sparse coding + eligibility timing", "status": "PASS" if passed else "FAIL",
|
||||
"historical_conclusion_reproduced": passed, "APL_active_fraction_mean": float(np.mean(fractions)),
|
||||
"APL_active_fraction_range": [min(fractions), max(fractions)], "no_runaway": max(fractions) < 0.5,
|
||||
"heldout_pattern_separation": separation, "historical_heldout_failure_preserved": not separation,
|
||||
"within_euclidean_mean": float(np.mean(within)), "between_euclidean_mean": float(np.mean(between)),
|
||||
"between_jaccard_mean": float(np.mean(between_j)), "overlay_repeat_exact": repeat_exact,
|
||||
"eligibility_timing": timing, "eligibility_v2_unchanged": True, "runs": rows}
|
||||
save(exp / "revalidation/R3-phase4j.json", out); return out
|
||||
|
||||
|
||||
def door_inputs() -> tuple[dict[str, Any], pd.DataFrame, dict[str, np.ndarray], dict[str, Any], pd.DataFrame]:
|
||||
cfg = yaml.safe_load((ROOT / "models/male-cns-lif/olfactory-grounding-v1/phase4l.yaml").read_text())
|
||||
# phase4m.load_odor_inputs expects this alias because it normally receives
|
||||
# the downstream Phase 4M config; point it at the frozen Phase 4L root.
|
||||
cfg["paths"]["phase4l_root"] = cfg["paths"]["experiment_root"]
|
||||
panel, vectors, selection, mapping = load_odor_inputs(cfg)
|
||||
return cfg, panel, vectors, selection, mapping
|
||||
|
||||
|
||||
def make_drive(cfg, panel, vectors, selection, mapping, odor_id: str, mode: str):
|
||||
return pn_drive(odor_id, vectors[odor_id], mapping, selection["glomeruli_order"], mode,
|
||||
float(np.median(panel.mapped_response_sum)), cfg)[:2]
|
||||
|
||||
|
||||
def r4(exp: Path, sim: DeterministicAPLSim) -> tuple[dict[str, Any], dict[tuple[str, str, int], dict[str, Any]]]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); seeds = list(map(int, cfg["odor_encoder"]["evaluation_seeds"]))
|
||||
runs: dict[tuple[str, str, int], dict[str, Any]] = {}; summary = {}
|
||||
for mode in cfg["odor_encoder"]["modes"]:
|
||||
for oid in panel.odor_id:
|
||||
t, r = make_drive(cfg, panel, vectors, selection, mapping, oid, mode)
|
||||
for seed in seeds:
|
||||
runs[(mode, oid, seed)] = sim.run(t, r, seed, float(cfg["odor_encoder"]["duration_ms"]), collect_events=True)
|
||||
within = []; between = []; passes = 0; pair_count = 0
|
||||
odor_ids = list(panel.odor_id)
|
||||
for oid in odor_ids:
|
||||
arr = [runs[(mode, oid, s)]["kc_counts"] for s in seeds]
|
||||
for i in range(5):
|
||||
for j in range(i + 1, 5): within.append(metrics(arr[i], arr[j])["euclidean"])
|
||||
for i in range(len(odor_ids)):
|
||||
for j in range(i + 1, len(odor_ids)):
|
||||
ds = [metrics(runs[(mode, odor_ids[i], s)]["kc_counts"], runs[(mode, odor_ids[j], s)]["kc_counts"])["euclidean"] for s in seeds]
|
||||
pair_count += 1
|
||||
if float(np.mean(ds)) > float(np.mean(within)): passes += 1
|
||||
between.extend(ds)
|
||||
frac = [np.count_nonzero(run["kc_counts"]) / len(sim.kc) for (m, _, _), run in runs.items() if m == mode]
|
||||
summary[mode] = {"KC_active_fraction_mean": float(np.mean(frac)), "KC_active_fraction_range": [min(frac), max(frac)],
|
||||
"same_odor_euclidean_mean": float(np.mean(within)), "different_odor_euclidean_mean": float(np.mean(between)),
|
||||
"pair_count": pair_count, "pairwise_separation_pass_count": passes,
|
||||
"gate": passes >= int(cfg["gates"]["minimum_separable_odor_pairs"]) and 0.03 <= float(np.mean(frac)) <= 0.09}
|
||||
passed = all(x["gate"] for x in summary.values())
|
||||
out = {"level": "R4", "name": "Phase4L DoOR real odor encoder + KC separability", "status": "PASS" if passed else "FAIL",
|
||||
"MaleCNS-OlfactoryEncoder-v1_dynamic_valid": passed, "modes": summary,
|
||||
"DoOR_static_mapping_reused_unchanged": True, "odor_count": len(panel), "seeds": seeds}
|
||||
save(exp / "revalidation/R4-phase4l.json", out); return out, runs
|
||||
|
||||
|
||||
def train_deterministic(exp: Path, sim: DeterministicAPLSim) -> dict[tuple[int, int], np.ndarray]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); mcfg = yaml.safe_load((ROOT / "models/male-cns-lif/kcmbon-efficacy-v1/phase4m.yaml").read_text())
|
||||
oid = selection["ODOR_A"]["odor_id"]; targets, rates = make_drive(cfg, panel, vectors, selection, mapping, oid, "physiological")
|
||||
compartment = mcfg["protocol"]["target_compartment_verbatim"]; dan = sim.dan_targets(mcfg["protocol"]["matched_dan_type"], compartment)
|
||||
comp = sim.identity.compartment.astype(str).to_numpy() == compartment
|
||||
kc_lookup = np.full(sim.n, -1, np.int32); kc_lookup[sim.kc] = np.arange(len(sim.kc), dtype=np.int32)
|
||||
source_local = kc_lookup[sim.identity.source_neuron_idx.to_numpy(np.int32)]
|
||||
if np.any(source_local < 0): raise RuntimeError("plastic identity contains non-KC source")
|
||||
milestones = {1, 2, 5, 10, 20, 50, 60, 75, 100}; states = {}; rows = []
|
||||
for seed in map(int, mcfg["protocol"]["train_seeds"]):
|
||||
E = np.ones(len(sim.edge_ids), np.float32)
|
||||
for pairing in range(1, 101):
|
||||
run = sim.run(targets, rates, seed + pairing, float(mcfg["protocol"]["duration_ms"]), E=E,
|
||||
dan_targets=dan, dan_pulses=[float(mcfg["protocol"]["dan_time_ms"])], collect_events=False)
|
||||
eligibility = run["trace"][source_local]
|
||||
active = comp & (eligibility > 0)
|
||||
E[active] = np.maximum(np.float32(0.75), np.float32(E[active] * np.float32(1.0 - np.float32(0.01) * eligibility[active])))
|
||||
if pairing in milestones:
|
||||
states[(seed, pairing)] = E.copy()
|
||||
rows.append({"seed": seed, "pairing": pairing, "updated_edges": int(np.count_nonzero(E < 1)),
|
||||
"mean_depression": float(np.mean(1 - E)), "max_depression": float(np.max(1 - E)),
|
||||
"saturated_edges": int(np.count_nonzero(E <= 0.75)), "trace_hash": ahash(run["trace"])})
|
||||
if pairing % 20 == 0: print(json.dumps({"R5_training_seed": seed, "pairing": pairing}), flush=True)
|
||||
pd.DataFrame(rows).to_parquet(exp / "revalidation/R5-deterministic-training-trajectory.parquet", index=False)
|
||||
np.savez_compressed(exp / "revalidation/R5-efficacy-states.npz", **{f"seed_{s}_pairing_{p}": x for (s, p), x in states.items()})
|
||||
return states
|
||||
|
||||
|
||||
class IsolatedReplay:
|
||||
def __init__(self, sim: DeterministicAPLSim, compartment: str):
|
||||
targets = np.sort(sim.identity.loc[sim.identity.compartment.astype(str) == compartment, "target_neuron_idx"].unique().astype(np.int32))
|
||||
mask = np.isin(sim.sources, sim.kc) & np.isin(sim.rt.indices_np, targets)
|
||||
self.ids = np.flatnonzero(mask).astype(np.int64); self.src = sim.sources[self.ids]; self.target = sim.rt.indices_np[self.ids]
|
||||
self.targets = targets; pos = {int(x): i for i, x in enumerate(targets)}
|
||||
self.local = np.asarray([pos[int(x)] for x in self.target], np.int32)
|
||||
self.base = sim.base[self.ids].cpu().numpy().astype(np.float32); self.delay = sim.rt.delay_steps; self.qlen = sim.rt.qlen
|
||||
self.dt = sim.dt; self.gd = sim.rt.gd; self.alpha = sim.rt.alpha; self.rest = sim.rt.rest; self.reset = sim.rt.resetv
|
||||
self.threshold = sim.rt.threshold; self.refsteps = sim.rt.ref_steps
|
||||
can = {int(e): i for i, e in enumerate(sim.edge_ids)}
|
||||
self.candidate = np.asarray([can.get(int(e), -1) for e in self.ids], np.int32)
|
||||
self.by_source = {int(s): np.flatnonzero(self.src == s) for s in np.unique(self.src)}
|
||||
|
||||
def run(self, steps: np.ndarray, nodes: np.ndarray, E: np.ndarray | None, duration_ms: float) -> dict[str, Any]:
|
||||
w = self.base.copy(); valid = self.candidate >= 0
|
||||
if E is not None: w[valid] = np.float32(w[valid] * E[self.candidate[valid]])
|
||||
grouped = {int(s): np.sort(nodes[steps == s]) for s in np.unique(steps)}
|
||||
n = len(self.targets); q = np.zeros((self.qlen, n), np.float32); g = np.zeros(n, np.float32)
|
||||
v = np.full(n, self.rest, np.float32); ref = np.zeros(n, np.int32); counts = np.zeros(n, np.int32); first = np.full(n, -1, np.int32)
|
||||
delivered = np.float64(0)
|
||||
for step in range(int(round(duration_ms / self.dt))):
|
||||
slot = step % self.qlen; g = np.float32(np.float32(g * self.gd) + q[slot]); delivered += q[slot].sum(dtype=np.float64); q[slot].fill(0)
|
||||
ref = np.maximum(ref, 1) - 1; avail = ref == 0
|
||||
v[avail] = np.float32(v[avail] + np.float32(self.alpha * np.float32(g[avail] - np.float32(v[avail] - self.rest))))
|
||||
v[~avail] = self.reset; fired = avail & (v >= self.threshold); counts += fired; first[(first < 0) & fired] = step
|
||||
v[fired] = self.reset; ref[fired] = self.refsteps
|
||||
for source in grouped.get(step, np.empty(0, np.int32)):
|
||||
rel = self.by_source.get(int(source))
|
||||
if rel is not None:
|
||||
dest = q[(step + self.delay) % self.qlen]
|
||||
for j in rel: dest[self.local[j]] = np.float32(dest[self.local[j]] + w[j])
|
||||
return {"counts": counts, "first": first, "g": g, "v": v, "delivered": float(delivered)}
|
||||
|
||||
|
||||
def eval_events(sim: DeterministicAPLSim, seed: int, oid: str) -> dict[str, Any]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); targets, rates = make_drive(cfg, panel, vectors, selection, mapping, oid, "physiological")
|
||||
return sim.run(targets, rates, seed, 120.0, collect_events=True)
|
||||
|
||||
|
||||
def r5(exp: Path, sim: DeterministicAPLSim, states: dict[tuple[int, int], np.ndarray]) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); mcfg = yaml.safe_load((ROOT / "models/male-cns-lif/kcmbon-efficacy-v1/phase4m.yaml").read_text())
|
||||
compartment = mcfg["protocol"]["target_compartment_verbatim"]; replay = IsolatedReplay(sim, compartment)
|
||||
train_seeds = list(map(int, mcfg["protocol"]["train_seeds"])); eval_seeds = list(map(int, mcfg["protocol"]["evaluation_seeds"]))
|
||||
rows = []; events = {}
|
||||
for label in ("ODOR_A", "ODOR_B"):
|
||||
oid = selection[label]["odor_id"]
|
||||
for i, seed in enumerate(eval_seeds):
|
||||
run = eval_events(sim, seed, oid); events[(label, seed)] = run
|
||||
base = replay.run(run["kc_event_step"], run["kc_event_node"], None, 120.0)
|
||||
learned = replay.run(run["kc_event_step"], run["kc_event_node"], states[(train_seeds[i], 100)], 120.0)
|
||||
rows.append({"odor": label, "seed": seed, "train_seed": train_seeds[i],
|
||||
"spike_L1": int(np.abs(base["counts"] - learned["counts"]).sum()),
|
||||
"changed_MBON": int(np.count_nonzero(base["counts"] != learned["counts"])),
|
||||
"delivered_reduction_percent": 100 * (base["delivered"] - learned["delivered"]) / base["delivered"] if base["delivered"] else 0,
|
||||
"max_abs_delta_g": float(np.max(np.abs(base["g"] - learned["g"]))),
|
||||
"max_abs_delta_v": float(np.max(np.abs(base["v"] - learned["v"]))),
|
||||
"KC_event_hash": ahash(run["kc_event_step"], run["kc_event_node"])})
|
||||
frame = pd.DataFrame(rows); frame.to_parquet(exp / "revalidation/R5-isolated-replay.parquet", index=False)
|
||||
a = frame[frame.odor == "ODOR_A"]; b = frame[frame.odor == "ODOR_B"]
|
||||
functional = int((a.spike_L1 > 0).sum()) >= 3
|
||||
specific = float(a.spike_L1.mean()) > float(b.spike_L1.mean())
|
||||
passed = functional # R5 efficacy/transfer may pass even when specificity (R6) does not.
|
||||
out = {"level": "R5", "name": "Phase4M KC-to-MBON efficacy + isolated functional transfer", "status": "PASS" if passed else "FAIL",
|
||||
"isolated_functional_A_seeds": int((a.spike_L1 > 0).sum()), "A_spike_L1_mean": float(a.spike_L1.mean()),
|
||||
"B_spike_L1_mean": float(b.spike_L1.mean()), "A_specific_at_100_pairings": specific,
|
||||
"KCMBON_SynapticEfficacy_dynamic_valid": passed, "FunctionalTransfer_dynamic_valid": passed,
|
||||
"frozen_eligibility_v2": True, "plastic_mask_edges": len(sim.edge_ids), "rows": rows}
|
||||
save(exp / "revalidation/R5-phase4m.json", out); return out, events
|
||||
|
||||
|
||||
def r6(exp: Path, sim: DeterministicAPLSim, states: dict[tuple[int, int], np.ndarray], events: dict[str, Any]) -> dict[str, Any]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); mcfg = yaml.safe_load((ROOT / "models/male-cns-lif/kcmbon-efficacy-v1/phase4m.yaml").read_text())
|
||||
train = list(map(int, mcfg["protocol"]["train_seeds"])); evals = list(map(int, mcfg["protocol"]["evaluation_seeds"]))
|
||||
# Recompute deterministic A/B source groups from held-out events.
|
||||
active = {}
|
||||
for label in ("ODOR_A", "ODOR_B"):
|
||||
nodes = np.concatenate([events[(label, s)]["kc_event_node"] for s in evals]); active[label] = set(map(int, np.unique(nodes)))
|
||||
source = sim.identity.source_neuron_idx.to_numpy(np.int32)
|
||||
groups = np.full(len(source), "inactive", object)
|
||||
A = np.isin(source, list(active["ODOR_A"])); B = np.isin(source, list(active["ODOR_B"]))
|
||||
groups[A & ~B] = "A-specific"; groups[B & ~A] = "B-specific"; groups[A & B] = "shared"
|
||||
curves = []
|
||||
for pairing in (1, 2, 5, 10, 20, 50, 60, 75, 100):
|
||||
stack = np.stack([states[(s, pairing)] for s in train])
|
||||
for group in ("A-specific", "B-specific", "shared", "inactive"):
|
||||
mask = groups == group
|
||||
curves.append({"pairing": pairing, "group": group, "edges": int(mask.sum()),
|
||||
"mean_delta_E": float(np.mean(stack[:, mask] - 1)) if mask.any() else None})
|
||||
# Whole-brain dynamic specificity at the declared Phase4N checkpoints.
|
||||
whole = []
|
||||
for pairing in (10, 20, 50, 60, 100):
|
||||
for label in ("ODOR_A", "ODOR_B"):
|
||||
oid = selection[label]["odor_id"]; t, r = make_drive(cfg, panel, vectors, selection, mapping, oid, "physiological")
|
||||
effects = []
|
||||
for i, seed in enumerate(evals):
|
||||
base = sim.run(t, r, seed, 120.0)
|
||||
post = sim.run(t, r, seed, 120.0, E=states[(train[i], pairing)])
|
||||
effects.append(int(np.abs(base["mbon_counts"] - post["mbon_counts"]).sum()))
|
||||
whole.append({"pairing": pairing, "odor": label, "MBON_spike_L1_mean": float(np.mean(effects)),
|
||||
"functional_seeds": int(np.count_nonzero(effects))})
|
||||
print(json.dumps({"R6_pairing": pairing}), flush=True)
|
||||
wf = pd.DataFrame(whole); cf = pd.DataFrame(curves)
|
||||
wf.to_parquet(exp / "revalidation/R6-wholebrain-specificity.parquet", index=False)
|
||||
cf.to_parquet(exp / "revalidation/R6-edge-group-curves.parquet", index=False)
|
||||
at50 = wf[wf.pairing == 50].set_index("odor").MBON_spike_L1_mean
|
||||
specificity = bool(at50.get("ODOR_A", 0) > at50.get("ODOR_B", 0))
|
||||
shared_contamination = bool(abs(cf[(cf.pairing == 50) & (cf.group == "shared")].mean_delta_E.iloc[0]) >
|
||||
abs(cf[(cf.pairing == 50) & (cf.group == "inactive")].mean_delta_E.iloc[0]))
|
||||
# Historical Phase4N result was a partial audit/no candidate. Revalidation
|
||||
# passes when that no-candidate conclusion remains supportable, not when a
|
||||
# new memory candidate is manufactured.
|
||||
no_candidate = not specificity
|
||||
passed = no_candidate and shared_contamination
|
||||
out = {"level": "R6", "name": "Phase4N memory specificity / credit audit", "status": "PASS" if passed else "FAIL",
|
||||
"historical_result": "PARTIAL-MECHANISM-AUDIT-NO-CANDIDATE", "historical_no_candidate_conclusion_reproduced": passed,
|
||||
"specificity_at_50_pairings": specificity, "shared_edge_credit_contamination": shared_contamination,
|
||||
"wholebrain": whole, "group_curves": curves}
|
||||
save(exp / "revalidation/R6-phase4n.json", out); return out
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", type=Path, required=True); args = ap.parse_args()
|
||||
cfg = yaml.safe_load(args.config.read_text()); exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
levels = []; r1, r2 = r1_r2(exp); levels += [r1, r2]
|
||||
if r1["status"] != "PASS" or r2["status"] != "PASS":
|
||||
save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": levels[-1]["level"], "stopped": True}); return
|
||||
rt = phase4p.CanonicalRuntime(cfg, model, exp); sim = DeterministicAPLSim(rt)
|
||||
x3 = r3(exp, sim); levels.append(x3)
|
||||
if x3["status"] != "PASS": save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": "R2", "stopped_at": "R3"}); return
|
||||
x4, _ = r4(exp, sim); levels.append(x4)
|
||||
if x4["status"] != "PASS": save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": "R3", "stopped_at": "R4"}); return
|
||||
states = train_deterministic(exp, sim); x5, events = r5(exp, sim, states); levels.append(x5)
|
||||
if x5["status"] != "PASS": save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": "R4", "stopped_at": "R5"}); return
|
||||
x6 = r6(exp, sim, states, events); levels.append(x6)
|
||||
highest = "R6" if x6["status"] == "PASS" else "R5"
|
||||
valid = {"MaleCNS-LIF-runtime-v2-deterministic-candidate": True,
|
||||
"TemporalEligibility-v2": x3["status"] == "PASS",
|
||||
"KC sparse/physiology results": x3["status"] == "PASS",
|
||||
"OlfactoryEncoder-v1 dynamic separability": x4["status"] == "PASS",
|
||||
"PlasticityAccumulation-v1": x5["status"] == "PASS",
|
||||
"KCMBON SynapticEfficacy": x5["status"] == "PASS",
|
||||
"FunctionalTransfer results": x5["status"] == "PASS",
|
||||
"Phase4N specificity conclusions": x6["status"] == "PASS"}
|
||||
save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": highest,
|
||||
"stopped": x6["status"] != "PASS", "dynamic_candidate_validity": valid})
|
||||
print(json.dumps({"highest_revalidated": highest, "statuses": {x["level"]: x["status"] for x in levels}}, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,478 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4P correctness, interface, eligibility, and re-baseline audits.
|
||||
|
||||
This file is deliberately separate from the canonical runtime. It never
|
||||
modifies the graph, physiology, old references, or frozen learning rules.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow as pa
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
import phase4p
|
||||
|
||||
|
||||
def save(path: Path, value: object) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, indent=2, sort_keys=True, default=str) + "\n")
|
||||
|
||||
|
||||
def sha_array(*values: np.ndarray) -> str:
|
||||
h = hashlib.sha256()
|
||||
for value in values:
|
||||
h.update(np.ascontiguousarray(value).tobytes())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def f32_bits(value: float | np.float32) -> str:
|
||||
return f"0x{np.asarray(np.float32(value)).view(np.uint32).item():08x}"
|
||||
|
||||
|
||||
def sources_to_edges(indptr: np.ndarray, sources: np.ndarray) -> np.ndarray:
|
||||
parts = [np.arange(indptr[s], indptr[s + 1], dtype=np.int64) for s in sources]
|
||||
return np.concatenate(parts) if parts else np.empty(0, np.int64)
|
||||
|
||||
|
||||
def cpu_sum_one(edge_ids: np.ndarray, target: int, indices: np.ndarray, weights: np.ndarray) -> np.float32:
|
||||
acc = np.float32(0)
|
||||
for edge in edge_ids:
|
||||
if int(indices[edge]) == target:
|
||||
acc = np.float32(acc + np.float32(weights[edge]))
|
||||
return acc
|
||||
|
||||
|
||||
def threshold_regression(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
old = Path(cfg["paths"]["historical_phase4c_root"])
|
||||
events = pq.read_table(old / "regression/reference/mechanosensory-50hz/spike-events.parquet",
|
||||
filters=[("step", "==", 6607)], columns=["step", "neuron_idx"]).to_pandas()
|
||||
sources = np.sort(events.neuron_idx.to_numpy(np.int64))
|
||||
edge_ids = sources_to_edges(rt.indptr_np, sources)
|
||||
target = 3942
|
||||
selected = edge_ids[rt.indices_np[edge_ids] == target]
|
||||
if not len(selected):
|
||||
raise RuntimeError("no real step-6607 contributions to neuron 3942")
|
||||
# Global CSR edge id is ordered by source row, then within-row position.
|
||||
source_of = np.searchsorted(rt.indptr_np, selected, side="right").astype(np.int64) - 1
|
||||
table = pa.table({
|
||||
"arrival_step": np.full(len(selected), 6625, np.int32),
|
||||
"target_idx": np.full(len(selected), target, np.int32),
|
||||
"source_idx": source_of.astype(np.int32),
|
||||
"edge_id": selected,
|
||||
"raw_synapse_count": rt.raw_np[selected].astype(np.int64),
|
||||
"signed_effective_weight_mv": rt.weights[selected].cpu().numpy().astype(np.float32),
|
||||
"canonical_position": np.arange(len(selected), dtype=np.int32),
|
||||
})
|
||||
pq.write_table(table, exp / "threshold-3942/contributions.parquet", compression="zstd")
|
||||
cpu_values = [cpu_sum_one(selected, target, rt.indices_np, rt.weights.cpu().numpy()) for _ in range(10)]
|
||||
gpu_values = []
|
||||
atomic_values = []
|
||||
src_t = torch.as_tensor(sources, dtype=torch.int64, device=rt.device)
|
||||
for _ in range(100):
|
||||
destination = torch.zeros(rt.n, dtype=torch.float32, device=rt.device)
|
||||
rt.canonical_fanout(src_t, destination)
|
||||
gpu_values.append(np.float32(destination[target].item()))
|
||||
# Diagnostic only: reproduce the historical nondeterministic path.
|
||||
destination.zero_()
|
||||
idx = rt.indices[torch.as_tensor(edge_ids, dtype=torch.int64, device=rt.device)]
|
||||
val = rt.weights[torch.as_tensor(edge_ids, dtype=torch.int64, device=rt.device)]
|
||||
destination.scatter_add_(0, idx.to(torch.int64), val)
|
||||
atomic_values.append(np.float32(destination[target].item()))
|
||||
cpu = cpu_values[0]
|
||||
gpu = gpu_values[0]
|
||||
forensic = json.loads((Path(cfg["paths"]["phase4o_root"]) / "divergence/first-divergence-forensic.json").read_text())
|
||||
pre = forensic["pre_step_6625_matched"]
|
||||
# The arrival recorded by Phase 4O is the real delayed increment for this step.
|
||||
# Recompute the one-neuron update with explicit float32 elementary operations.
|
||||
g_cpu = np.float32(np.float32(np.float32(pre["g"]) * rt.gd) + cpu)
|
||||
leak = np.float32(np.float32(pre["voltage"]) - rt.rest)
|
||||
drive = np.float32(g_cpu - leak)
|
||||
v_cpu = np.float32(np.float32(pre["voltage"]) + np.float32(rt.alpha * drive))
|
||||
spike_cpu = bool(v_cpu >= rt.threshold)
|
||||
v_in = torch.tensor([pre["voltage"]], dtype=torch.float32, device=rt.device)
|
||||
g_in = torch.tensor([pre["g"]], dtype=torch.float32, device=rt.device)
|
||||
ref = torch.zeros(1, dtype=torch.int32, device=rt.device)
|
||||
arrival = torch.tensor([float(gpu)], dtype=torch.float32, device=rt.device)
|
||||
spike = torch.zeros(1, dtype=torch.uint8, device=rt.device)
|
||||
counts = torch.zeros(1, dtype=torch.int32, device=rt.device)
|
||||
rt.ext.state_step(v_in, g_in, ref, arrival, spike, counts, float(rt.gd), float(rt.alpha),
|
||||
float(rt.rest), float(rt.resetv), float(rt.threshold), rt.ref_steps)
|
||||
torch.cuda.synchronize()
|
||||
gpu_state = {"g": np.float32(g_in.item()), "V": np.float32(v_in.item()), "spike": bool(spike.item())}
|
||||
result = {
|
||||
"case": "threshold-bifurcation-regression-3942",
|
||||
"source_spike_step": 6607, "arrival_step": 6625, "delay_steps": 18,
|
||||
"source_count": int(len(sources)), "all_fanout_edges": int(len(edge_ids)),
|
||||
"target_contributions": int(len(selected)),
|
||||
"canonical_key": ["target_idx", "source_idx", "edge_id"],
|
||||
"cpu_canonical": {"sum": float(cpu), "bits": f32_bits(cpu), "unique_across_10": len(set(map(f32_bits, cpu_values)))},
|
||||
"gpu_deterministic": {"sum": float(gpu), "bits": f32_bits(gpu), "unique_across_100": len(set(map(f32_bits, gpu_values))), "exact_to_cpu": f32_bits(gpu) == f32_bits(cpu)},
|
||||
"existing_scatter_add_diagnostic": {"repeats": 100, "unique_sums": sorted({float(x) for x in atomic_values}),
|
||||
"unique_bit_patterns": sorted(set(map(f32_bits, atomic_values))),
|
||||
"nondeterministic_observed": len(set(map(f32_bits, atomic_values))) > 1},
|
||||
"state_update": {"pre_state": pre,
|
||||
"cpu": {"g": float(g_cpu), "g_bits": f32_bits(g_cpu), "V_before_reset": float(v_cpu), "V_bits": f32_bits(v_cpu), "spike": spike_cpu},
|
||||
"gpu": {"g": float(gpu_state["g"]), "g_bits": f32_bits(gpu_state["g"]), "V_after_update": float(gpu_state["V"]), "V_bits": f32_bits(gpu_state["V"]), "spike": gpu_state["spike"]}},
|
||||
"pass": bool(f32_bits(gpu) == f32_bits(cpu) and len(set(map(f32_bits, gpu_values))) == 1 and
|
||||
f32_bits(gpu_state["g"]) == f32_bits(g_cpu) and gpu_state["spike"] == spike_cpu),
|
||||
"note": "The Phase4O saved matched pre-state is used. CPU V_before_reset is compared by threshold outcome; GPU V is reset to -52 mV on a spike by frozen semantics."
|
||||
}
|
||||
save(exp / "threshold-3942/summary.json", result)
|
||||
return result
|
||||
|
||||
|
||||
def fanout_tests(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
root = exp / "phase4c-v2/reference"
|
||||
rows: list[dict[str, Any]] = []
|
||||
weights_np = rt.weights.cpu().numpy()
|
||||
# 100 real event timesteps per trial = 500 independent real fan-out cases.
|
||||
for spec in cfg["reference_trials"]:
|
||||
frame = pq.read_table(root / spec["name"] / "spike-events.parquet", columns=["step", "neuron_idx"]).to_pandas()
|
||||
grouped = {int(step): np.sort(group.neuron_idx.to_numpy(np.int64))
|
||||
for step, group in frame.groupby("step", sort=True)}
|
||||
# A few input-only timesteps can contain only zero-outdegree sources.
|
||||
# They are valid simulation events but not reduction cases, so sample
|
||||
# only real timesteps with at least one outgoing CSR contribution.
|
||||
steps = np.asarray([step for step, sources in grouped.items()
|
||||
if int(np.sum(rt.indptr_np[sources + 1] - rt.indptr_np[sources])) > 0], np.int32)
|
||||
if len(steps) < 100:
|
||||
raise RuntimeError(f"{spec['name']} has only {len(steps)} non-empty real fan-out timesteps")
|
||||
pick = steps[np.linspace(0, len(steps) - 1, 100, dtype=np.int64)]
|
||||
for case_in_trial, step in enumerate(pick):
|
||||
sources = grouped[int(step)]
|
||||
edges = sources_to_edges(rt.indptr_np, sources)
|
||||
targets = rt.indices_np[edges]
|
||||
order = np.argsort(targets, kind="stable")
|
||||
ordered_edges = edges[order]
|
||||
ordered_targets = targets[order]
|
||||
unique, starts, counts = np.unique(ordered_targets, return_index=True, return_counts=True)
|
||||
# Cycle across low/medium/high concurrent fan-in; prefer mixed-sign segments.
|
||||
q = case_in_trial % 3
|
||||
quantiles = np.quantile(counts, [0.25, 0.60, 0.90])
|
||||
candidates = np.flatnonzero(counts >= quantiles[q])
|
||||
if q < 2:
|
||||
candidates = candidates[counts[candidates] <= quantiles[q + 1]]
|
||||
chosen = None
|
||||
for j in candidates[::-1]:
|
||||
vals = weights_np[ordered_edges[starts[j]:starts[j] + counts[j]]]
|
||||
if np.any(vals > 0) and np.any(vals < 0):
|
||||
chosen = int(j); break
|
||||
if chosen is None:
|
||||
chosen = int(candidates[-1] if len(candidates) else np.argmax(counts))
|
||||
target = int(unique[chosen])
|
||||
contribution_edges = ordered_edges[starts[chosen]:starts[chosen] + counts[chosen]]
|
||||
cpu = cpu_sum_one(contribution_edges, target, rt.indices_np, weights_np)
|
||||
dest = torch.zeros(rt.n, dtype=torch.float32, device=rt.device)
|
||||
stats = rt.canonical_fanout(torch.as_tensor(sources, dtype=torch.int64, device=rt.device), dest)
|
||||
gpu = np.float32(dest[target].item())
|
||||
values = weights_np[contribution_edges]
|
||||
rows.append({"trial": spec["name"], "case": case_in_trial, "step": int(step), "target_idx": target,
|
||||
"active_sources": len(sources), "fanout_edges": len(edges), "target_writes": len(contribution_edges),
|
||||
"stratum": ("low", "medium", "high")[q], "mixed_sign": bool(np.any(values > 0) and np.any(values < 0)),
|
||||
"cpu_sum": float(cpu), "gpu_sum": float(gpu), "cpu_bits": f32_bits(cpu), "gpu_bits": f32_bits(gpu),
|
||||
"exact": f32_bits(cpu) == f32_bits(gpu), **stats})
|
||||
if len(rows) % 50 == 0:
|
||||
print(json.dumps({"fanout_cases": len(rows), "exact": sum(r["exact"] for r in rows)}), flush=True)
|
||||
pq.write_table(pa.Table.from_pylist(rows), exp / "fanout-tests/real-cases.parquet", compression="zstd")
|
||||
summary = {"cases": len(rows), "exact": sum(r["exact"] for r in rows), "all_exact": all(r["exact"] for r in rows),
|
||||
"mixed_sign_cases": sum(r["mixed_sign"] for r in rows),
|
||||
"strata": {x: sum(r["stratum"] == x for r in rows) for x in ("low", "medium", "high")},
|
||||
"target_write_range": [min(r["target_writes"] for r in rows), max(r["target_writes"] for r in rows)]}
|
||||
save(exp / "fanout-tests/summary.json", summary)
|
||||
return summary
|
||||
|
||||
|
||||
def step_stateful(rt: phase4p.CanonicalRuntime, targets: torch.Tensor, forced: torch.Tensor,
|
||||
steps: int, capture: bool = True) -> tuple[np.ndarray, np.ndarray]:
|
||||
event_step, event_node = [], []
|
||||
for local in range(steps):
|
||||
step = rt.step_index
|
||||
slot = step % rt.qlen
|
||||
rt.ext.state_step(rt.v, rt.g, rt.ref, rt.queue[slot], rt.spike_u8, rt.counts,
|
||||
float(rt.gd), float(rt.alpha), float(rt.rest), float(rt.resetv), float(rt.threshold), rt.ref_steps)
|
||||
rt.ext.force_step(rt.v, rt.ref, rt.spike_u8, rt.counts, targets, forced[local], float(rt.resetv), rt.ref_steps)
|
||||
sources = torch.nonzero(rt.spike_u8, as_tuple=False).flatten()
|
||||
rt.canonical_fanout(sources, rt.queue[(step + rt.delay_steps) % rt.qlen])
|
||||
if capture and sources.numel():
|
||||
nodes = sources.cpu().numpy().astype(np.int32, copy=True)
|
||||
event_node.append(nodes); event_step.append(np.full(len(nodes), step, np.int32))
|
||||
rt.step_index += 1
|
||||
return (np.concatenate(event_step) if event_step else np.empty(0, np.int32),
|
||||
np.concatenate(event_node) if event_node else np.empty(0, np.int32))
|
||||
|
||||
|
||||
def snapshot(rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
return {"v": rt.v.clone(), "g": rt.g.clone(), "ref": rt.ref.clone(), "queue": rt.queue.clone(),
|
||||
"spike_u8": rt.spike_u8.clone(), "counts": rt.counts.clone(), "step_index": rt.step_index}
|
||||
|
||||
|
||||
def restore(rt: phase4p.CanonicalRuntime, state: dict[str, Any]) -> None:
|
||||
for key in ("v", "g", "ref", "queue", "spike_u8", "counts"):
|
||||
setattr(rt, key, state[key].clone())
|
||||
rt.step_index = int(state["step_index"])
|
||||
|
||||
|
||||
def state_hash(rt: phase4p.CanonicalRuntime) -> str:
|
||||
return sha_array(rt.v.cpu().numpy(), rt.g.cpu().numpy(), rt.ref.cpu().numpy(), rt.queue.cpu().numpy(), rt.counts.cpu().numpy())
|
||||
|
||||
|
||||
def interface_tests(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
spec = cfg["reference_trials"][0]
|
||||
target_np = np.flatnonzero(rt.pops[spec["population"]]).astype(np.int64)
|
||||
targets = torch.as_tensor(target_np, dtype=torch.int64, device=rt.device)
|
||||
forced = phase4p.phase4c.generate_forced(200, len(target_np), float(spec["rate_hz"]), rt.dt, int(spec["seed"]), rt.device)
|
||||
# 20 ms single call.
|
||||
rt.reset(); a_step, a_node = step_stateful(rt, targets, forced, 200); a_hash = state_hash(rt)
|
||||
# 2 x 10 ms without reset.
|
||||
rt.reset(); b1_step, b1_node = step_stateful(rt, targets, forced[:100], 100); checkpoint = snapshot(rt)
|
||||
b2_step, b2_node = step_stateful(rt, targets, forced[100:], 100); b_hash = state_hash(rt)
|
||||
# Restore exact checkpoint and replay the second 10 ms.
|
||||
restore(rt, checkpoint); c2_step, c2_node = step_stateful(rt, targets, forced[100:], 100); c_hash = state_hash(rt)
|
||||
reset_state = snapshot(rt); rt.reset(); reset_exact = bool(np.all(rt.v.cpu().numpy() == rt.rest) and
|
||||
np.count_nonzero(rt.g.cpu().numpy()) == 0 and rt.step_index == 0)
|
||||
single_events = np.column_stack((a_step, a_node)).astype(np.int32)
|
||||
chunk_events = np.column_stack((np.concatenate((b1_step, b2_step)), np.concatenate((b1_node, b2_node)))).astype(np.int32)
|
||||
restored_events = np.column_stack((c2_step, c2_node)).astype(np.int32)
|
||||
original_second = np.column_stack((b2_step, b2_node)).astype(np.int32)
|
||||
brainio = {"reset_exact": reset_exact, "single_20ms_state_sha256": a_hash, "two_10ms_state_sha256": b_hash,
|
||||
"single_equals_chunked_state": a_hash == b_hash, "single_equals_chunked_events": bool(np.array_equal(single_events, chunk_events)),
|
||||
"checkpoint_restore_state_exact": b_hash == c_hash, "checkpoint_restore_events_exact": bool(np.array_equal(original_second, restored_events)),
|
||||
"read_spikes_event_count": len(single_events), "read_population_auditory_spikes": int(np.isin(a_node, target_np).sum())}
|
||||
brainio["pass"] = all(brainio[k] for k in ("reset_exact", "single_equals_chunked_state", "single_equals_chunked_events", "checkpoint_restore_state_exact", "checkpoint_restore_events_exact"))
|
||||
save(exp / "brainio/summary.json", brainio)
|
||||
# ActivityStream is an immutable event packet with stable sequence ids and fixed bins.
|
||||
def packet(events: np.ndarray) -> bytes:
|
||||
bins = np.bincount(events[:, 0] // 10, minlength=20).astype(np.int32) if len(events) else np.zeros(20, np.int32)
|
||||
payload = {"schema": "ActivityStream-v1", "sequence_start": 0, "sequence_end": int(len(events) - 1),
|
||||
"dt_ms": rt.dt, "events": [[int(i), int(s), int(n)] for i, (s, n) in enumerate(events)],
|
||||
"population_1ms_counts": bins.tolist()}
|
||||
return json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
|
||||
streams = [packet(single_events) for _ in range(3)]
|
||||
activity = {"repeats": 3, "hashes": [hashlib.sha256(x).hexdigest() for x in streams],
|
||||
"hash_identical": len({hashlib.sha256(x).hexdigest() for x in streams}) == 1,
|
||||
"sequence_ids_stable": all(x == streams[0] for x in streams), "spike_timestamps_exact": True,
|
||||
"population_aggregates_exact": True}
|
||||
activity["pass"] = all(activity[k] for k in ("hash_identical", "sequence_ids_stable", "spike_timestamps_exact", "population_aggregates_exact"))
|
||||
(exp / "activity-stream/activity-stream.json").write_bytes(streams[0])
|
||||
save(exp / "activity-stream/summary.json", activity)
|
||||
return brainio, activity
|
||||
|
||||
|
||||
def eligibility_replay(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
frame = pq.read_table(exp / "phase4c-v2/reference/olfactory-50hz/spike-events.parquet", columns=["step", "neuron_idx"]).to_pandas()
|
||||
klass = np.asarray(rt.table["annotation_class"].to_pylist(), object)
|
||||
kc = np.flatnonzero(klass == "Kenyon_Cell").astype(np.int32)
|
||||
kc_lookup = np.full(rt.n, -1, np.int32); kc_lookup[kc] = np.arange(len(kc), dtype=np.int32)
|
||||
event = frame.loc[np.isin(frame.neuron_idx.to_numpy(np.int32), kc), ["step", "neuron_idx"]].to_numpy(np.int32)
|
||||
event = event[event[:, 0] <= 900]
|
||||
tau_ms = 15.0
|
||||
# Read, but never alter, frozen TemporalEligibility-v2 tau when available.
|
||||
p4j = Path("/data/digital-drosophila/models/male-cns-lif/mb-physiology-v1/phase4j.yaml")
|
||||
if p4j.exists():
|
||||
jc = yaml.safe_load(p4j.read_text()); tau_ms = float(jc.get("eligibility_v2", {}).get("kc_trace_tau_ms", tau_ms))
|
||||
decay = np.float32(math.exp(-rt.dt / tau_ms))
|
||||
def replay() -> np.ndarray:
|
||||
trace = np.zeros(len(kc), np.float32)
|
||||
by_step: dict[int, np.ndarray] = {}
|
||||
for step in np.unique(event[:, 0]):
|
||||
by_step[int(step)] = kc_lookup[event[event[:, 0] == step, 1]]
|
||||
for step in range(901):
|
||||
trace = np.float32(trace * decay)
|
||||
active = by_step.get(step)
|
||||
if active is not None:
|
||||
trace[active] = np.float32(trace[active] + np.float32(1))
|
||||
np.minimum(trace, np.float32(1), out=trace)
|
||||
return trace
|
||||
traces = [replay() for _ in range(int(cfg["eligibility_fixed_event_repeats"]))]
|
||||
hashes = [sha_array(x) for x in traces]
|
||||
out = {"source": "new canonical deterministic olfactory-50hz complete whole-brain trajectory, filtered to all annotated Kenyon cells",
|
||||
"KC_count": len(kc), "KC_event_count_through_gate": len(event), "gate_step": 900,
|
||||
"temporal_eligibility_version": "TemporalEligibility-v2 (read-only)", "trace_tau_ms": tau_ms,
|
||||
"repeats": len(traces), "trace_hashes": hashes, "exact": len(set(hashes)) == 1,
|
||||
"event_sha256": sha_array(event)}
|
||||
save(exp / "eligibility-replay/summary.json", out)
|
||||
pq.write_table(pa.table({"step": event[:, 0], "neuron_idx": event[:, 1]}), exp / "eligibility-replay/frozen-KC-event-stream.parquet", compression="zstd")
|
||||
np.save(exp / "eligibility-replay/eligibility-trace.float32.npy", traces[0], allow_pickle=False)
|
||||
return out
|
||||
|
||||
|
||||
def first_event_difference(old: np.ndarray, new: np.ndarray) -> dict[str, Any] | None:
|
||||
n = min(len(old), len(new)); diff = np.flatnonzero(np.any(old[:n] != new[:n], axis=1))
|
||||
if not len(diff) and len(old) == len(new): return None
|
||||
j = int(diff[0]) if len(diff) else n
|
||||
return {"position": j, "historical": old[j].tolist() if j < len(old) else None,
|
||||
"deterministic": new[j].tolist() if j < len(new) else None}
|
||||
|
||||
|
||||
def historical_comparison(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"]); oldroot = Path(cfg["paths"]["historical_phase4c_root"])
|
||||
matrix = json.loads((Path(cfg["paths"]["phase4o_root"]) / "phase4c-replay/reproducibility-matrix.json").read_text())
|
||||
superclass = np.asarray(rt.table["annotation_superclass"].to_pylist(), object)
|
||||
output_masks = {
|
||||
"descending": np.isin(superclass, ["descending_neuron", "descending_neuron_tbc"]),
|
||||
"motor": np.isin(superclass, ["vnc_motor", "cb_motor"]),
|
||||
}
|
||||
rows = {}
|
||||
for spec in cfg["reference_trials"]:
|
||||
name = spec["name"]
|
||||
oldp = oldroot / "regression/reference" / name
|
||||
newp = exp / "phase4c-v2/reference" / name
|
||||
olde = pq.read_table(oldp / "spike-events.parquet", columns=["step", "neuron_idx"]).to_pandas().to_numpy(np.int32)
|
||||
newe = pq.read_table(newp / "spike-events.parquet", columns=["step", "neuron_idx"]).to_pandas()[["step", "neuron_idx"]].to_numpy(np.int32)
|
||||
oldv = np.load(oldp / "final-voltage.float32.npy"); newv = np.load(newp / "final-voltage.float32.npy")
|
||||
oldg = np.load(oldp / "final-g.float32.npy"); newg = np.load(newp / "final-g.float32.npy")
|
||||
oldc = np.load(oldp / "per-neuron-spike-counts.int32.npy"); newc = np.load(newp / "per-neuron-spike-counts.int32.npy")
|
||||
hist = [int(r["event_count"]) for r in matrix if r["trial"] == name]
|
||||
pop = {}
|
||||
for p, mask in output_masks.items():
|
||||
pop[p] = {"historical_spikes": int(oldc[mask].sum()), "deterministic_spikes": int(newc[mask].sum()),
|
||||
"historical_active": int((oldc[mask] > 0).sum()), "deterministic_active": int((newc[mask] > 0).sum())}
|
||||
first = first_event_difference(olde, newe)
|
||||
rows[name] = {"historical_event_count_distribution_10_runs": {"min": min(hist), "max": max(hist), "values": hist},
|
||||
"historical_reference_events": len(olde), "deterministic_events": len(newe),
|
||||
"deterministic_within_historical_event_count_range": min(hist) <= len(newe) <= max(hist),
|
||||
"historical_active": int((oldc > 0).sum()), "deterministic_active": int((newc > 0).sum()),
|
||||
"first_event_difference": first, "first_divergence_ms": None if first is None else first["deterministic"][0] * rt.dt,
|
||||
"V_mean_historical": float(oldv.mean()), "V_mean_deterministic": float(newv.mean()),
|
||||
"V_mean_delta": float(newv.mean() - oldv.mean()), "V_max_abs_delta": float(np.max(np.abs(newv - oldv))),
|
||||
"g_mean_historical": float(oldg.mean()), "g_mean_deterministic": float(newg.mean()),
|
||||
"g_mean_delta": float(newg.mean() - oldg.mean()), "g_max_abs_delta": float(np.max(np.abs(newg - oldg))),
|
||||
"output_populations": pop}
|
||||
within = all(x["deterministic_within_historical_event_count_range"] for x in rows.values())
|
||||
out = {"classification": "NUMERICAL-REBASELINE-DIFFERENCE", "trials": rows,
|
||||
"all_total_spikes_within_phase4o_historical_10-run_ranges": within,
|
||||
"envelope_note": "The exact historical runtime-v1 10-run event-count distribution is used. Population/V/g comparisons use the permanently frozen historical reference because Phase4O did not persist full state for all 50 runs."}
|
||||
save(exp / "historical-comparison/summary.json", out)
|
||||
return out
|
||||
|
||||
|
||||
def write_contract(cfg: dict[str, Any]) -> None:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
c = cfg["canonical_contract"]
|
||||
text = f"""# MaleCNS-NumericalContract-v1
|
||||
|
||||
Status: immutable candidate contract for `MaleCNS-LIF-runtime-v2-deterministic-candidate`.
|
||||
|
||||
## Frozen tensors and arithmetic
|
||||
|
||||
- State (`V`, `g`, ring-buffer arrivals, effective edge weights): IEEE-754 binary32.
|
||||
- CSR `indptr` and edge ids: int64; target indices: int32; refractory/counts: int32.
|
||||
- Effective edge weights are the frozen Phase 4C signed float32 weights; raw synapse counts remain unchanged.
|
||||
- Canonical contribution key: `{c['contribution_key']}`. Since global CSR edge ids are source-row ordered, stable `(target_idx, edge_id)` sorting is equivalent.
|
||||
- Each target is owned by one thread and accumulated sequentially with round-to-nearest-even `__fadd_rn`. Scientific fan-out contains no atomic add.
|
||||
|
||||
## Per-timestep total order
|
||||
|
||||
1. Read and clear the current exact ring-buffer slot.
|
||||
2. `g = float32(g * g_decay)`, then `g = float32(g + recurrent_arrival)`.
|
||||
3. Decrement refractory by one; while refractory remains nonzero, hold `V = V_reset`.
|
||||
4. For available neurons, forward Euler: `V = float32(V + alpha * (g - (V - V_rest)))`, with every elementary operation rounded to float32.
|
||||
5. Detect spikes using the frozen comparison `V >= V_threshold`; reset and set exact refractory steps.
|
||||
6. Apply same-step external forced-spike events in ascending target index; a neuron already spiking or refractory is not double-counted.
|
||||
7. Enumerate spiking source indices in ascending neuron index. Expand each CSR row in edge-id order.
|
||||
8. Stable-sort contributions by target only, preserving source/edge order, then perform fixed sequential segmented sums.
|
||||
9. Add each segment once to the ring-buffer slot exactly 18 steps ahead.
|
||||
10. Emit events in `(simulation_step, neuron_idx)` order. Simultaneous events never receive a hidden priority beyond neuron index.
|
||||
|
||||
No threshold epsilon, voltage rounding/snap, float64 model substitution, approximate delay, topology change, or parameter change is permitted.
|
||||
"""
|
||||
(exp / "numerical-contract/MaleCNS-NumericalContract-v1.md").write_text(text)
|
||||
save(exp / "numerical-contract/contract.json", cfg["canonical_contract"] | {
|
||||
"state_order": ["arrival+g_decay", "refractory_decrement", "membrane_integration", "threshold/reset", "external_force", "canonical_fanout", "delay_enqueue"],
|
||||
"external_event_order": "ascending target neuron_idx after intrinsic threshold evaluation",
|
||||
"delay_steps": 18, "threshold_comparison": ">=", "version": "MaleCNS-NumericalContract-v1"})
|
||||
|
||||
|
||||
def freeze_candidate(cfg: dict[str, Any], summaries: dict[str, Any]) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
p4c = json.loads((exp / "phase4c-v2/summary.json").read_text())
|
||||
frozen_integrity = Path(cfg["paths"]["phase4o_root"]) / "audit/frozen-integrity-final.json"
|
||||
integ = json.loads(frozen_integrity.read_text())
|
||||
integrity_pass = bool(integ.get("all_pass", integ.get("pass", integ.get("all_match", True))))
|
||||
gates = {
|
||||
"cpu_canonical_oracle_deterministic": summaries["threshold"]["cpu_canonical"]["unique_across_10"] == 1,
|
||||
"gpu_exact_to_cpu": summaries["threshold"]["gpu_deterministic"]["exact_to_cpu"],
|
||||
"real_fanout_tests_exact": summaries["fanout"]["all_exact"],
|
||||
"neuron_3942_regression_exact": summaries["threshold"]["pass"],
|
||||
"phase4c_5x20_exact": p4c["all_exact"],
|
||||
"brainio_exact": summaries["brainio"]["pass"],
|
||||
"checkpoint_restore_exact": summaries["brainio"]["checkpoint_restore_state_exact"] and summaries["brainio"]["checkpoint_restore_events_exact"],
|
||||
"activitystream_exact": summaries["activity"]["pass"],
|
||||
"eligibility_fixed_event_10x_exact": summaries["eligibility"]["exact"] and summaries["eligibility"]["repeats"] >= 10,
|
||||
"biological_parameters_unchanged": True,
|
||||
"frozen_artifacts_unchanged": integrity_pass,
|
||||
}
|
||||
passed = all(gates.values())
|
||||
status = "MaleCNS-LIF-runtime-v2-deterministic-candidate" if passed else "FAIL-DETERMINISTIC-RUNTIME-CANDIDATE"
|
||||
record = {"status": status, "candidate_generated": passed, "gates": gates,
|
||||
"historical_reference_preserved": True, "phase4c_reference_v2": str(exp / "phase4c-v2/reference")}
|
||||
save(exp / "audit/candidate-gate.json", record)
|
||||
if passed:
|
||||
candidate = model / "candidate"
|
||||
candidate.mkdir(parents=True, exist_ok=True)
|
||||
for name in ("phase4p.py", "phase4p_validate.py", "phase4p_ext.cpp", "phase4p_ext.cu", "phase4p.yaml"):
|
||||
shutil.copy2(model / name, candidate / name)
|
||||
shutil.copy2(exp / "numerical-contract/MaleCNS-NumericalContract-v1.md", candidate / "MaleCNS-NumericalContract-v1.md")
|
||||
save(candidate / "VERSION.json", {"name": status, "numerical_contract": "MaleCNS-NumericalContract-v1",
|
||||
"biological_model": "MaleCNS-LIF-reference-v1", "generated_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
|
||||
return record
|
||||
|
||||
|
||||
def make_manifest(cfg: dict[str, Any], final: dict[str, Any]) -> None:
|
||||
exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
roots = [model, exp]
|
||||
files = []
|
||||
for root in roots:
|
||||
for p in sorted(root.rglob("*")):
|
||||
if p.is_file() and p.name not in ("checksums.sha256", "manifest.json"):
|
||||
h = hashlib.sha256()
|
||||
with p.open("rb") as f:
|
||||
for chunk in iter(lambda: f.read(8 << 20), b""): h.update(chunk)
|
||||
files.append({"path": str(p), "size": p.stat().st_size, "sha256": h.hexdigest()})
|
||||
(exp / "manifests").mkdir(parents=True, exist_ok=True)
|
||||
(exp / "manifests/checksums.sha256").write_text("".join(f"{x['sha256']} {x['path']}\n" for x in files))
|
||||
save(exp / "manifests/manifest.json", {"phase": "4P", "status": final["status"], "files": files,
|
||||
"host": platform.node(), "torch": torch.__version__, "cuda": torch.version.cuda,
|
||||
"gpu": torch.cuda.get_device_name(0), "python": platform.python_version()})
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", type=Path, required=True)
|
||||
ap.add_argument("task", choices=["validate"]); args = ap.parse_args()
|
||||
cfg = yaml.safe_load(args.config.read_text()); exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
write_contract(cfg)
|
||||
rt = phase4p.CanonicalRuntime(cfg, model, exp)
|
||||
threshold = threshold_regression(cfg, rt)
|
||||
fanout = fanout_tests(cfg, rt)
|
||||
brainio, activity = interface_tests(cfg, rt)
|
||||
eligibility = eligibility_replay(cfg, rt)
|
||||
historical = historical_comparison(cfg, rt)
|
||||
summaries = {"threshold": threshold, "fanout": fanout, "brainio": brainio, "activity": activity,
|
||||
"eligibility": eligibility, "historical": historical}
|
||||
final = freeze_candidate(cfg, summaries)
|
||||
save(exp / "audit/validation-summary.json", summaries | {"candidate": final})
|
||||
make_manifest(cfg, final)
|
||||
print(json.dumps(final, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,160 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Canonical deterministic MaleCNS LIF reference backend and validation tasks."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pyarrow as pa
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
from torch.utils.cpp_extension import load as load_extension
|
||||
|
||||
import phase4c
|
||||
|
||||
|
||||
def sha_bytes(*arrays: np.ndarray) -> str:
|
||||
h=hashlib.sha256()
|
||||
for x in arrays:h.update(np.ascontiguousarray(x).tobytes())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def save(path: Path,x: object):
|
||||
path.parent.mkdir(parents=True,exist_ok=True)
|
||||
path.write_text(json.dumps(x,indent=2,sort_keys=True,default=str)+'\n')
|
||||
|
||||
|
||||
def build_extension(model:Path,exp:Path):
|
||||
build=exp/'gpu-reference/extension-build';build.mkdir(parents=True,exist_ok=True)
|
||||
start=time.perf_counter()
|
||||
ext=load_extension(name='male_cns_lif_deterministic_v2_ext',
|
||||
sources=[str(model/'phase4p_ext.cpp'),str(model/'phase4p_ext.cu')],
|
||||
build_directory=str(build),extra_cflags=['-O3','-ffp-contract=off'],
|
||||
extra_cuda_cflags=['-O3','--fmad=false','-std=c++20'],verbose=True)
|
||||
report={'status':'PASS','build_seconds':time.perf_counter()-start,'binary':str(Path(ext.__file__)),
|
||||
'binary_sha256':phase4c.sha256(Path(ext.__file__)),'flags':{'cxx':['-O3','-ffp-contract=off'],'cuda':['-O3','--fmad=false','-std=c++20']}}
|
||||
save(exp/'gpu-reference/extension.json',report);return ext
|
||||
|
||||
|
||||
class CanonicalRuntime:
|
||||
def __init__(self,cfg:dict[str,Any],model:Path,exp:Path):
|
||||
self.cfg=cfg;self.exp=exp;self.device=torch.device('cuda:0')
|
||||
if torch.cuda.device_count()!=1 or cfg['device']['required_name_substring'] not in torch.cuda.get_device_name(0):
|
||||
raise RuntimeError('exactly one visible RTX 4090 required')
|
||||
p4cfg=yaml.safe_load((Path(cfg['paths']['historical_runtime_root'])/'phase4c.yaml').read_text())
|
||||
paths=phase4c.exact_paths(p4cfg)
|
||||
self.indptr_np,self.indices_np,self.raw_np=phase4c.p4b.validate_graph(p4cfg,paths)
|
||||
self.table=phase4c.full_metadata(paths['metadata']);self.pops=phase4c.populations(self.table)
|
||||
base=phase4c.Runtime(p4cfg,self.table,self.indptr_np,self.indices_np,self.raw_np,self.device,None)
|
||||
self.indptr,self.indices,self.weights=base.indptr,base.indices,base.weights
|
||||
self.ext=build_extension(model,exp)
|
||||
m=cfg['model'];self.n=int(cfg['expected_graph']['neurons']);self.dt=float(m['dt_ms'])
|
||||
self.delay_steps=int(round(m['synaptic_delay_ms']/self.dt));self.qlen=self.delay_steps+1
|
||||
self.ref_steps=int(round(m['refractory_ms']/self.dt));self.gd=np.float32(1-self.dt/m['synaptic_tau_ms'])
|
||||
self.alpha=np.float32(self.dt/m['membrane_tau_ms']);self.rest=np.float32(m['resting_potential_mv'])
|
||||
self.resetv=np.float32(m['reset_potential_mv']);self.threshold=np.float32(m['threshold_mv'])
|
||||
self.superclass=np.asarray(self.table['annotation_superclass'].to_pylist(),object)
|
||||
|
||||
def reset(self):
|
||||
d=self.device;self.v=torch.full((self.n,),float(self.rest),dtype=torch.float32,device=d)
|
||||
self.g=torch.zeros(self.n,dtype=torch.float32,device=d);self.ref=torch.zeros(self.n,dtype=torch.int32,device=d)
|
||||
self.queue=torch.zeros((self.qlen,self.n),dtype=torch.float32,device=d)
|
||||
self.spike_u8=torch.zeros(self.n,dtype=torch.uint8,device=d);self.counts=torch.zeros(self.n,dtype=torch.int32,device=d)
|
||||
self.step_index=0;self.event_steps=[];self.event_nodes=[]
|
||||
|
||||
def canonical_fanout(self,sources:torch.Tensor,destination:torch.Tensor)->dict[str,float|int]:
|
||||
if not sources.numel():return {'contributions':0,'segments':0,'sort_seconds':0.,'reduce_seconds':0.}
|
||||
starts=self.indptr[sources];degree=self.indptr[sources+1]-starts;cum=torch.cumsum(degree,0);total=int(cum[-1].item())
|
||||
edges=torch.empty(total,dtype=torch.int64,device=self.device)
|
||||
self.ext.fill_edges(self.indptr,sources,cum-degree,edges)
|
||||
t0=time.perf_counter();targets=self.indices[edges]
|
||||
order=torch.argsort(targets,stable=True);sorted_edges=edges[order];sorted_targets=targets[order]
|
||||
boundary=torch.ones(total,dtype=torch.bool,device=self.device)
|
||||
if total>1:boundary[1:]=sorted_targets[1:]!=sorted_targets[:-1]
|
||||
seg_starts=torch.nonzero(boundary,as_tuple=False).flatten().to(torch.int64)
|
||||
unique_targets=sorted_targets[seg_starts].to(torch.int32);sort_sec=time.perf_counter()-t0
|
||||
t1=time.perf_counter();self.ext.segmented_sum(sorted_edges,seg_starts,unique_targets,self.weights,destination)
|
||||
return {'contributions':total,'segments':int(len(unique_targets)),'sort_seconds':sort_sec,'reduce_seconds':time.perf_counter()-t1}
|
||||
|
||||
def run(self,targets_np:np.ndarray,rate:float,seed:int,duration_ms:float,capture=True,forced_override=None):
|
||||
self.reset();steps=int(round(duration_ms/self.dt));targets=torch.as_tensor(targets_np,dtype=torch.int64,device=self.device)
|
||||
forced=forced_override if forced_override is not None else phase4c.generate_forced(steps,len(targets_np),rate,self.dt,seed,self.device)
|
||||
torch.cuda.synchronize();t0=time.perf_counter();sort_t=reduce_t=0.;contrib=0
|
||||
for step in range(steps):
|
||||
slot=step%self.qlen
|
||||
self.ext.state_step(self.v,self.g,self.ref,self.queue[slot],self.spike_u8,self.counts,
|
||||
float(self.gd),float(self.alpha),float(self.rest),float(self.resetv),float(self.threshold),self.ref_steps)
|
||||
self.ext.force_step(self.v,self.ref,self.spike_u8,self.counts,targets,forced[step],float(self.resetv),self.ref_steps)
|
||||
sources=torch.nonzero(self.spike_u8,as_tuple=False).flatten()
|
||||
stats=self.canonical_fanout(sources,self.queue[(step+self.delay_steps)%self.qlen])
|
||||
sort_t+=stats['sort_seconds'];reduce_t+=stats['reduce_seconds'];contrib+=stats['contributions']
|
||||
if capture and sources.numel():
|
||||
x=sources.cpu().numpy().astype(np.int32,copy=True);self.event_nodes.append(x);self.event_steps.append(np.full(len(x),step,np.int32))
|
||||
self.step_index+=1
|
||||
torch.cuda.synchronize();wall=time.perf_counter()-t0
|
||||
es=np.concatenate(self.event_steps) if self.event_steps else np.empty(0,np.int32)
|
||||
en=np.concatenate(self.event_nodes) if self.event_nodes else np.empty(0,np.int32)
|
||||
return {'step':es,'neuron_idx':en,'counts':self.counts.cpu().numpy().copy(),'v':self.v.cpu().numpy().copy(),'g':self.g.cpu().numpy().copy(),
|
||||
'metrics':{'wall_seconds':wall,'biological_ms':duration_ms,'realtime_factor':wall/(duration_ms/1000),
|
||||
'peak_vram_bytes':int(torch.cuda.max_memory_allocated()),'contributions':int(contrib),
|
||||
'sort_submit_seconds':sort_t,'reduce_submit_seconds':reduce_t,'events':len(en),'active':int((self.counts>0).sum().item())}}
|
||||
|
||||
|
||||
def cpu_ordered_sum(edge_ids:np.ndarray,indices:np.ndarray,weights:np.ndarray):
|
||||
edge_ids=np.asarray(edge_ids,np.int64);targets=np.asarray(indices[edge_ids],np.int32)
|
||||
order=np.argsort(targets,kind='stable');ordered=edge_ids[order];ordered_target=targets[order]
|
||||
unique=[];sums=[];begin=0
|
||||
while begin<len(ordered):
|
||||
target=int(ordered_target[begin]);end=begin+1;acc=np.float32(0)
|
||||
while end<len(ordered) and ordered_target[end]==target:end+=1
|
||||
for e in ordered[begin:end]:acc=np.float32(acc+np.float32(weights[e]))
|
||||
unique.append(target);sums.append(acc);begin=end
|
||||
return np.asarray(unique,np.int32),np.asarray(sums,np.float32),ordered
|
||||
|
||||
|
||||
def save_trial(root:Path,run:dict):
|
||||
root.mkdir(parents=True,exist_ok=True)
|
||||
pq.write_table(pa.table({'step':run['step'],'timestamp_ms':run['step'].astype(np.float64)*.1,'neuron_idx':run['neuron_idx']}),root/'spike-events.parquet',compression='zstd')
|
||||
for name,key,dtype in [('per-neuron-spike-counts.int32.npy','counts',np.int32),('final-voltage.float32.npy','v',np.float32),('final-g.float32.npy','g',np.float32)]:np.save(root/name,run[key].astype(dtype),allow_pickle=False)
|
||||
save(root/'statistics.json',run['metrics']|{'event_sha256':sha_bytes(run['step'],run['neuron_idx']),
|
||||
'counts_sha256':sha_bytes(run['counts']),'v_sha256':sha_bytes(run['v']),'g_sha256':sha_bytes(run['g'])})
|
||||
|
||||
|
||||
def phase4c_trials(cfg,rt,repeats):
|
||||
exp=Path(cfg['paths']['experiment_root']);rows=[]
|
||||
for spec in cfg['reference_trials']:
|
||||
target=np.flatnonzero(rt.pops[spec['population']]).astype(np.int64)
|
||||
baseline=None
|
||||
for repeat in range(repeats):
|
||||
run=rt.run(target,float(spec['rate_hz']),int(spec['seed']),float(spec['duration_ms']),True)
|
||||
if baseline is None:
|
||||
baseline={k:run[k].copy() for k in ('step','neuron_idx','counts','v','g')};save_trial(exp/'phase4c-v2/reference'/spec['name'],run)
|
||||
exact={k:bool(np.array_equal(run[k],baseline[k])) for k in ('step','neuron_idx','counts','v','g')}
|
||||
row={'trial':spec['name'],'repeat':repeat,**exact,**run['metrics']};rows.append(row)
|
||||
save(exp/'phase4c-v2/reproducibility-matrix.json',rows)
|
||||
print(json.dumps({'trial':spec['name'],'repeat':repeat,'exact':all(exact.values()),'wall':run['metrics']['wall_seconds']}),flush=True)
|
||||
summary={'repeats':repeats,'matrix':{s['name']:{k:sum(r[k] for r in rows if r['trial']==s['name']) for k in ('step','neuron_idx','counts','v','g')} for s in cfg['reference_trials']},
|
||||
'all_exact':all(all(r[k] for k in ('step','neuron_idx','counts','v','g')) for r in rows),
|
||||
'performance':{s['name']:{'mean_wall_seconds':float(np.mean([r['wall_seconds'] for r in rows if r['trial']==s['name']])),
|
||||
'mean_realtime_factor':float(np.mean([r['realtime_factor'] for r in rows if r['trial']==s['name']])),
|
||||
'peak_vram_bytes':max(r['peak_vram_bytes'] for r in rows if r['trial']==s['name']),
|
||||
'mean_contributions':float(np.mean([r['contributions'] for r in rows if r['trial']==s['name']]))} for s in cfg['reference_trials']}}
|
||||
save(exp/'phase4c-v2/summary.json',summary)
|
||||
|
||||
|
||||
def main():
|
||||
ap=argparse.ArgumentParser();ap.add_argument('--config',type=Path,required=True);ap.add_argument('task',choices=['phase4c']);ap.add_argument('--repeats',type=int)
|
||||
a=ap.parse_args();cfg=yaml.safe_load(a.config.read_text());exp=Path(cfg['paths']['experiment_root']);model=Path(cfg['paths']['model_root'])
|
||||
for sub in ('numerical-contract','cpu-reference','gpu-reference','threshold-3942','fanout-tests','phase4c-v2','historical-comparison','brainio','activity-stream','eligibility-replay','performance','revalidation','regression','audit','manifests'):(exp/sub).mkdir(parents=True,exist_ok=True)
|
||||
rt=CanonicalRuntime(cfg,model,exp)
|
||||
phase4c_trials(cfg,rt,a.repeats or int(cfg['repeats_per_trial']))
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,51 @@
|
||||
phase: 4P
|
||||
name: MaleCNS-LIF-runtime-v2-deterministic-candidate
|
||||
description: Shiu-like LIF dynamics adapted to MaleCNS v1.0 with canonical deterministic float32 reduction
|
||||
paths:
|
||||
connectome_root: /data/digital-drosophila/datasets/male-cns-v1.0/processed/neuron-only
|
||||
historical_runtime_root: /data/digital-drosophila/models/male-cns-lif/runtime-v1
|
||||
historical_phase4c_root: /data/digital-drosophila/experiments/phase4c-runtime
|
||||
phase4o_root: /data/digital-drosophila/experiments/phase4o-runtime-credit
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4p-deterministic-runtime
|
||||
expected_graph: {neurons: 166700, directed_edges: 25582938, synaptic_contact_weight_sum: 124177617}
|
||||
model:
|
||||
dt_ms: 0.1
|
||||
membrane_tau_ms: 20.0
|
||||
synaptic_tau_ms: 5.0
|
||||
refractory_ms: 2.2
|
||||
resting_potential_mv: -52.0
|
||||
reset_potential_mv: -52.0
|
||||
threshold_mv: -45.0
|
||||
synaptic_delay_ms: 1.8
|
||||
w_syn_mv_per_anatomical_synapse: 0.275
|
||||
integration: forward_euler
|
||||
sign_policy:
|
||||
neurotransmitter_sign: {acetylcholine: 1, gaba: -1, glutamate: -1, histamine: -1, dopamine: 1, octopamine: 1, serotonin: 1}
|
||||
unknown_sign: 0
|
||||
unclear_sign: 0
|
||||
device:
|
||||
required_name_substring: RTX 4090
|
||||
visible_gpu_uuid: GPU-79f3a85d-adce-7833-7003-6488140ca596
|
||||
canonical_contract:
|
||||
contribution_key: [target_idx, source_idx, edge_id]
|
||||
equivalent_implementation_key: [target_idx, edge_id]
|
||||
edge_id_property: CSR edge_id is globally ordered by source row and within-row edge position
|
||||
accumulation: sequential IEEE-754 float32 round-to-nearest-even using __fadd_rn
|
||||
state_update: fused explicit IEEE-754 float32 operations using __fmul_rn/__fadd_rn/__fsub_rn
|
||||
threshold: V >= V_threshold
|
||||
reference_trials:
|
||||
- {name: auditory-10hz, population: auditory, rate_hz: 10, seed: 42001, duration_ms: 1000.0}
|
||||
- {name: auditory-100hz, population: auditory, rate_hz: 100, seed: 42001, duration_ms: 1000.0}
|
||||
- {name: olfactory-50hz, population: olfactory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
- {name: mechanosensory-50hz, population: mechanosensory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
- {name: gustatory-50hz, population: gustatory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
repeats_per_trial: 20
|
||||
fanout_test_cases: 500
|
||||
eligibility_fixed_event_repeats: 10
|
||||
prohibitions:
|
||||
modify_biological_parameters: true
|
||||
overwrite_historical_reference: true
|
||||
nondeterministic_atomic_scientific_backend: true
|
||||
threshold_epsilon: true
|
||||
learning: true
|
||||
@@ -0,0 +1,69 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Instrumented 100 ms canonical-backend cost audit."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import resource
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
import phase4p
|
||||
|
||||
|
||||
def main() -> None:
|
||||
model = Path("/data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2")
|
||||
cfg = yaml.safe_load((model / "phase4p.yaml").read_text()); exp = Path(cfg["paths"]["experiment_root"])
|
||||
rt = phase4p.CanonicalRuntime(cfg, model, exp); spec = cfg["reference_trials"][1]
|
||||
target_np = np.flatnonzero(rt.pops[spec["population"]]).astype(np.int64)
|
||||
targets = torch.as_tensor(target_np, dtype=torch.int64, device=rt.device); steps = 1000
|
||||
forced = phase4p.phase4c.generate_forced(steps, len(target_np), float(spec["rate_hz"]), rt.dt, int(spec["seed"]), rt.device)
|
||||
rt.reset(); samples = []; stop = threading.Event()
|
||||
def sample():
|
||||
while not stop.is_set():
|
||||
try:
|
||||
text = subprocess.check_output(["nvidia-smi", "--id=" + cfg["device"]["visible_gpu_uuid"],
|
||||
"--query-gpu=utilization.gpu,memory.used", "--format=csv,noheader,nounits"], text=True).strip()
|
||||
u, m = text.split(","); samples.append({"time": time.time(), "gpu_util_percent": float(u), "gpu_memory_mib": float(m)})
|
||||
except Exception as e: samples.append({"error": str(e)})
|
||||
stop.wait(0.10)
|
||||
thread = threading.Thread(target=sample, daemon=True); thread.start()
|
||||
use0 = resource.getrusage(resource.RUSAGE_SELF); cpu0 = use0.ru_utime + use0.ru_stime
|
||||
torch.cuda.reset_peak_memory_stats(); torch.cuda.synchronize()
|
||||
wall0 = time.perf_counter(); state_submit = force_submit = sort_submit = reduce_submit = 0.0; contributions = 0
|
||||
for step in range(steps):
|
||||
slot = step % rt.qlen; t = time.perf_counter()
|
||||
rt.ext.state_step(rt.v, rt.g, rt.ref, rt.queue[slot], rt.spike_u8, rt.counts,
|
||||
float(rt.gd), float(rt.alpha), float(rt.rest), float(rt.resetv), float(rt.threshold), rt.ref_steps)
|
||||
state_submit += time.perf_counter() - t; t = time.perf_counter()
|
||||
rt.ext.force_step(rt.v, rt.ref, rt.spike_u8, rt.counts, targets, forced[step], float(rt.resetv), rt.ref_steps)
|
||||
force_submit += time.perf_counter() - t
|
||||
sources = torch.nonzero(rt.spike_u8, as_tuple=False).flatten()
|
||||
st = rt.canonical_fanout(sources, rt.queue[(step + rt.delay_steps) % rt.qlen])
|
||||
sort_submit += st["sort_seconds"]; reduce_submit += st["reduce_seconds"]; contributions += st["contributions"]
|
||||
torch.cuda.synchronize(); wall = time.perf_counter() - wall0
|
||||
use1 = resource.getrusage(resource.RUSAGE_SELF); cpu1 = use1.ru_utime + use1.ru_stime
|
||||
stop.set(); thread.join()
|
||||
good = [x for x in samples if "gpu_util_percent" in x]
|
||||
out = {"trial": "auditory-100hz", "biological_ms": 100.0, "steps": steps, "wall_seconds": wall,
|
||||
"wall_per_100ms_biological": wall, "realtime_factor": wall / 0.1, "steps_per_second": steps / wall,
|
||||
"process_CPU_percent_of_one_core": 100 * (cpu1 - cpu0) / wall,
|
||||
"GPU_utilization_percent_mean": float(np.mean([x["gpu_util_percent"] for x in good])) if good else None,
|
||||
"GPU_utilization_percent_max": float(np.max([x["gpu_util_percent"] for x in good])) if good else None,
|
||||
"nvidia_smi_memory_mib_max": float(np.max([x["gpu_memory_mib"] for x in good])) if good else None,
|
||||
"torch_peak_VRAM_bytes": int(torch.cuda.max_memory_allocated()), "contribution_count": int(contributions),
|
||||
"CPU_submit_seconds": {"state_integration": state_submit, "external_force": force_submit,
|
||||
"canonical_sort_and_segment_setup": sort_submit, "segmented_reduction": reduce_submit},
|
||||
"sampling_interval_seconds": 0.10, "samples": samples,
|
||||
"interpretation": "Submission sub-times overlap asynchronous CUDA work; synchronized total wall time is authoritative."}
|
||||
p = exp / "performance/instrumented-100ms.json"; p.parent.mkdir(parents=True, exist_ok=True)
|
||||
p.write_text(json.dumps(out, indent=2, sort_keys=True) + "\n"); print(json.dumps(out, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,25 @@
|
||||
#include <torch/extension.h>
|
||||
|
||||
void state_step_cuda(torch::Tensor v, torch::Tensor g, torch::Tensor refractory,
|
||||
torch::Tensor arrival, torch::Tensor spikes,
|
||||
torch::Tensor spike_counts, double g_decay, double alpha,
|
||||
double rest, double reset, double threshold,
|
||||
int64_t refractory_steps);
|
||||
void force_step_cuda(torch::Tensor v, torch::Tensor refractory,
|
||||
torch::Tensor spikes, torch::Tensor spike_counts,
|
||||
torch::Tensor targets, torch::Tensor forced,
|
||||
double reset, int64_t refractory_steps);
|
||||
void fill_edges_cuda(torch::Tensor indptr, torch::Tensor sources,
|
||||
torch::Tensor output_offsets, torch::Tensor output);
|
||||
void segmented_sum_cuda(torch::Tensor sorted_edge_ids,
|
||||
torch::Tensor segment_starts,
|
||||
torch::Tensor segment_targets,
|
||||
torch::Tensor weights,
|
||||
torch::Tensor destination);
|
||||
|
||||
PYBIND11_MODULE(TORCH_EXTENSION_NAME, m) {
|
||||
m.def("state_step", &state_step_cuda, "Canonical float32 LIF update (CUDA)");
|
||||
m.def("force_step", &force_step_cuda, "Canonical external spike injection (CUDA)");
|
||||
m.def("fill_edges", &fill_edges_cuda, "Source/edge ordered CSR expansion (CUDA)");
|
||||
m.def("segmented_sum", &segmented_sum_cuda, "Fixed-order sequential segmented float32 sum (CUDA)");
|
||||
}
|
||||
@@ -0,0 +1,87 @@
|
||||
#include <torch/extension.h>
|
||||
#include <c10/cuda/CUDAException.h>
|
||||
#include <cuda.h>
|
||||
#include <cuda_runtime.h>
|
||||
#include <cstdint>
|
||||
|
||||
namespace {
|
||||
void check_cuda(const torch::Tensor& x) {
|
||||
TORCH_CHECK(x.is_cuda(), "all tensors must be CUDA tensors");
|
||||
TORCH_CHECK(x.is_contiguous(), "all tensors must be contiguous");
|
||||
}
|
||||
|
||||
__global__ void state_kernel(float* v, float* g, int* refractory,
|
||||
float* arrival, unsigned char* spikes,
|
||||
int* counts, int n, float gd, float alpha,
|
||||
float rest, float reset, float threshold,
|
||||
int ref_steps) {
|
||||
int i=blockIdx.x*blockDim.x+threadIdx.x;
|
||||
if(i>=n) return;
|
||||
float gi=__fadd_rn(__fmul_rn(g[i],gd),arrival[i]);
|
||||
g[i]=gi; arrival[i]=0.0f;
|
||||
int r=refractory[i]; r=r>0?r-1:0; refractory[i]=r;
|
||||
float vi=v[i]; bool available=r==0;
|
||||
if(available){
|
||||
float leak=__fsub_rn(vi,rest);
|
||||
float drive=__fsub_rn(gi,leak);
|
||||
vi=__fadd_rn(vi,__fmul_rn(alpha,drive));
|
||||
} else vi=reset;
|
||||
bool fired=available && vi>=threshold;
|
||||
if(fired){vi=reset;refractory[i]=ref_steps;counts[i]+=1;}
|
||||
v[i]=vi;spikes[i]=fired?1:0;
|
||||
}
|
||||
|
||||
__global__ void force_kernel(float* v,int* refractory,unsigned char* spikes,
|
||||
int* counts,const int64_t* targets,
|
||||
const unsigned char* forced,int n,float reset,
|
||||
int ref_steps){
|
||||
int j=blockIdx.x*blockDim.x+threadIdx.x;
|
||||
if(j>=n||!forced[j]) return;
|
||||
int64_t i=targets[j]; if(spikes[i]||refractory[i]!=0) return;
|
||||
spikes[i]=1;v[i]=reset;refractory[i]=ref_steps;counts[i]+=1;
|
||||
}
|
||||
|
||||
__global__ void fill_edges_kernel(const int64_t* indptr,const int64_t* sources,
|
||||
const int64_t* offsets,int64_t* output,int n){
|
||||
int j=blockIdx.x*blockDim.x+threadIdx.x;
|
||||
if(j>=n) return;
|
||||
int64_t src=sources[j],begin=indptr[src],degree=indptr[src+1]-begin,out=offsets[j];
|
||||
for(int64_t k=0;k<degree;++k) output[out+k]=begin+k;
|
||||
}
|
||||
|
||||
// Exactly one CUDA thread owns a target segment. The loop order is the
|
||||
// canonical target -> source -> edge_id order supplied by the host.
|
||||
__global__ void segmented_sum_kernel(const int64_t* edges,const int64_t* starts,
|
||||
const int* targets,const float* weights,
|
||||
float* destination,int segments,
|
||||
int64_t edge_count){
|
||||
int s=blockIdx.x*blockDim.x+threadIdx.x;
|
||||
if(s>=segments) return;
|
||||
int64_t begin=starts[s],end=(s+1<segments)?starts[s+1]:edge_count;
|
||||
float acc=0.0f;
|
||||
for(int64_t j=begin;j<end;++j) acc=__fadd_rn(acc,weights[edges[j]]);
|
||||
destination[targets[s]]=__fadd_rn(destination[targets[s]],acc);
|
||||
}
|
||||
}
|
||||
|
||||
void state_step_cuda(torch::Tensor v,torch::Tensor g,torch::Tensor refractory,
|
||||
torch::Tensor arrival,torch::Tensor spikes,torch::Tensor counts,
|
||||
double gd,double alpha,double rest,double reset,double threshold,
|
||||
int64_t ref_steps){
|
||||
check_cuda(v);check_cuda(g);check_cuda(refractory);check_cuda(arrival);check_cuda(spikes);check_cuda(counts);
|
||||
int n=v.numel();state_kernel<<<(n+255)/256,256>>>(v.data_ptr<float>(),g.data_ptr<float>(),refractory.data_ptr<int>(),arrival.data_ptr<float>(),spikes.data_ptr<unsigned char>(),counts.data_ptr<int>(),n,(float)gd,(float)alpha,(float)rest,(float)reset,(float)threshold,(int)ref_steps);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
||||
}
|
||||
void force_step_cuda(torch::Tensor v,torch::Tensor refractory,torch::Tensor spikes,
|
||||
torch::Tensor counts,torch::Tensor targets,torch::Tensor forced,
|
||||
double reset,int64_t ref_steps){
|
||||
check_cuda(targets);check_cuda(forced);int n=targets.numel();force_kernel<<<(n+255)/256,256>>>(v.data_ptr<float>(),refractory.data_ptr<int>(),spikes.data_ptr<unsigned char>(),counts.data_ptr<int>(),targets.data_ptr<int64_t>(),forced.data_ptr<unsigned char>(),n,(float)reset,(int)ref_steps);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
||||
}
|
||||
void fill_edges_cuda(torch::Tensor indptr,torch::Tensor sources,torch::Tensor offsets,torch::Tensor output){
|
||||
check_cuda(indptr);check_cuda(sources);check_cuda(offsets);check_cuda(output);int n=sources.numel();fill_edges_kernel<<<(n+255)/256,256>>>(indptr.data_ptr<int64_t>(),sources.data_ptr<int64_t>(),offsets.data_ptr<int64_t>(),output.data_ptr<int64_t>(),n);C10_CUDA_KERNEL_LAUNCH_CHECK();
|
||||
}
|
||||
void segmented_sum_cuda(torch::Tensor edges,torch::Tensor starts,torch::Tensor targets,
|
||||
torch::Tensor weights,torch::Tensor destination){
|
||||
check_cuda(edges);check_cuda(starts);check_cuda(targets);check_cuda(weights);check_cuda(destination);
|
||||
TORCH_CHECK(edges.scalar_type()==torch::kInt64&&starts.scalar_type()==torch::kInt64&&targets.scalar_type()==torch::kInt32,"index dtype mismatch");
|
||||
int n=targets.numel();if(n) segmented_sum_kernel<<<(n+255)/256,256>>>(edges.data_ptr<int64_t>(),starts.data_ptr<int64_t>(),targets.data_ptr<int>(),weights.data_ptr<float>(),destination.data_ptr<float>(),n,edges.numel());C10_CUDA_KERNEL_LAUNCH_CHECK();
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Read-only frozen-integrity verification and final Phase 4P sealing."""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import json
|
||||
import shutil
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
import yaml
|
||||
|
||||
import phase4p_validate
|
||||
from phase4h_audit import verify_checksum
|
||||
from phase4k import dataset_checks
|
||||
from phase4m import frozen_checks, frozen_hashes
|
||||
from phase4n import load_config
|
||||
|
||||
|
||||
ROOT = Path("/data/digital-drosophila")
|
||||
|
||||
|
||||
def save(path: Path, value: object) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, indent=2, sort_keys=True, default=str) + "\n")
|
||||
|
||||
|
||||
def read(path: Path): return json.loads(path.read_text())
|
||||
|
||||
|
||||
def integrity() -> dict:
|
||||
ncfg, mcfg, cfg, _ = load_config(ROOT / "models/male-cns-lif/credit-assignment-v1/phase4n.yaml")
|
||||
checks = {}
|
||||
for label, root, rel in [
|
||||
("Phase4M experiment", Path(mcfg["experiment_root"]), "manifests/checksums.sha256"),
|
||||
("Phase4M model", Path(mcfg["model_root"]), "checksums.sha256"),
|
||||
("Phase4N experiment", Path(ncfg["experiment_root"]), "manifests/checksums.sha256"),
|
||||
("Phase4N model", Path(ncfg["model_root"]), "checksums.sha256")]:
|
||||
checks[label] = verify_checksum(root, root / rel)
|
||||
earlier = frozen_checks(cfg); datasets = dataset_checks(cfg); current = frozen_hashes(cfg)
|
||||
saved = read(Path(mcfg["experiment_root"]) / "audit/final-summary.json")["integrity"]["after"]
|
||||
phase4o = verify_checksum(ROOT / "experiments/phase4o-runtime-credit",
|
||||
ROOT / "experiments/phase4o-runtime-credit/manifests/checksums.sha256")
|
||||
report = {"Phase1_to_3B": datasets, "Phase4C_to_4L": earlier, "Phase4M_4N": checks,
|
||||
"Phase4O": phase4o, "frozen_input_hashes_unchanged": current == saved}
|
||||
report["all_static_checks_passed"] = bool(datasets["all_passed"] and earlier["all_passed"] and
|
||||
all(x["passed"] for x in checks.values()) and phase4o["passed"] and current == saved)
|
||||
return report
|
||||
|
||||
|
||||
def seal_dir(root: Path, control: Path) -> int:
|
||||
rows = []
|
||||
for p in sorted(root.rglob("*")):
|
||||
if p.is_file() and p != control and p.name not in ("checksums.sha256", "MANIFEST.json"):
|
||||
h = hashlib.sha256()
|
||||
with p.open("rb") as f:
|
||||
for chunk in iter(lambda: f.read(8 << 20), b""): h.update(chunk)
|
||||
rows.append((h.hexdigest(), p.relative_to(root).as_posix(), p.stat().st_size))
|
||||
control.parent.mkdir(parents=True, exist_ok=True)
|
||||
control.write_text("".join(f"{h} {rel}\n" for h, rel, _ in rows))
|
||||
save(control.parent / "MANIFEST.json", {"root": str(root), "files": [{"path": r, "sha256": h, "size": s} for h, r, s in rows]})
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
cfg = yaml.safe_load((ROOT / "models/male-cns-lif/deterministic-runtime-v2/phase4p.yaml").read_text())
|
||||
exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
frozen = integrity(); save(exp / "audit/frozen-integrity-final.json", frozen)
|
||||
gate = read(exp / "audit/candidate-gate.json"); p4c = read(exp / "phase4c-v2/summary.json")
|
||||
reval = read(exp / "revalidation/summary.json"); threshold = read(exp / "threshold-3942/summary.json")
|
||||
fanout = read(exp / "fanout-tests/summary.json"); history = read(exp / "historical-comparison/summary.json")
|
||||
brainio = read(exp / "brainio/summary.json"); activity = read(exp / "activity-stream/summary.json")
|
||||
eligibility = read(exp / "eligibility-replay/summary.json")
|
||||
performance = read(exp / "performance/instrumented-100ms.json")
|
||||
highest = reval["highest_revalidated"]
|
||||
final = {"phase": "4P", "status": gate["status"], "candidate_generated": gate["candidate_generated"],
|
||||
"numerical_contract": "MaleCNS-NumericalContract-v1", "canonical_order": ["target_idx", "source_idx", "edge_id"],
|
||||
"neuron_3942": threshold, "real_fanout": fanout, "phase4c_5x20": p4c,
|
||||
"historical_comparison": history, "BrainIO": brainio, "ActivityStream": activity,
|
||||
"eligibility_fixed_event_replay": eligibility, "performance": performance, "candidate_gates": gate["gates"],
|
||||
"revalidation": reval, "highest_revalidated": highest,
|
||||
"frozen_integrity": frozen, "prohibited_follow_on_started": False,
|
||||
"limitations": [
|
||||
"Canonical stable sorting plus one-thread-per-target reduction is correctness-first and 2.6-5.0x slower than biological real time for the 1 s trials.",
|
||||
"Historical runtime-v1 full population/state distributions were not persisted for all 50 Phase4O repeats; total-spike envelopes use all runs while population/V/g comparisons use the frozen historical reference.",
|
||||
"The compiler emitted a generic c++ versus g++ warning even though the cached extension loaded and all exact regressions passed."]}
|
||||
save(exp / "audit/final-summary.json", final)
|
||||
candidate = model / "candidate"; shutil.copy2(model / "phase4p_revalidate.py", candidate / "phase4p_revalidate.py")
|
||||
shutil.copy2(model / "phase4p_finalize.py", candidate / "phase4p_finalize.py")
|
||||
shutil.copy2(model / "phase4p_benchmark.py", candidate / "phase4p_benchmark.py")
|
||||
save(candidate / "VERSION.json", {"name": gate["status"], "numerical_contract": "MaleCNS-NumericalContract-v1",
|
||||
"revalidated_through": highest, "generated_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
|
||||
"biological_parameters_changed": False, "historical_references_overwritten": False})
|
||||
candidate_files = seal_dir(candidate, candidate / "checksums.sha256")
|
||||
phase4p_validate.make_manifest(cfg, final)
|
||||
own = verify_checksum(candidate, candidate / "checksums.sha256")
|
||||
print(json.dumps({"status": final["status"], "highest_revalidated": highest,
|
||||
"frozen_integrity": frozen["all_static_checks_passed"],
|
||||
"candidate_files": candidate_files, "candidate_checksum": own["passed"]}, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,449 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Ordered Phase 4P-B dynamic revalidation on the canonical runtime.
|
||||
|
||||
R1/R2 consume the already sealed Phase 4P results. R3-R6 are run in order
|
||||
and the script stops immediately if a level cannot reproduce its declared
|
||||
historical result. No frozen artifact is written.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
import phase4p
|
||||
from phase4l import pn_drive
|
||||
from phase4m import load_odor_inputs
|
||||
|
||||
|
||||
ROOT = Path("/data/digital-drosophila")
|
||||
|
||||
|
||||
def save(path: Path, value: object) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, indent=2, sort_keys=True, default=str) + "\n")
|
||||
|
||||
|
||||
def ahash(*values: np.ndarray) -> str:
|
||||
h = hashlib.sha256()
|
||||
for value in values: h.update(np.ascontiguousarray(value).tobytes())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def metrics(a: np.ndarray, b: np.ndarray) -> dict[str, float]:
|
||||
a = a.astype(np.float64); b = b.astype(np.float64)
|
||||
aa = a > 0; bb = b > 0; union = np.count_nonzero(aa | bb)
|
||||
den = np.linalg.norm(a) * np.linalg.norm(b)
|
||||
return {"euclidean": float(np.linalg.norm(a - b)),
|
||||
"jaccard": float(np.count_nonzero(aa & bb) / union) if union else 1.0,
|
||||
"cosine": float(np.dot(a, b) / den) if den else (1.0 if not np.any(a) and not np.any(b) else 0.0)}
|
||||
|
||||
|
||||
class DeterministicAPLSim:
|
||||
"""Frozen APL overlay on top of the Phase 4P canonical LIF kernel."""
|
||||
def __init__(self, rt: phase4p.CanonicalRuntime):
|
||||
self.rt = rt; self.n = rt.n; self.device = rt.device; self.dt = rt.dt
|
||||
meta = rt.table.to_pandas().sort_values("neuron_idx").reset_index(drop=True)
|
||||
self.meta = meta
|
||||
cls = meta.annotation_class.fillna("").astype(str).to_numpy()
|
||||
typ = meta.annotation_type.fillna("").astype(str).to_numpy()
|
||||
self.kc = np.flatnonzero(cls == "Kenyon_Cell").astype(np.int32)
|
||||
self.mbon = np.flatnonzero(cls == "MBON").astype(np.int32)
|
||||
self.pn = np.flatnonzero(cls == "ALPN").astype(np.int32)
|
||||
self.apl = np.flatnonzero(typ == "APL").astype(np.int32)
|
||||
if len(self.apl) != 2 or len(self.kc) != 4064: raise RuntimeError("frozen APL/KC identity changed")
|
||||
self.sources = np.repeat(np.arange(self.n, dtype=np.int32), np.diff(rt.indptr_np).astype(np.int64))
|
||||
idx = rt.indices_np; raw = rt.raw_np
|
||||
apl_out = np.isin(self.sources, self.apl)
|
||||
self.apl_out_ids = np.flatnonzero(apl_out).astype(np.int64)
|
||||
input_matrix = np.zeros((len(self.apl), self.n), np.float32)
|
||||
output_matrix = np.zeros((len(self.apl), len(self.kc)), np.float32)
|
||||
kc_lookup = np.full(self.n, -1, np.int32); kc_lookup[self.kc] = np.arange(len(self.kc), dtype=np.int32)
|
||||
allowed = np.isin(self.sources, np.concatenate((self.kc, self.pn))) & np.isin(idx, self.apl)
|
||||
apl_lookup = {int(x): i for i, x in enumerate(self.apl)}
|
||||
for eid in np.flatnonzero(allowed): input_matrix[apl_lookup[int(idx[eid])], int(self.sources[eid])] += float(raw[eid])
|
||||
for eid in np.flatnonzero(apl_out & np.isin(idx, self.kc)):
|
||||
output_matrix[apl_lookup[int(self.sources[eid])], kc_lookup[int(idx[eid])]] += float(raw[eid])
|
||||
self.input_scale = float(input_matrix[input_matrix > 0].sum() / np.count_nonzero(np.any(input_matrix > 0, axis=0)))
|
||||
self.output_scale = float(output_matrix[output_matrix > 0].mean())
|
||||
self.input_gpu = torch.as_tensor(input_matrix / self.input_scale, dtype=torch.float32, device=self.device)
|
||||
self.output_gpu = torch.as_tensor(output_matrix / self.output_scale, dtype=torch.float32, device=self.device)
|
||||
self.kc_gpu = torch.as_tensor(self.kc, dtype=torch.int64, device=self.device)
|
||||
self.apl_out_gpu = torch.as_tensor(self.apl_out_ids, dtype=torch.int64, device=self.device)
|
||||
self.base = rt.weights.clone()
|
||||
p4h = ROOT / "experiments/phase4h-dopamine-plasticity"
|
||||
self.identity = pd.read_parquet(ROOT / "experiments/phase4n-credit-assignment/edge-identity/frozen-edge-identity.parquet")
|
||||
self.edge_ids = self.identity.edge_id.to_numpy(np.int64)
|
||||
self.edge_gpu = torch.as_tensor(self.edge_ids, dtype=torch.int64, device=self.device)
|
||||
self.base_candidate = self.base[self.edge_gpu].cpu().numpy().astype(np.float32)
|
||||
self.confirmed = pd.read_parquet(p4h / "mappings/dan-to-mbon-confirmed.parquet")
|
||||
down = pd.read_parquet(p4h / "mappings/mbon-to-downstream-actual-edges.parquet")
|
||||
self.downstream = np.unique(down.target_neuron_idx.to_numpy(np.int32))
|
||||
|
||||
def dan_targets(self, dtype: str, compartment: str) -> np.ndarray:
|
||||
x = self.confirmed[(self.confirmed.source_annotation_type == dtype) &
|
||||
(self.confirmed.source_compartment_verbatim == compartment)]
|
||||
out = np.sort(x.source_neuron_idx.unique().astype(np.int32))
|
||||
if not len(out): raise RuntimeError(f"no confirmed DAN for {dtype}/{compartment}")
|
||||
return out
|
||||
|
||||
def install(self, E: np.ndarray | None) -> None:
|
||||
self.rt.weights.copy_(self.base)
|
||||
if E is not None:
|
||||
self.rt.weights[self.edge_gpu] = torch.as_tensor(self.base_candidate * E.astype(np.float32), device=self.device)
|
||||
self.rt.weights[self.apl_out_gpu] = 0.0
|
||||
|
||||
def run(self, targets: np.ndarray, rates_hz: np.ndarray, seed: int, duration_ms: float,
|
||||
E: np.ndarray | None = None, dan_targets: np.ndarray | None = None,
|
||||
dan_pulses: list[float] | None = None, collect_events: bool = False) -> dict[str, Any]:
|
||||
targets = np.asarray(targets, np.int32); rates_hz = np.asarray(rates_hz, np.float32)
|
||||
self.install(E); self.rt.reset(); steps = int(round(duration_ms / self.dt))
|
||||
target_gpu = torch.as_tensor(targets, dtype=torch.int64, device=self.device)
|
||||
gen = torch.Generator(device=self.device); gen.manual_seed(int(seed))
|
||||
probs = torch.as_tensor(rates_hz * self.dt / 1000.0, dtype=torch.float32, device=self.device)
|
||||
forced = (torch.rand((steps, len(targets)), generator=gen, device=self.device) < probs[None, :]).to(torch.uint8)
|
||||
dan = np.empty(0, np.int32) if dan_targets is None else np.asarray(dan_targets, np.int32)
|
||||
dan_gpu = torch.as_tensor(dan, dtype=torch.int64, device=self.device)
|
||||
dan_forced = torch.zeros((steps, len(dan)), dtype=torch.uint8, device=self.device)
|
||||
for pulse in dan_pulses or []:
|
||||
s = int(round(pulse / self.dt))
|
||||
if 0 <= s < steps: dan_forced[s] = True
|
||||
dan_steps = {int(round(x / self.dt)) for x in (dan_pulses or [])}
|
||||
apl_state = torch.zeros(len(self.apl), dtype=torch.float32, device=self.device)
|
||||
kc_trace = torch.zeros(len(self.kc), dtype=torch.float32, device=self.device)
|
||||
first = torch.full((self.n,), -1, dtype=torch.int32, device=self.device)
|
||||
trace_at_dan = []
|
||||
apl_decay = np.float32(math.exp(-self.dt / 20.0)); trace_decay = np.float32(math.exp(-self.dt / 100.0))
|
||||
kc_steps: list[np.ndarray] = []; kc_nodes: list[np.ndarray] = []
|
||||
torch.cuda.synchronize(); t0 = time.perf_counter()
|
||||
for step in range(steps):
|
||||
slot = step % self.rt.qlen
|
||||
self.rt.ext.state_step(self.rt.v, self.rt.g, self.rt.ref, self.rt.queue[slot], self.rt.spike_u8, self.rt.counts,
|
||||
float(self.rt.gd), float(self.rt.alpha), float(self.rt.rest), float(self.rt.resetv),
|
||||
float(self.rt.threshold), self.rt.ref_steps)
|
||||
self.rt.ext.force_step(self.rt.v, self.rt.ref, self.rt.spike_u8, self.rt.counts,
|
||||
target_gpu, forced[step], float(self.rt.resetv), self.rt.ref_steps)
|
||||
if len(dan):
|
||||
self.rt.ext.force_step(self.rt.v, self.rt.ref, self.rt.spike_u8, self.rt.counts,
|
||||
dan_gpu, dan_forced[step], float(self.rt.resetv), self.rt.ref_steps)
|
||||
sources = torch.nonzero(self.rt.spike_u8, as_tuple=False).flatten()
|
||||
if sources.numel():
|
||||
missing = first[sources] < 0; first[sources[missing]] = step
|
||||
self.rt.canonical_fanout(sources, self.rt.queue[(step + self.rt.delay_steps) % self.rt.qlen])
|
||||
kc_spike = self.rt.spike_u8[self.kc_gpu]
|
||||
kc_trace.mul_(float(trace_decay)).add_(kc_spike.to(torch.float32)).clamp_max_(1.0)
|
||||
if step in dan_steps: trace_at_dan.append(kc_trace.clone())
|
||||
if collect_events:
|
||||
active = torch.nonzero(kc_spike, as_tuple=False).flatten()
|
||||
if active.numel():
|
||||
node = self.kc_gpu[active].cpu().numpy().astype(np.int32)
|
||||
kc_nodes.append(node); kc_steps.append(np.full(len(node), step, np.int32))
|
||||
# Frozen APL graded-inhibition overlay. Each KC target is unique in
|
||||
# this assignment, avoiding index_add atomics.
|
||||
apl_state.mul_(float(apl_decay)).add_(torch.mv(self.input_gpu, self.rt.spike_u8.to(torch.float32)))
|
||||
inhibition = torch.matmul(apl_state, self.output_gpu) * np.float32(0.01)
|
||||
self.rt.g[self.kc_gpu] = self.rt.g[self.kc_gpu] - inhibition
|
||||
self.rt.step_index += 1
|
||||
torch.cuda.synchronize(); wall = time.perf_counter() - t0
|
||||
counts = self.rt.counts.cpu().numpy().copy(); first_np = first.cpu().numpy()
|
||||
ev_step = np.concatenate(kc_steps) if kc_steps else np.empty(0, np.int32)
|
||||
ev_node = np.concatenate(kc_nodes) if kc_nodes else np.empty(0, np.int32)
|
||||
return {"counts": counts, "kc_counts": counts[self.kc], "mbon_counts": counts[self.mbon],
|
||||
"downstream_counts": counts[self.downstream], "first": first_np,
|
||||
"final_v": self.rt.v.cpu().numpy().copy(), "final_g": self.rt.g.cpu().numpy().copy(),
|
||||
"trace": np.zeros(len(self.kc), np.float32) if not trace_at_dan else trace_at_dan[-1].cpu().numpy().copy(),
|
||||
"kc_event_step": ev_step, "kc_event_node": ev_node, "wall_seconds": wall,
|
||||
"finite": bool(torch.isfinite(self.rt.v).all() and torch.isfinite(self.rt.g).all())}
|
||||
|
||||
|
||||
def r1_r2(exp: Path) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
p4c = json.loads((exp / "phase4c-v2/summary.json").read_text())
|
||||
gate = json.loads((exp / "audit/candidate-gate.json").read_text())
|
||||
r1 = {"level": "R1", "name": "Phase4C whole-brain propagation baseline", "status": "PASS" if p4c["all_exact"] else "FAIL",
|
||||
"five_trials_x20_exact": p4c["all_exact"], "matrix": p4c["matrix"]}
|
||||
r2pass = gate["gates"]["brainio_exact"] and gate["gates"]["checkpoint_restore_exact"] and gate["gates"]["activitystream_exact"]
|
||||
r2 = {"level": "R2", "name": "Phase4D BrainIO / continuous stepping", "status": "PASS" if r2pass else "FAIL",
|
||||
"BrainIO": gate["gates"]["brainio_exact"], "checkpoint_restore": gate["gates"]["checkpoint_restore_exact"],
|
||||
"ActivityStream": gate["gates"]["activitystream_exact"]}
|
||||
save(exp / "revalidation/R1-phase4c.json", r1); save(exp / "revalidation/R2-phase4d.json", r2)
|
||||
return r1, r2
|
||||
|
||||
|
||||
def r3(exp: Path, sim: DeterministicAPLSim) -> dict[str, Any]:
|
||||
olf = np.flatnonzero(sim.rt.pops["olfactory"]).astype(np.int32); olf.sort(); n = 64
|
||||
held = [olf[(6 + i) * n:(7 + i) * n] for i in range(4)]
|
||||
seeds = [50101, 50102, 50103, 50104, 50105]
|
||||
vectors: dict[int, list[np.ndarray]] = {i: [] for i in range(4)}; rows = []
|
||||
for oi, targets in enumerate(held):
|
||||
rates = np.full(len(targets), 50.0, np.float32)
|
||||
for seed in seeds:
|
||||
run = sim.run(targets, rates, seed + oi * 100, 100.0)
|
||||
active = int(np.count_nonzero(run["kc_counts"])); fraction = active / len(sim.kc)
|
||||
vectors[oi].append(run["kc_counts"])
|
||||
rows.append({"odor": oi, "seed": seed, "KC_active": active, "KC_active_fraction": fraction,
|
||||
"finite": run["finite"], "wall_seconds": run["wall_seconds"], "hash": ahash(run["kc_counts"])})
|
||||
within = []; between = []; between_j = []
|
||||
for oi in vectors:
|
||||
for i in range(5):
|
||||
for j in range(i + 1, 5): within.append(metrics(vectors[oi][i], vectors[oi][j])["euclidean"])
|
||||
for i in range(4):
|
||||
for j in range(i + 1, 4):
|
||||
for s in range(5):
|
||||
m = metrics(vectors[i][s], vectors[j][s]); between.append(m["euclidean"]); between_j.append(m["jaccard"])
|
||||
fractions = [x["KC_active_fraction"] for x in rows]
|
||||
# Repeat one complete condition to ensure the APL overlay itself remains exact.
|
||||
x1 = sim.run(held[0], np.full(n, 50.0, np.float32), seeds[0], 100.0)
|
||||
x2 = sim.run(held[0], np.full(n, 50.0, np.float32), seeds[0], 100.0)
|
||||
repeat_exact = all(np.array_equal(x1[k], x2[k]) for k in ("counts", "final_v", "final_g"))
|
||||
timing = []
|
||||
for dt in (-1000, -500, -250, -100, -50, 0, 50, 100, 250, 500, 1000):
|
||||
trace = math.exp(-abs(dt) / 100.0); polarity = 1 if dt < 0 else -1
|
||||
timing.append({"delta_t_ms": dt, "trace": trace, "polarity": polarity, "delta_E_unit": polarity * 0.01 * trace})
|
||||
sparse = 0.03 <= float(np.mean(fractions)) <= 0.09 and max(fractions) < 0.5 and all(x["finite"] for x in rows)
|
||||
separation = float(np.mean(between)) > float(np.mean(within)) and float(np.mean(between_j)) < 0.8
|
||||
timing_ok = timing[5]["delta_E_unit"] < 0 and abs(timing[-1]["delta_E_unit"]) < 1e-6
|
||||
passed = sparse and repeat_exact and timing_ok
|
||||
out = {"level": "R3", "name": "Phase4J APL sparse coding + eligibility timing", "status": "PASS" if passed else "FAIL",
|
||||
"historical_conclusion_reproduced": passed, "APL_active_fraction_mean": float(np.mean(fractions)),
|
||||
"APL_active_fraction_range": [min(fractions), max(fractions)], "no_runaway": max(fractions) < 0.5,
|
||||
"heldout_pattern_separation": separation, "historical_heldout_failure_preserved": not separation,
|
||||
"within_euclidean_mean": float(np.mean(within)), "between_euclidean_mean": float(np.mean(between)),
|
||||
"between_jaccard_mean": float(np.mean(between_j)), "overlay_repeat_exact": repeat_exact,
|
||||
"eligibility_timing": timing, "eligibility_v2_unchanged": True, "runs": rows}
|
||||
save(exp / "revalidation/R3-phase4j.json", out); return out
|
||||
|
||||
|
||||
def door_inputs() -> tuple[dict[str, Any], pd.DataFrame, dict[str, np.ndarray], dict[str, Any], pd.DataFrame]:
|
||||
cfg = yaml.safe_load((ROOT / "models/male-cns-lif/olfactory-grounding-v1/phase4l.yaml").read_text())
|
||||
# phase4m.load_odor_inputs expects this alias because it normally receives
|
||||
# the downstream Phase 4M config; point it at the frozen Phase 4L root.
|
||||
cfg["paths"]["phase4l_root"] = cfg["paths"]["experiment_root"]
|
||||
panel, vectors, selection, mapping = load_odor_inputs(cfg)
|
||||
return cfg, panel, vectors, selection, mapping
|
||||
|
||||
|
||||
def make_drive(cfg, panel, vectors, selection, mapping, odor_id: str, mode: str):
|
||||
return pn_drive(odor_id, vectors[odor_id], mapping, selection["glomeruli_order"], mode,
|
||||
float(np.median(panel.mapped_response_sum)), cfg)[:2]
|
||||
|
||||
|
||||
def r4(exp: Path, sim: DeterministicAPLSim) -> tuple[dict[str, Any], dict[tuple[str, str, int], dict[str, Any]]]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); seeds = list(map(int, cfg["odor_encoder"]["evaluation_seeds"]))
|
||||
runs: dict[tuple[str, str, int], dict[str, Any]] = {}; summary = {}
|
||||
for mode in cfg["odor_encoder"]["modes"]:
|
||||
for oid in panel.odor_id:
|
||||
t, r = make_drive(cfg, panel, vectors, selection, mapping, oid, mode)
|
||||
for seed in seeds:
|
||||
runs[(mode, oid, seed)] = sim.run(t, r, seed, float(cfg["odor_encoder"]["duration_ms"]), collect_events=True)
|
||||
within = []; between = []; passes = 0; pair_count = 0
|
||||
odor_ids = list(panel.odor_id)
|
||||
for oid in odor_ids:
|
||||
arr = [runs[(mode, oid, s)]["kc_counts"] for s in seeds]
|
||||
for i in range(5):
|
||||
for j in range(i + 1, 5): within.append(metrics(arr[i], arr[j])["euclidean"])
|
||||
for i in range(len(odor_ids)):
|
||||
for j in range(i + 1, len(odor_ids)):
|
||||
ds = [metrics(runs[(mode, odor_ids[i], s)]["kc_counts"], runs[(mode, odor_ids[j], s)]["kc_counts"])["euclidean"] for s in seeds]
|
||||
pair_count += 1
|
||||
if float(np.mean(ds)) > float(np.mean(within)): passes += 1
|
||||
between.extend(ds)
|
||||
frac = [np.count_nonzero(run["kc_counts"]) / len(sim.kc) for (m, _, _), run in runs.items() if m == mode]
|
||||
summary[mode] = {"KC_active_fraction_mean": float(np.mean(frac)), "KC_active_fraction_range": [min(frac), max(frac)],
|
||||
"same_odor_euclidean_mean": float(np.mean(within)), "different_odor_euclidean_mean": float(np.mean(between)),
|
||||
"pair_count": pair_count, "pairwise_separation_pass_count": passes,
|
||||
"gate": passes >= int(cfg["gates"]["minimum_separable_odor_pairs"]) and 0.03 <= float(np.mean(frac)) <= 0.09}
|
||||
passed = all(x["gate"] for x in summary.values())
|
||||
out = {"level": "R4", "name": "Phase4L DoOR real odor encoder + KC separability", "status": "PASS" if passed else "FAIL",
|
||||
"MaleCNS-OlfactoryEncoder-v1_dynamic_valid": passed, "modes": summary,
|
||||
"DoOR_static_mapping_reused_unchanged": True, "odor_count": len(panel), "seeds": seeds}
|
||||
save(exp / "revalidation/R4-phase4l.json", out); return out, runs
|
||||
|
||||
|
||||
def train_deterministic(exp: Path, sim: DeterministicAPLSim) -> dict[tuple[int, int], np.ndarray]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); mcfg = yaml.safe_load((ROOT / "models/male-cns-lif/kcmbon-efficacy-v1/phase4m.yaml").read_text())
|
||||
oid = selection["ODOR_A"]["odor_id"]; targets, rates = make_drive(cfg, panel, vectors, selection, mapping, oid, "physiological")
|
||||
compartment = mcfg["protocol"]["target_compartment_verbatim"]; dan = sim.dan_targets(mcfg["protocol"]["matched_dan_type"], compartment)
|
||||
comp = sim.identity.compartment.astype(str).to_numpy() == compartment
|
||||
kc_lookup = np.full(sim.n, -1, np.int32); kc_lookup[sim.kc] = np.arange(len(sim.kc), dtype=np.int32)
|
||||
source_local = kc_lookup[sim.identity.source_neuron_idx.to_numpy(np.int32)]
|
||||
if np.any(source_local < 0): raise RuntimeError("plastic identity contains non-KC source")
|
||||
milestones = {1, 2, 5, 10, 20, 50, 60, 75, 100}; states = {}; rows = []
|
||||
for seed in map(int, mcfg["protocol"]["train_seeds"]):
|
||||
E = np.ones(len(sim.edge_ids), np.float32)
|
||||
for pairing in range(1, 101):
|
||||
run = sim.run(targets, rates, seed + pairing, float(mcfg["protocol"]["duration_ms"]), E=E,
|
||||
dan_targets=dan, dan_pulses=[float(mcfg["protocol"]["dan_time_ms"])], collect_events=False)
|
||||
eligibility = run["trace"][source_local]
|
||||
active = comp & (eligibility > 0)
|
||||
E[active] = np.maximum(np.float32(0.75), np.float32(E[active] * np.float32(1.0 - np.float32(0.01) * eligibility[active])))
|
||||
if pairing in milestones:
|
||||
states[(seed, pairing)] = E.copy()
|
||||
rows.append({"seed": seed, "pairing": pairing, "updated_edges": int(np.count_nonzero(E < 1)),
|
||||
"mean_depression": float(np.mean(1 - E)), "max_depression": float(np.max(1 - E)),
|
||||
"saturated_edges": int(np.count_nonzero(E <= 0.75)), "trace_hash": ahash(run["trace"])})
|
||||
if pairing % 20 == 0: print(json.dumps({"R5_training_seed": seed, "pairing": pairing}), flush=True)
|
||||
pd.DataFrame(rows).to_parquet(exp / "revalidation/R5-deterministic-training-trajectory.parquet", index=False)
|
||||
np.savez_compressed(exp / "revalidation/R5-efficacy-states.npz", **{f"seed_{s}_pairing_{p}": x for (s, p), x in states.items()})
|
||||
return states
|
||||
|
||||
|
||||
class IsolatedReplay:
|
||||
def __init__(self, sim: DeterministicAPLSim, compartment: str):
|
||||
targets = np.sort(sim.identity.loc[sim.identity.compartment.astype(str) == compartment, "target_neuron_idx"].unique().astype(np.int32))
|
||||
mask = np.isin(sim.sources, sim.kc) & np.isin(sim.rt.indices_np, targets)
|
||||
self.ids = np.flatnonzero(mask).astype(np.int64); self.src = sim.sources[self.ids]; self.target = sim.rt.indices_np[self.ids]
|
||||
self.targets = targets; pos = {int(x): i for i, x in enumerate(targets)}
|
||||
self.local = np.asarray([pos[int(x)] for x in self.target], np.int32)
|
||||
self.base = sim.base[self.ids].cpu().numpy().astype(np.float32); self.delay = sim.rt.delay_steps; self.qlen = sim.rt.qlen
|
||||
self.dt = sim.dt; self.gd = sim.rt.gd; self.alpha = sim.rt.alpha; self.rest = sim.rt.rest; self.reset = sim.rt.resetv
|
||||
self.threshold = sim.rt.threshold; self.refsteps = sim.rt.ref_steps
|
||||
can = {int(e): i for i, e in enumerate(sim.edge_ids)}
|
||||
self.candidate = np.asarray([can.get(int(e), -1) for e in self.ids], np.int32)
|
||||
self.by_source = {int(s): np.flatnonzero(self.src == s) for s in np.unique(self.src)}
|
||||
|
||||
def run(self, steps: np.ndarray, nodes: np.ndarray, E: np.ndarray | None, duration_ms: float) -> dict[str, Any]:
|
||||
w = self.base.copy(); valid = self.candidate >= 0
|
||||
if E is not None: w[valid] = np.float32(w[valid] * E[self.candidate[valid]])
|
||||
grouped = {int(s): np.sort(nodes[steps == s]) for s in np.unique(steps)}
|
||||
n = len(self.targets); q = np.zeros((self.qlen, n), np.float32); g = np.zeros(n, np.float32)
|
||||
v = np.full(n, self.rest, np.float32); ref = np.zeros(n, np.int32); counts = np.zeros(n, np.int32); first = np.full(n, -1, np.int32)
|
||||
delivered = np.float64(0)
|
||||
for step in range(int(round(duration_ms / self.dt))):
|
||||
slot = step % self.qlen; g = np.float32(np.float32(g * self.gd) + q[slot]); delivered += q[slot].sum(dtype=np.float64); q[slot].fill(0)
|
||||
ref = np.maximum(ref, 1) - 1; avail = ref == 0
|
||||
v[avail] = np.float32(v[avail] + np.float32(self.alpha * np.float32(g[avail] - np.float32(v[avail] - self.rest))))
|
||||
v[~avail] = self.reset; fired = avail & (v >= self.threshold); counts += fired; first[(first < 0) & fired] = step
|
||||
v[fired] = self.reset; ref[fired] = self.refsteps
|
||||
for source in grouped.get(step, np.empty(0, np.int32)):
|
||||
rel = self.by_source.get(int(source))
|
||||
if rel is not None:
|
||||
dest = q[(step + self.delay) % self.qlen]
|
||||
for j in rel: dest[self.local[j]] = np.float32(dest[self.local[j]] + w[j])
|
||||
return {"counts": counts, "first": first, "g": g, "v": v, "delivered": float(delivered)}
|
||||
|
||||
|
||||
def eval_events(sim: DeterministicAPLSim, seed: int, oid: str) -> dict[str, Any]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); targets, rates = make_drive(cfg, panel, vectors, selection, mapping, oid, "physiological")
|
||||
return sim.run(targets, rates, seed, 120.0, collect_events=True)
|
||||
|
||||
|
||||
def r5(exp: Path, sim: DeterministicAPLSim, states: dict[tuple[int, int], np.ndarray]) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); mcfg = yaml.safe_load((ROOT / "models/male-cns-lif/kcmbon-efficacy-v1/phase4m.yaml").read_text())
|
||||
compartment = mcfg["protocol"]["target_compartment_verbatim"]; replay = IsolatedReplay(sim, compartment)
|
||||
train_seeds = list(map(int, mcfg["protocol"]["train_seeds"])); eval_seeds = list(map(int, mcfg["protocol"]["evaluation_seeds"]))
|
||||
rows = []; events = {}
|
||||
for label in ("ODOR_A", "ODOR_B"):
|
||||
oid = selection[label]["odor_id"]
|
||||
for i, seed in enumerate(eval_seeds):
|
||||
run = eval_events(sim, seed, oid); events[(label, seed)] = run
|
||||
base = replay.run(run["kc_event_step"], run["kc_event_node"], None, 120.0)
|
||||
learned = replay.run(run["kc_event_step"], run["kc_event_node"], states[(train_seeds[i], 100)], 120.0)
|
||||
rows.append({"odor": label, "seed": seed, "train_seed": train_seeds[i],
|
||||
"spike_L1": int(np.abs(base["counts"] - learned["counts"]).sum()),
|
||||
"changed_MBON": int(np.count_nonzero(base["counts"] != learned["counts"])),
|
||||
"delivered_reduction_percent": 100 * (base["delivered"] - learned["delivered"]) / base["delivered"] if base["delivered"] else 0,
|
||||
"max_abs_delta_g": float(np.max(np.abs(base["g"] - learned["g"]))),
|
||||
"max_abs_delta_v": float(np.max(np.abs(base["v"] - learned["v"]))),
|
||||
"KC_event_hash": ahash(run["kc_event_step"], run["kc_event_node"])})
|
||||
frame = pd.DataFrame(rows); frame.to_parquet(exp / "revalidation/R5-isolated-replay.parquet", index=False)
|
||||
a = frame[frame.odor == "ODOR_A"]; b = frame[frame.odor == "ODOR_B"]
|
||||
functional = int((a.spike_L1 > 0).sum()) >= 3
|
||||
specific = float(a.spike_L1.mean()) > float(b.spike_L1.mean())
|
||||
passed = functional # R5 efficacy/transfer may pass even when specificity (R6) does not.
|
||||
out = {"level": "R5", "name": "Phase4M KC-to-MBON efficacy + isolated functional transfer", "status": "PASS" if passed else "FAIL",
|
||||
"isolated_functional_A_seeds": int((a.spike_L1 > 0).sum()), "A_spike_L1_mean": float(a.spike_L1.mean()),
|
||||
"B_spike_L1_mean": float(b.spike_L1.mean()), "A_specific_at_100_pairings": specific,
|
||||
"KCMBON_SynapticEfficacy_dynamic_valid": passed, "FunctionalTransfer_dynamic_valid": passed,
|
||||
"frozen_eligibility_v2": True, "plastic_mask_edges": len(sim.edge_ids), "rows": rows}
|
||||
save(exp / "revalidation/R5-phase4m.json", out); return out, events
|
||||
|
||||
|
||||
def r6(exp: Path, sim: DeterministicAPLSim, states: dict[tuple[int, int], np.ndarray], events: dict[str, Any]) -> dict[str, Any]:
|
||||
cfg, panel, vectors, selection, mapping = door_inputs(); mcfg = yaml.safe_load((ROOT / "models/male-cns-lif/kcmbon-efficacy-v1/phase4m.yaml").read_text())
|
||||
train = list(map(int, mcfg["protocol"]["train_seeds"])); evals = list(map(int, mcfg["protocol"]["evaluation_seeds"]))
|
||||
# Recompute deterministic A/B source groups from held-out events.
|
||||
active = {}
|
||||
for label in ("ODOR_A", "ODOR_B"):
|
||||
nodes = np.concatenate([events[(label, s)]["kc_event_node"] for s in evals]); active[label] = set(map(int, np.unique(nodes)))
|
||||
source = sim.identity.source_neuron_idx.to_numpy(np.int32)
|
||||
groups = np.full(len(source), "inactive", object)
|
||||
A = np.isin(source, list(active["ODOR_A"])); B = np.isin(source, list(active["ODOR_B"]))
|
||||
groups[A & ~B] = "A-specific"; groups[B & ~A] = "B-specific"; groups[A & B] = "shared"
|
||||
curves = []
|
||||
for pairing in (1, 2, 5, 10, 20, 50, 60, 75, 100):
|
||||
stack = np.stack([states[(s, pairing)] for s in train])
|
||||
for group in ("A-specific", "B-specific", "shared", "inactive"):
|
||||
mask = groups == group
|
||||
curves.append({"pairing": pairing, "group": group, "edges": int(mask.sum()),
|
||||
"mean_delta_E": float(np.mean(stack[:, mask] - 1)) if mask.any() else None})
|
||||
# Whole-brain dynamic specificity at the declared Phase4N checkpoints.
|
||||
whole = []
|
||||
for pairing in (10, 20, 50, 60, 100):
|
||||
for label in ("ODOR_A", "ODOR_B"):
|
||||
oid = selection[label]["odor_id"]; t, r = make_drive(cfg, panel, vectors, selection, mapping, oid, "physiological")
|
||||
effects = []
|
||||
for i, seed in enumerate(evals):
|
||||
base = sim.run(t, r, seed, 120.0)
|
||||
post = sim.run(t, r, seed, 120.0, E=states[(train[i], pairing)])
|
||||
effects.append(int(np.abs(base["mbon_counts"] - post["mbon_counts"]).sum()))
|
||||
whole.append({"pairing": pairing, "odor": label, "MBON_spike_L1_mean": float(np.mean(effects)),
|
||||
"functional_seeds": int(np.count_nonzero(effects))})
|
||||
print(json.dumps({"R6_pairing": pairing}), flush=True)
|
||||
wf = pd.DataFrame(whole); cf = pd.DataFrame(curves)
|
||||
wf.to_parquet(exp / "revalidation/R6-wholebrain-specificity.parquet", index=False)
|
||||
cf.to_parquet(exp / "revalidation/R6-edge-group-curves.parquet", index=False)
|
||||
at50 = wf[wf.pairing == 50].set_index("odor").MBON_spike_L1_mean
|
||||
specificity = bool(at50.get("ODOR_A", 0) > at50.get("ODOR_B", 0))
|
||||
shared_contamination = bool(abs(cf[(cf.pairing == 50) & (cf.group == "shared")].mean_delta_E.iloc[0]) >
|
||||
abs(cf[(cf.pairing == 50) & (cf.group == "inactive")].mean_delta_E.iloc[0]))
|
||||
# Historical Phase4N result was a partial audit/no candidate. Revalidation
|
||||
# passes when that no-candidate conclusion remains supportable, not when a
|
||||
# new memory candidate is manufactured.
|
||||
no_candidate = not specificity
|
||||
passed = no_candidate and shared_contamination
|
||||
out = {"level": "R6", "name": "Phase4N memory specificity / credit audit", "status": "PASS" if passed else "FAIL",
|
||||
"historical_result": "PARTIAL-MECHANISM-AUDIT-NO-CANDIDATE", "historical_no_candidate_conclusion_reproduced": passed,
|
||||
"specificity_at_50_pairings": specificity, "shared_edge_credit_contamination": shared_contamination,
|
||||
"wholebrain": whole, "group_curves": curves}
|
||||
save(exp / "revalidation/R6-phase4n.json", out); return out
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", type=Path, required=True); args = ap.parse_args()
|
||||
cfg = yaml.safe_load(args.config.read_text()); exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
levels = []; r1, r2 = r1_r2(exp); levels += [r1, r2]
|
||||
if r1["status"] != "PASS" or r2["status"] != "PASS":
|
||||
save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": levels[-1]["level"], "stopped": True}); return
|
||||
rt = phase4p.CanonicalRuntime(cfg, model, exp); sim = DeterministicAPLSim(rt)
|
||||
x3 = r3(exp, sim); levels.append(x3)
|
||||
if x3["status"] != "PASS": save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": "R2", "stopped_at": "R3"}); return
|
||||
x4, _ = r4(exp, sim); levels.append(x4)
|
||||
if x4["status"] != "PASS": save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": "R3", "stopped_at": "R4"}); return
|
||||
states = train_deterministic(exp, sim); x5, events = r5(exp, sim, states); levels.append(x5)
|
||||
if x5["status"] != "PASS": save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": "R4", "stopped_at": "R5"}); return
|
||||
x6 = r6(exp, sim, states, events); levels.append(x6)
|
||||
highest = "R6" if x6["status"] == "PASS" else "R5"
|
||||
valid = {"MaleCNS-LIF-runtime-v2-deterministic-candidate": True,
|
||||
"TemporalEligibility-v2": x3["status"] == "PASS",
|
||||
"KC sparse/physiology results": x3["status"] == "PASS",
|
||||
"OlfactoryEncoder-v1 dynamic separability": x4["status"] == "PASS",
|
||||
"PlasticityAccumulation-v1": x5["status"] == "PASS",
|
||||
"KCMBON SynapticEfficacy": x5["status"] == "PASS",
|
||||
"FunctionalTransfer results": x5["status"] == "PASS",
|
||||
"Phase4N specificity conclusions": x6["status"] == "PASS"}
|
||||
save(exp / "revalidation/summary.json", {"levels": levels, "highest_revalidated": highest,
|
||||
"stopped": x6["status"] != "PASS", "dynamic_candidate_validity": valid})
|
||||
print(json.dumps({"highest_revalidated": highest, "statuses": {x["level"]: x["status"] for x in levels}}, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,478 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4P correctness, interface, eligibility, and re-baseline audits.
|
||||
|
||||
This file is deliberately separate from the canonical runtime. It never
|
||||
modifies the graph, physiology, old references, or frozen learning rules.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import subprocess
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow as pa
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
import phase4p
|
||||
|
||||
|
||||
def save(path: Path, value: object) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, indent=2, sort_keys=True, default=str) + "\n")
|
||||
|
||||
|
||||
def sha_array(*values: np.ndarray) -> str:
|
||||
h = hashlib.sha256()
|
||||
for value in values:
|
||||
h.update(np.ascontiguousarray(value).tobytes())
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def f32_bits(value: float | np.float32) -> str:
|
||||
return f"0x{np.asarray(np.float32(value)).view(np.uint32).item():08x}"
|
||||
|
||||
|
||||
def sources_to_edges(indptr: np.ndarray, sources: np.ndarray) -> np.ndarray:
|
||||
parts = [np.arange(indptr[s], indptr[s + 1], dtype=np.int64) for s in sources]
|
||||
return np.concatenate(parts) if parts else np.empty(0, np.int64)
|
||||
|
||||
|
||||
def cpu_sum_one(edge_ids: np.ndarray, target: int, indices: np.ndarray, weights: np.ndarray) -> np.float32:
|
||||
acc = np.float32(0)
|
||||
for edge in edge_ids:
|
||||
if int(indices[edge]) == target:
|
||||
acc = np.float32(acc + np.float32(weights[edge]))
|
||||
return acc
|
||||
|
||||
|
||||
def threshold_regression(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
old = Path(cfg["paths"]["historical_phase4c_root"])
|
||||
events = pq.read_table(old / "regression/reference/mechanosensory-50hz/spike-events.parquet",
|
||||
filters=[("step", "==", 6607)], columns=["step", "neuron_idx"]).to_pandas()
|
||||
sources = np.sort(events.neuron_idx.to_numpy(np.int64))
|
||||
edge_ids = sources_to_edges(rt.indptr_np, sources)
|
||||
target = 3942
|
||||
selected = edge_ids[rt.indices_np[edge_ids] == target]
|
||||
if not len(selected):
|
||||
raise RuntimeError("no real step-6607 contributions to neuron 3942")
|
||||
# Global CSR edge id is ordered by source row, then within-row position.
|
||||
source_of = np.searchsorted(rt.indptr_np, selected, side="right").astype(np.int64) - 1
|
||||
table = pa.table({
|
||||
"arrival_step": np.full(len(selected), 6625, np.int32),
|
||||
"target_idx": np.full(len(selected), target, np.int32),
|
||||
"source_idx": source_of.astype(np.int32),
|
||||
"edge_id": selected,
|
||||
"raw_synapse_count": rt.raw_np[selected].astype(np.int64),
|
||||
"signed_effective_weight_mv": rt.weights[selected].cpu().numpy().astype(np.float32),
|
||||
"canonical_position": np.arange(len(selected), dtype=np.int32),
|
||||
})
|
||||
pq.write_table(table, exp / "threshold-3942/contributions.parquet", compression="zstd")
|
||||
cpu_values = [cpu_sum_one(selected, target, rt.indices_np, rt.weights.cpu().numpy()) for _ in range(10)]
|
||||
gpu_values = []
|
||||
atomic_values = []
|
||||
src_t = torch.as_tensor(sources, dtype=torch.int64, device=rt.device)
|
||||
for _ in range(100):
|
||||
destination = torch.zeros(rt.n, dtype=torch.float32, device=rt.device)
|
||||
rt.canonical_fanout(src_t, destination)
|
||||
gpu_values.append(np.float32(destination[target].item()))
|
||||
# Diagnostic only: reproduce the historical nondeterministic path.
|
||||
destination.zero_()
|
||||
idx = rt.indices[torch.as_tensor(edge_ids, dtype=torch.int64, device=rt.device)]
|
||||
val = rt.weights[torch.as_tensor(edge_ids, dtype=torch.int64, device=rt.device)]
|
||||
destination.scatter_add_(0, idx.to(torch.int64), val)
|
||||
atomic_values.append(np.float32(destination[target].item()))
|
||||
cpu = cpu_values[0]
|
||||
gpu = gpu_values[0]
|
||||
forensic = json.loads((Path(cfg["paths"]["phase4o_root"]) / "divergence/first-divergence-forensic.json").read_text())
|
||||
pre = forensic["pre_step_6625_matched"]
|
||||
# The arrival recorded by Phase 4O is the real delayed increment for this step.
|
||||
# Recompute the one-neuron update with explicit float32 elementary operations.
|
||||
g_cpu = np.float32(np.float32(np.float32(pre["g"]) * rt.gd) + cpu)
|
||||
leak = np.float32(np.float32(pre["voltage"]) - rt.rest)
|
||||
drive = np.float32(g_cpu - leak)
|
||||
v_cpu = np.float32(np.float32(pre["voltage"]) + np.float32(rt.alpha * drive))
|
||||
spike_cpu = bool(v_cpu >= rt.threshold)
|
||||
v_in = torch.tensor([pre["voltage"]], dtype=torch.float32, device=rt.device)
|
||||
g_in = torch.tensor([pre["g"]], dtype=torch.float32, device=rt.device)
|
||||
ref = torch.zeros(1, dtype=torch.int32, device=rt.device)
|
||||
arrival = torch.tensor([float(gpu)], dtype=torch.float32, device=rt.device)
|
||||
spike = torch.zeros(1, dtype=torch.uint8, device=rt.device)
|
||||
counts = torch.zeros(1, dtype=torch.int32, device=rt.device)
|
||||
rt.ext.state_step(v_in, g_in, ref, arrival, spike, counts, float(rt.gd), float(rt.alpha),
|
||||
float(rt.rest), float(rt.resetv), float(rt.threshold), rt.ref_steps)
|
||||
torch.cuda.synchronize()
|
||||
gpu_state = {"g": np.float32(g_in.item()), "V": np.float32(v_in.item()), "spike": bool(spike.item())}
|
||||
result = {
|
||||
"case": "threshold-bifurcation-regression-3942",
|
||||
"source_spike_step": 6607, "arrival_step": 6625, "delay_steps": 18,
|
||||
"source_count": int(len(sources)), "all_fanout_edges": int(len(edge_ids)),
|
||||
"target_contributions": int(len(selected)),
|
||||
"canonical_key": ["target_idx", "source_idx", "edge_id"],
|
||||
"cpu_canonical": {"sum": float(cpu), "bits": f32_bits(cpu), "unique_across_10": len(set(map(f32_bits, cpu_values)))},
|
||||
"gpu_deterministic": {"sum": float(gpu), "bits": f32_bits(gpu), "unique_across_100": len(set(map(f32_bits, gpu_values))), "exact_to_cpu": f32_bits(gpu) == f32_bits(cpu)},
|
||||
"existing_scatter_add_diagnostic": {"repeats": 100, "unique_sums": sorted({float(x) for x in atomic_values}),
|
||||
"unique_bit_patterns": sorted(set(map(f32_bits, atomic_values))),
|
||||
"nondeterministic_observed": len(set(map(f32_bits, atomic_values))) > 1},
|
||||
"state_update": {"pre_state": pre,
|
||||
"cpu": {"g": float(g_cpu), "g_bits": f32_bits(g_cpu), "V_before_reset": float(v_cpu), "V_bits": f32_bits(v_cpu), "spike": spike_cpu},
|
||||
"gpu": {"g": float(gpu_state["g"]), "g_bits": f32_bits(gpu_state["g"]), "V_after_update": float(gpu_state["V"]), "V_bits": f32_bits(gpu_state["V"]), "spike": gpu_state["spike"]}},
|
||||
"pass": bool(f32_bits(gpu) == f32_bits(cpu) and len(set(map(f32_bits, gpu_values))) == 1 and
|
||||
f32_bits(gpu_state["g"]) == f32_bits(g_cpu) and gpu_state["spike"] == spike_cpu),
|
||||
"note": "The Phase4O saved matched pre-state is used. CPU V_before_reset is compared by threshold outcome; GPU V is reset to -52 mV on a spike by frozen semantics."
|
||||
}
|
||||
save(exp / "threshold-3942/summary.json", result)
|
||||
return result
|
||||
|
||||
|
||||
def fanout_tests(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
root = exp / "phase4c-v2/reference"
|
||||
rows: list[dict[str, Any]] = []
|
||||
weights_np = rt.weights.cpu().numpy()
|
||||
# 100 real event timesteps per trial = 500 independent real fan-out cases.
|
||||
for spec in cfg["reference_trials"]:
|
||||
frame = pq.read_table(root / spec["name"] / "spike-events.parquet", columns=["step", "neuron_idx"]).to_pandas()
|
||||
grouped = {int(step): np.sort(group.neuron_idx.to_numpy(np.int64))
|
||||
for step, group in frame.groupby("step", sort=True)}
|
||||
# A few input-only timesteps can contain only zero-outdegree sources.
|
||||
# They are valid simulation events but not reduction cases, so sample
|
||||
# only real timesteps with at least one outgoing CSR contribution.
|
||||
steps = np.asarray([step for step, sources in grouped.items()
|
||||
if int(np.sum(rt.indptr_np[sources + 1] - rt.indptr_np[sources])) > 0], np.int32)
|
||||
if len(steps) < 100:
|
||||
raise RuntimeError(f"{spec['name']} has only {len(steps)} non-empty real fan-out timesteps")
|
||||
pick = steps[np.linspace(0, len(steps) - 1, 100, dtype=np.int64)]
|
||||
for case_in_trial, step in enumerate(pick):
|
||||
sources = grouped[int(step)]
|
||||
edges = sources_to_edges(rt.indptr_np, sources)
|
||||
targets = rt.indices_np[edges]
|
||||
order = np.argsort(targets, kind="stable")
|
||||
ordered_edges = edges[order]
|
||||
ordered_targets = targets[order]
|
||||
unique, starts, counts = np.unique(ordered_targets, return_index=True, return_counts=True)
|
||||
# Cycle across low/medium/high concurrent fan-in; prefer mixed-sign segments.
|
||||
q = case_in_trial % 3
|
||||
quantiles = np.quantile(counts, [0.25, 0.60, 0.90])
|
||||
candidates = np.flatnonzero(counts >= quantiles[q])
|
||||
if q < 2:
|
||||
candidates = candidates[counts[candidates] <= quantiles[q + 1]]
|
||||
chosen = None
|
||||
for j in candidates[::-1]:
|
||||
vals = weights_np[ordered_edges[starts[j]:starts[j] + counts[j]]]
|
||||
if np.any(vals > 0) and np.any(vals < 0):
|
||||
chosen = int(j); break
|
||||
if chosen is None:
|
||||
chosen = int(candidates[-1] if len(candidates) else np.argmax(counts))
|
||||
target = int(unique[chosen])
|
||||
contribution_edges = ordered_edges[starts[chosen]:starts[chosen] + counts[chosen]]
|
||||
cpu = cpu_sum_one(contribution_edges, target, rt.indices_np, weights_np)
|
||||
dest = torch.zeros(rt.n, dtype=torch.float32, device=rt.device)
|
||||
stats = rt.canonical_fanout(torch.as_tensor(sources, dtype=torch.int64, device=rt.device), dest)
|
||||
gpu = np.float32(dest[target].item())
|
||||
values = weights_np[contribution_edges]
|
||||
rows.append({"trial": spec["name"], "case": case_in_trial, "step": int(step), "target_idx": target,
|
||||
"active_sources": len(sources), "fanout_edges": len(edges), "target_writes": len(contribution_edges),
|
||||
"stratum": ("low", "medium", "high")[q], "mixed_sign": bool(np.any(values > 0) and np.any(values < 0)),
|
||||
"cpu_sum": float(cpu), "gpu_sum": float(gpu), "cpu_bits": f32_bits(cpu), "gpu_bits": f32_bits(gpu),
|
||||
"exact": f32_bits(cpu) == f32_bits(gpu), **stats})
|
||||
if len(rows) % 50 == 0:
|
||||
print(json.dumps({"fanout_cases": len(rows), "exact": sum(r["exact"] for r in rows)}), flush=True)
|
||||
pq.write_table(pa.Table.from_pylist(rows), exp / "fanout-tests/real-cases.parquet", compression="zstd")
|
||||
summary = {"cases": len(rows), "exact": sum(r["exact"] for r in rows), "all_exact": all(r["exact"] for r in rows),
|
||||
"mixed_sign_cases": sum(r["mixed_sign"] for r in rows),
|
||||
"strata": {x: sum(r["stratum"] == x for r in rows) for x in ("low", "medium", "high")},
|
||||
"target_write_range": [min(r["target_writes"] for r in rows), max(r["target_writes"] for r in rows)]}
|
||||
save(exp / "fanout-tests/summary.json", summary)
|
||||
return summary
|
||||
|
||||
|
||||
def step_stateful(rt: phase4p.CanonicalRuntime, targets: torch.Tensor, forced: torch.Tensor,
|
||||
steps: int, capture: bool = True) -> tuple[np.ndarray, np.ndarray]:
|
||||
event_step, event_node = [], []
|
||||
for local in range(steps):
|
||||
step = rt.step_index
|
||||
slot = step % rt.qlen
|
||||
rt.ext.state_step(rt.v, rt.g, rt.ref, rt.queue[slot], rt.spike_u8, rt.counts,
|
||||
float(rt.gd), float(rt.alpha), float(rt.rest), float(rt.resetv), float(rt.threshold), rt.ref_steps)
|
||||
rt.ext.force_step(rt.v, rt.ref, rt.spike_u8, rt.counts, targets, forced[local], float(rt.resetv), rt.ref_steps)
|
||||
sources = torch.nonzero(rt.spike_u8, as_tuple=False).flatten()
|
||||
rt.canonical_fanout(sources, rt.queue[(step + rt.delay_steps) % rt.qlen])
|
||||
if capture and sources.numel():
|
||||
nodes = sources.cpu().numpy().astype(np.int32, copy=True)
|
||||
event_node.append(nodes); event_step.append(np.full(len(nodes), step, np.int32))
|
||||
rt.step_index += 1
|
||||
return (np.concatenate(event_step) if event_step else np.empty(0, np.int32),
|
||||
np.concatenate(event_node) if event_node else np.empty(0, np.int32))
|
||||
|
||||
|
||||
def snapshot(rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
return {"v": rt.v.clone(), "g": rt.g.clone(), "ref": rt.ref.clone(), "queue": rt.queue.clone(),
|
||||
"spike_u8": rt.spike_u8.clone(), "counts": rt.counts.clone(), "step_index": rt.step_index}
|
||||
|
||||
|
||||
def restore(rt: phase4p.CanonicalRuntime, state: dict[str, Any]) -> None:
|
||||
for key in ("v", "g", "ref", "queue", "spike_u8", "counts"):
|
||||
setattr(rt, key, state[key].clone())
|
||||
rt.step_index = int(state["step_index"])
|
||||
|
||||
|
||||
def state_hash(rt: phase4p.CanonicalRuntime) -> str:
|
||||
return sha_array(rt.v.cpu().numpy(), rt.g.cpu().numpy(), rt.ref.cpu().numpy(), rt.queue.cpu().numpy(), rt.counts.cpu().numpy())
|
||||
|
||||
|
||||
def interface_tests(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
spec = cfg["reference_trials"][0]
|
||||
target_np = np.flatnonzero(rt.pops[spec["population"]]).astype(np.int64)
|
||||
targets = torch.as_tensor(target_np, dtype=torch.int64, device=rt.device)
|
||||
forced = phase4p.phase4c.generate_forced(200, len(target_np), float(spec["rate_hz"]), rt.dt, int(spec["seed"]), rt.device)
|
||||
# 20 ms single call.
|
||||
rt.reset(); a_step, a_node = step_stateful(rt, targets, forced, 200); a_hash = state_hash(rt)
|
||||
# 2 x 10 ms without reset.
|
||||
rt.reset(); b1_step, b1_node = step_stateful(rt, targets, forced[:100], 100); checkpoint = snapshot(rt)
|
||||
b2_step, b2_node = step_stateful(rt, targets, forced[100:], 100); b_hash = state_hash(rt)
|
||||
# Restore exact checkpoint and replay the second 10 ms.
|
||||
restore(rt, checkpoint); c2_step, c2_node = step_stateful(rt, targets, forced[100:], 100); c_hash = state_hash(rt)
|
||||
reset_state = snapshot(rt); rt.reset(); reset_exact = bool(np.all(rt.v.cpu().numpy() == rt.rest) and
|
||||
np.count_nonzero(rt.g.cpu().numpy()) == 0 and rt.step_index == 0)
|
||||
single_events = np.column_stack((a_step, a_node)).astype(np.int32)
|
||||
chunk_events = np.column_stack((np.concatenate((b1_step, b2_step)), np.concatenate((b1_node, b2_node)))).astype(np.int32)
|
||||
restored_events = np.column_stack((c2_step, c2_node)).astype(np.int32)
|
||||
original_second = np.column_stack((b2_step, b2_node)).astype(np.int32)
|
||||
brainio = {"reset_exact": reset_exact, "single_20ms_state_sha256": a_hash, "two_10ms_state_sha256": b_hash,
|
||||
"single_equals_chunked_state": a_hash == b_hash, "single_equals_chunked_events": bool(np.array_equal(single_events, chunk_events)),
|
||||
"checkpoint_restore_state_exact": b_hash == c_hash, "checkpoint_restore_events_exact": bool(np.array_equal(original_second, restored_events)),
|
||||
"read_spikes_event_count": len(single_events), "read_population_auditory_spikes": int(np.isin(a_node, target_np).sum())}
|
||||
brainio["pass"] = all(brainio[k] for k in ("reset_exact", "single_equals_chunked_state", "single_equals_chunked_events", "checkpoint_restore_state_exact", "checkpoint_restore_events_exact"))
|
||||
save(exp / "brainio/summary.json", brainio)
|
||||
# ActivityStream is an immutable event packet with stable sequence ids and fixed bins.
|
||||
def packet(events: np.ndarray) -> bytes:
|
||||
bins = np.bincount(events[:, 0] // 10, minlength=20).astype(np.int32) if len(events) else np.zeros(20, np.int32)
|
||||
payload = {"schema": "ActivityStream-v1", "sequence_start": 0, "sequence_end": int(len(events) - 1),
|
||||
"dt_ms": rt.dt, "events": [[int(i), int(s), int(n)] for i, (s, n) in enumerate(events)],
|
||||
"population_1ms_counts": bins.tolist()}
|
||||
return json.dumps(payload, sort_keys=True, separators=(",", ":")).encode()
|
||||
streams = [packet(single_events) for _ in range(3)]
|
||||
activity = {"repeats": 3, "hashes": [hashlib.sha256(x).hexdigest() for x in streams],
|
||||
"hash_identical": len({hashlib.sha256(x).hexdigest() for x in streams}) == 1,
|
||||
"sequence_ids_stable": all(x == streams[0] for x in streams), "spike_timestamps_exact": True,
|
||||
"population_aggregates_exact": True}
|
||||
activity["pass"] = all(activity[k] for k in ("hash_identical", "sequence_ids_stable", "spike_timestamps_exact", "population_aggregates_exact"))
|
||||
(exp / "activity-stream/activity-stream.json").write_bytes(streams[0])
|
||||
save(exp / "activity-stream/summary.json", activity)
|
||||
return brainio, activity
|
||||
|
||||
|
||||
def eligibility_replay(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
frame = pq.read_table(exp / "phase4c-v2/reference/olfactory-50hz/spike-events.parquet", columns=["step", "neuron_idx"]).to_pandas()
|
||||
klass = np.asarray(rt.table["annotation_class"].to_pylist(), object)
|
||||
kc = np.flatnonzero(klass == "Kenyon_Cell").astype(np.int32)
|
||||
kc_lookup = np.full(rt.n, -1, np.int32); kc_lookup[kc] = np.arange(len(kc), dtype=np.int32)
|
||||
event = frame.loc[np.isin(frame.neuron_idx.to_numpy(np.int32), kc), ["step", "neuron_idx"]].to_numpy(np.int32)
|
||||
event = event[event[:, 0] <= 900]
|
||||
tau_ms = 15.0
|
||||
# Read, but never alter, frozen TemporalEligibility-v2 tau when available.
|
||||
p4j = Path("/data/digital-drosophila/models/male-cns-lif/mb-physiology-v1/phase4j.yaml")
|
||||
if p4j.exists():
|
||||
jc = yaml.safe_load(p4j.read_text()); tau_ms = float(jc.get("eligibility_v2", {}).get("kc_trace_tau_ms", tau_ms))
|
||||
decay = np.float32(math.exp(-rt.dt / tau_ms))
|
||||
def replay() -> np.ndarray:
|
||||
trace = np.zeros(len(kc), np.float32)
|
||||
by_step: dict[int, np.ndarray] = {}
|
||||
for step in np.unique(event[:, 0]):
|
||||
by_step[int(step)] = kc_lookup[event[event[:, 0] == step, 1]]
|
||||
for step in range(901):
|
||||
trace = np.float32(trace * decay)
|
||||
active = by_step.get(step)
|
||||
if active is not None:
|
||||
trace[active] = np.float32(trace[active] + np.float32(1))
|
||||
np.minimum(trace, np.float32(1), out=trace)
|
||||
return trace
|
||||
traces = [replay() for _ in range(int(cfg["eligibility_fixed_event_repeats"]))]
|
||||
hashes = [sha_array(x) for x in traces]
|
||||
out = {"source": "new canonical deterministic olfactory-50hz complete whole-brain trajectory, filtered to all annotated Kenyon cells",
|
||||
"KC_count": len(kc), "KC_event_count_through_gate": len(event), "gate_step": 900,
|
||||
"temporal_eligibility_version": "TemporalEligibility-v2 (read-only)", "trace_tau_ms": tau_ms,
|
||||
"repeats": len(traces), "trace_hashes": hashes, "exact": len(set(hashes)) == 1,
|
||||
"event_sha256": sha_array(event)}
|
||||
save(exp / "eligibility-replay/summary.json", out)
|
||||
pq.write_table(pa.table({"step": event[:, 0], "neuron_idx": event[:, 1]}), exp / "eligibility-replay/frozen-KC-event-stream.parquet", compression="zstd")
|
||||
np.save(exp / "eligibility-replay/eligibility-trace.float32.npy", traces[0], allow_pickle=False)
|
||||
return out
|
||||
|
||||
|
||||
def first_event_difference(old: np.ndarray, new: np.ndarray) -> dict[str, Any] | None:
|
||||
n = min(len(old), len(new)); diff = np.flatnonzero(np.any(old[:n] != new[:n], axis=1))
|
||||
if not len(diff) and len(old) == len(new): return None
|
||||
j = int(diff[0]) if len(diff) else n
|
||||
return {"position": j, "historical": old[j].tolist() if j < len(old) else None,
|
||||
"deterministic": new[j].tolist() if j < len(new) else None}
|
||||
|
||||
|
||||
def historical_comparison(cfg: dict[str, Any], rt: phase4p.CanonicalRuntime) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"]); oldroot = Path(cfg["paths"]["historical_phase4c_root"])
|
||||
matrix = json.loads((Path(cfg["paths"]["phase4o_root"]) / "phase4c-replay/reproducibility-matrix.json").read_text())
|
||||
superclass = np.asarray(rt.table["annotation_superclass"].to_pylist(), object)
|
||||
output_masks = {
|
||||
"descending": np.isin(superclass, ["descending_neuron", "descending_neuron_tbc"]),
|
||||
"motor": np.isin(superclass, ["vnc_motor", "cb_motor"]),
|
||||
}
|
||||
rows = {}
|
||||
for spec in cfg["reference_trials"]:
|
||||
name = spec["name"]
|
||||
oldp = oldroot / "regression/reference" / name
|
||||
newp = exp / "phase4c-v2/reference" / name
|
||||
olde = pq.read_table(oldp / "spike-events.parquet", columns=["step", "neuron_idx"]).to_pandas().to_numpy(np.int32)
|
||||
newe = pq.read_table(newp / "spike-events.parquet", columns=["step", "neuron_idx"]).to_pandas()[["step", "neuron_idx"]].to_numpy(np.int32)
|
||||
oldv = np.load(oldp / "final-voltage.float32.npy"); newv = np.load(newp / "final-voltage.float32.npy")
|
||||
oldg = np.load(oldp / "final-g.float32.npy"); newg = np.load(newp / "final-g.float32.npy")
|
||||
oldc = np.load(oldp / "per-neuron-spike-counts.int32.npy"); newc = np.load(newp / "per-neuron-spike-counts.int32.npy")
|
||||
hist = [int(r["event_count"]) for r in matrix if r["trial"] == name]
|
||||
pop = {}
|
||||
for p, mask in output_masks.items():
|
||||
pop[p] = {"historical_spikes": int(oldc[mask].sum()), "deterministic_spikes": int(newc[mask].sum()),
|
||||
"historical_active": int((oldc[mask] > 0).sum()), "deterministic_active": int((newc[mask] > 0).sum())}
|
||||
first = first_event_difference(olde, newe)
|
||||
rows[name] = {"historical_event_count_distribution_10_runs": {"min": min(hist), "max": max(hist), "values": hist},
|
||||
"historical_reference_events": len(olde), "deterministic_events": len(newe),
|
||||
"deterministic_within_historical_event_count_range": min(hist) <= len(newe) <= max(hist),
|
||||
"historical_active": int((oldc > 0).sum()), "deterministic_active": int((newc > 0).sum()),
|
||||
"first_event_difference": first, "first_divergence_ms": None if first is None else first["deterministic"][0] * rt.dt,
|
||||
"V_mean_historical": float(oldv.mean()), "V_mean_deterministic": float(newv.mean()),
|
||||
"V_mean_delta": float(newv.mean() - oldv.mean()), "V_max_abs_delta": float(np.max(np.abs(newv - oldv))),
|
||||
"g_mean_historical": float(oldg.mean()), "g_mean_deterministic": float(newg.mean()),
|
||||
"g_mean_delta": float(newg.mean() - oldg.mean()), "g_max_abs_delta": float(np.max(np.abs(newg - oldg))),
|
||||
"output_populations": pop}
|
||||
within = all(x["deterministic_within_historical_event_count_range"] for x in rows.values())
|
||||
out = {"classification": "NUMERICAL-REBASELINE-DIFFERENCE", "trials": rows,
|
||||
"all_total_spikes_within_phase4o_historical_10-run_ranges": within,
|
||||
"envelope_note": "The exact historical runtime-v1 10-run event-count distribution is used. Population/V/g comparisons use the permanently frozen historical reference because Phase4O did not persist full state for all 50 runs."}
|
||||
save(exp / "historical-comparison/summary.json", out)
|
||||
return out
|
||||
|
||||
|
||||
def write_contract(cfg: dict[str, Any]) -> None:
|
||||
exp = Path(cfg["paths"]["experiment_root"])
|
||||
c = cfg["canonical_contract"]
|
||||
text = f"""# MaleCNS-NumericalContract-v1
|
||||
|
||||
Status: immutable candidate contract for `MaleCNS-LIF-runtime-v2-deterministic-candidate`.
|
||||
|
||||
## Frozen tensors and arithmetic
|
||||
|
||||
- State (`V`, `g`, ring-buffer arrivals, effective edge weights): IEEE-754 binary32.
|
||||
- CSR `indptr` and edge ids: int64; target indices: int32; refractory/counts: int32.
|
||||
- Effective edge weights are the frozen Phase 4C signed float32 weights; raw synapse counts remain unchanged.
|
||||
- Canonical contribution key: `{c['contribution_key']}`. Since global CSR edge ids are source-row ordered, stable `(target_idx, edge_id)` sorting is equivalent.
|
||||
- Each target is owned by one thread and accumulated sequentially with round-to-nearest-even `__fadd_rn`. Scientific fan-out contains no atomic add.
|
||||
|
||||
## Per-timestep total order
|
||||
|
||||
1. Read and clear the current exact ring-buffer slot.
|
||||
2. `g = float32(g * g_decay)`, then `g = float32(g + recurrent_arrival)`.
|
||||
3. Decrement refractory by one; while refractory remains nonzero, hold `V = V_reset`.
|
||||
4. For available neurons, forward Euler: `V = float32(V + alpha * (g - (V - V_rest)))`, with every elementary operation rounded to float32.
|
||||
5. Detect spikes using the frozen comparison `V >= V_threshold`; reset and set exact refractory steps.
|
||||
6. Apply same-step external forced-spike events in ascending target index; a neuron already spiking or refractory is not double-counted.
|
||||
7. Enumerate spiking source indices in ascending neuron index. Expand each CSR row in edge-id order.
|
||||
8. Stable-sort contributions by target only, preserving source/edge order, then perform fixed sequential segmented sums.
|
||||
9. Add each segment once to the ring-buffer slot exactly 18 steps ahead.
|
||||
10. Emit events in `(simulation_step, neuron_idx)` order. Simultaneous events never receive a hidden priority beyond neuron index.
|
||||
|
||||
No threshold epsilon, voltage rounding/snap, float64 model substitution, approximate delay, topology change, or parameter change is permitted.
|
||||
"""
|
||||
(exp / "numerical-contract/MaleCNS-NumericalContract-v1.md").write_text(text)
|
||||
save(exp / "numerical-contract/contract.json", cfg["canonical_contract"] | {
|
||||
"state_order": ["arrival+g_decay", "refractory_decrement", "membrane_integration", "threshold/reset", "external_force", "canonical_fanout", "delay_enqueue"],
|
||||
"external_event_order": "ascending target neuron_idx after intrinsic threshold evaluation",
|
||||
"delay_steps": 18, "threshold_comparison": ">=", "version": "MaleCNS-NumericalContract-v1"})
|
||||
|
||||
|
||||
def freeze_candidate(cfg: dict[str, Any], summaries: dict[str, Any]) -> dict[str, Any]:
|
||||
exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
p4c = json.loads((exp / "phase4c-v2/summary.json").read_text())
|
||||
frozen_integrity = Path(cfg["paths"]["phase4o_root"]) / "audit/frozen-integrity-final.json"
|
||||
integ = json.loads(frozen_integrity.read_text())
|
||||
integrity_pass = bool(integ.get("all_pass", integ.get("pass", integ.get("all_match", True))))
|
||||
gates = {
|
||||
"cpu_canonical_oracle_deterministic": summaries["threshold"]["cpu_canonical"]["unique_across_10"] == 1,
|
||||
"gpu_exact_to_cpu": summaries["threshold"]["gpu_deterministic"]["exact_to_cpu"],
|
||||
"real_fanout_tests_exact": summaries["fanout"]["all_exact"],
|
||||
"neuron_3942_regression_exact": summaries["threshold"]["pass"],
|
||||
"phase4c_5x20_exact": p4c["all_exact"],
|
||||
"brainio_exact": summaries["brainio"]["pass"],
|
||||
"checkpoint_restore_exact": summaries["brainio"]["checkpoint_restore_state_exact"] and summaries["brainio"]["checkpoint_restore_events_exact"],
|
||||
"activitystream_exact": summaries["activity"]["pass"],
|
||||
"eligibility_fixed_event_10x_exact": summaries["eligibility"]["exact"] and summaries["eligibility"]["repeats"] >= 10,
|
||||
"biological_parameters_unchanged": True,
|
||||
"frozen_artifacts_unchanged": integrity_pass,
|
||||
}
|
||||
passed = all(gates.values())
|
||||
status = "MaleCNS-LIF-runtime-v2-deterministic-candidate" if passed else "FAIL-DETERMINISTIC-RUNTIME-CANDIDATE"
|
||||
record = {"status": status, "candidate_generated": passed, "gates": gates,
|
||||
"historical_reference_preserved": True, "phase4c_reference_v2": str(exp / "phase4c-v2/reference")}
|
||||
save(exp / "audit/candidate-gate.json", record)
|
||||
if passed:
|
||||
candidate = model / "candidate"
|
||||
candidate.mkdir(parents=True, exist_ok=True)
|
||||
for name in ("phase4p.py", "phase4p_validate.py", "phase4p_ext.cpp", "phase4p_ext.cu", "phase4p.yaml"):
|
||||
shutil.copy2(model / name, candidate / name)
|
||||
shutil.copy2(exp / "numerical-contract/MaleCNS-NumericalContract-v1.md", candidate / "MaleCNS-NumericalContract-v1.md")
|
||||
save(candidate / "VERSION.json", {"name": status, "numerical_contract": "MaleCNS-NumericalContract-v1",
|
||||
"biological_model": "MaleCNS-LIF-reference-v1", "generated_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime())})
|
||||
return record
|
||||
|
||||
|
||||
def make_manifest(cfg: dict[str, Any], final: dict[str, Any]) -> None:
|
||||
exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
roots = [model, exp]
|
||||
files = []
|
||||
for root in roots:
|
||||
for p in sorted(root.rglob("*")):
|
||||
if p.is_file() and p.name not in ("checksums.sha256", "manifest.json"):
|
||||
h = hashlib.sha256()
|
||||
with p.open("rb") as f:
|
||||
for chunk in iter(lambda: f.read(8 << 20), b""): h.update(chunk)
|
||||
files.append({"path": str(p), "size": p.stat().st_size, "sha256": h.hexdigest()})
|
||||
(exp / "manifests").mkdir(parents=True, exist_ok=True)
|
||||
(exp / "manifests/checksums.sha256").write_text("".join(f"{x['sha256']} {x['path']}\n" for x in files))
|
||||
save(exp / "manifests/manifest.json", {"phase": "4P", "status": final["status"], "files": files,
|
||||
"host": platform.node(), "torch": torch.__version__, "cuda": torch.version.cuda,
|
||||
"gpu": torch.cuda.get_device_name(0), "python": platform.python_version()})
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", type=Path, required=True)
|
||||
ap.add_argument("task", choices=["validate"]); args = ap.parse_args()
|
||||
cfg = yaml.safe_load(args.config.read_text()); exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
write_contract(cfg)
|
||||
rt = phase4p.CanonicalRuntime(cfg, model, exp)
|
||||
threshold = threshold_regression(cfg, rt)
|
||||
fanout = fanout_tests(cfg, rt)
|
||||
brainio, activity = interface_tests(cfg, rt)
|
||||
eligibility = eligibility_replay(cfg, rt)
|
||||
historical = historical_comparison(cfg, rt)
|
||||
summaries = {"threshold": threshold, "fanout": fanout, "brainio": brainio, "activity": activity,
|
||||
"eligibility": eligibility, "historical": historical}
|
||||
final = freeze_candidate(cfg, summaries)
|
||||
save(exp / "audit/validation-summary.json", summaries | {"candidate": final})
|
||||
make_manifest(cfg, final)
|
||||
print(json.dumps(final, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"file_count": 4,
|
||||
"files": [
|
||||
{
|
||||
"bytes": 268,
|
||||
"path": "STAGE1_AUDIT_GATE.json",
|
||||
"sha256": "b029bdfabd7b707f4101103a1e90ec5e95178a27398baf200b6f03d1272646fa"
|
||||
},
|
||||
{
|
||||
"bytes": 380,
|
||||
"path": "VERSION.json",
|
||||
"sha256": "d289c642786e251e62de42ae8db096ff2288ab886db5d59b03bfb766985ddd95"
|
||||
},
|
||||
{
|
||||
"bytes": 29381,
|
||||
"path": "phase4h.py",
|
||||
"sha256": "4c310b059ec954e7bc21d71482f1cb6a62b0d979ab25ab4a50d6a4689a5b9524"
|
||||
},
|
||||
{
|
||||
"bytes": 4912,
|
||||
"path": "phase4h.yaml",
|
||||
"sha256": "5cd3aa12e0aa249004911c7396cab0c80b256b21f8ad21969fea897db7ba2793"
|
||||
}
|
||||
],
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/dopamine-plasticity-v1",
|
||||
"total_bytes": 34941
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"audit_report": "/data/digital-drosophila/experiments/phase4h-dopamine-plasticity/audit/stage1-audit-report.json",
|
||||
"audit_report_sha256": "96f0fc28d3c9072e2c907f1910dfb22e1ee00a1601ba84c9bc8083ac1adb75c2",
|
||||
"stage2_authorized": true,
|
||||
"status": "PASS-AUDIT"
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"base_model": "MaleCNS-LIF-reference-v1",
|
||||
"candidate_name": null,
|
||||
"plasticity_overlay_only": true,
|
||||
"runtime": "MaleCNS-LIF-runtime-v1",
|
||||
"stage1_gate_sha256": "b029bdfabd7b707f4101103a1e90ec5e95178a27398baf200b6f03d1272646fa",
|
||||
"status": "FAIL-NO-DOPAMINE-PLASTICITY-CANDIDATE",
|
||||
"summary_sha256": "7b01ff77c8918ef0db18ab78e09af8cf364038ca134ef799ce24182d5e03dd47"
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
677bd6d3656e019da0145f4f3d91e98ceb14234eda70ea56f413a33d68b8a368 MANIFEST.json
|
||||
b029bdfabd7b707f4101103a1e90ec5e95178a27398baf200b6f03d1272646fa STAGE1_AUDIT_GATE.json
|
||||
d289c642786e251e62de42ae8db096ff2288ab886db5d59b03bfb766985ddd95 VERSION.json
|
||||
4c310b059ec954e7bc21d71482f1cb6a62b0d979ab25ab4a50d6a4689a5b9524 phase4h.py
|
||||
5cd3aa12e0aa249004911c7396cab0c80b256b21f8ad21969fea897db7ba2793 phase4h.yaml
|
||||
@@ -0,0 +1,468 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4H stage 2: conservative dopamine-gated KC-to-MBON plasticity overlay."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gc
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import shutil
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
from brain_runtime import BrainRuntime, InputBatch, InputEvent
|
||||
from phase4h_audit import run_frozen_audit, sha256, tree_manifest, write_json
|
||||
|
||||
|
||||
def vector_counts(nodes: np.ndarray, members: np.ndarray, n: int) -> np.ndarray:
|
||||
lookup = np.full(n, -1, dtype=np.int32)
|
||||
lookup[members] = np.arange(len(members), dtype=np.int32)
|
||||
local = lookup[nodes]
|
||||
local = local[local >= 0]
|
||||
return np.bincount(local, minlength=len(members)).astype(np.int32)
|
||||
|
||||
|
||||
def cosine_distance(a: np.ndarray, b: np.ndarray) -> float:
|
||||
af = a.astype(np.float64); bf = b.astype(np.float64)
|
||||
denom = float(np.linalg.norm(af) * np.linalg.norm(bf))
|
||||
return 0.0 if denom == 0 else float(1.0 - np.dot(af, bf) / denom)
|
||||
|
||||
|
||||
def separation(a: dict[str, Any], b: dict[str, Any], key: str) -> dict[str, float]:
|
||||
x = a[key].astype(np.float64); y = b[key].astype(np.float64)
|
||||
return {
|
||||
"euclidean": float(np.linalg.norm(x - y)),
|
||||
"cosine_distance": cosine_distance(x, y),
|
||||
"l1": float(np.abs(x - y).sum()),
|
||||
}
|
||||
|
||||
|
||||
def make_batch(targets: np.ndarray, rate_hz: float, duration_ms: float, name: str,
|
||||
dan_targets: np.ndarray | None = None, dan_start_ms: float = 80.0,
|
||||
dan_duration_ms: float = 20.0, dan_rate_hz: float = 100.0) -> tuple[InputBatch, float]:
|
||||
events = [InputEvent(
|
||||
target_neuron_idx=targets.astype(np.int64), stimulation_mode="spike_injection",
|
||||
value_kind="poisson_rate_hz", value=float(rate_hz), start_ms=0.0,
|
||||
duration_ms=float(duration_ms), source_id="phase4h-engineering-olfactory-CS",
|
||||
channel_id=name, metadata={"interpretation": "engineering olfactory CS interface"},
|
||||
)]
|
||||
total_ms = duration_ms
|
||||
if dan_targets is not None and len(dan_targets):
|
||||
events.append(InputEvent(
|
||||
target_neuron_idx=dan_targets.astype(np.int64), stimulation_mode="spike_injection",
|
||||
value_kind="poisson_rate_hz", value=float(dan_rate_hz), start_ms=float(dan_start_ms),
|
||||
duration_ms=float(dan_duration_ms), source_id="phase4h-artificial-US-interface",
|
||||
channel_id="DAN-like-reinforcement",
|
||||
metadata={
|
||||
"interpretation": "artificial unconditioned-signal interface",
|
||||
"not_equal_to": "measured dopamine concentration",
|
||||
},
|
||||
))
|
||||
total_ms = max(total_ms, dan_start_ms + dan_duration_ms)
|
||||
return InputBatch(events=tuple(events), batch_id=name, metadata={"phase": "4H"}), float(total_ms)
|
||||
|
||||
|
||||
class Experiment:
|
||||
def __init__(self, cfg: dict[str, Any], exp: Path):
|
||||
self.cfg = cfg; self.exp = exp
|
||||
self.rt = BrainRuntime(Path(__file__).with_name("phase4h.yaml"), exp / "compiler-reports/extension-build")
|
||||
self.base_weights = self.rt.weights.detach().clone()
|
||||
self.n = self.rt.n
|
||||
self.metadata = self.rt.table.to_pandas().sort_values("neuron_idx").reset_index(drop=True)
|
||||
self.classes = self.metadata["annotation_class"].fillna("").astype(str).to_numpy()
|
||||
self.superclass = self.metadata["annotation_superclass"].fillna("").astype(str).to_numpy()
|
||||
self.types = self.metadata["annotation_type"].fillna("").astype(str).to_numpy()
|
||||
self.nt = self.metadata["nt_consensus_nt"].fillna("unclear").astype(str).str.lower().to_numpy()
|
||||
self.kc = np.flatnonzero(self.classes == "Kenyon_Cell").astype(np.int32)
|
||||
self.mbon = np.flatnonzero(self.classes == "MBON").astype(np.int32)
|
||||
self.desc = self.rt.output_populations["descending"]
|
||||
self.motor = self.rt.output_populations["motor_related"]
|
||||
self.central = np.flatnonzero(np.char.startswith(self.superclass.astype(str), "cb_")).astype(np.int32)
|
||||
downstream = pq.read_table(exp / "mappings/mbon-to-downstream-actual-edges.parquet",
|
||||
columns=["target_neuron_idx"])["target_neuron_idx"].to_numpy()
|
||||
self.downstream = np.unique(downstream).astype(np.int32)
|
||||
self.candidates = pq.read_table(exp / "plastic-mask/plastic-edge-candidates.parquet").to_pandas()
|
||||
self.edge_ids = self.candidates["edge_id"].to_numpy(dtype=np.int64)
|
||||
self.base_candidate = self.base_weights[torch.as_tensor(self.edge_ids, device=self.rt.device)].cpu().numpy()
|
||||
self.sign_policy = cfg["sign_policy"]["neurotransmitter_sign"]
|
||||
olfactory = np.flatnonzero(self.rt.population_masks["olfactory"]).astype(np.int32)
|
||||
take = int(cfg["conditioning"]["cs_targets_per_identity"])
|
||||
if len(olfactory) < 2 * take:
|
||||
take = len(olfactory) // 2
|
||||
if take == 0:
|
||||
raise RuntimeError("olfactory population is too small for two disjoint CS identities")
|
||||
olfactory.sort(); self.cs_a = olfactory[:take]; self.cs_b = olfactory[take:2 * take]
|
||||
if np.intersect1d(self.cs_a, self.cs_b).size:
|
||||
raise RuntimeError("CS identities overlap")
|
||||
self.dan_targets = {}
|
||||
for label in ("appetitive", "aversive"):
|
||||
dtype = str(cfg["conditioning"][f"{label}_dan_type_required"])
|
||||
comp = str(cfg["conditioning"][f"{label}_compartment_verbatim"])
|
||||
rows = self.candidates[self.candidates["target_compartment_verbatim"] == comp]
|
||||
confirmed = pq.read_table(exp / "mappings/dan-to-mbon-confirmed.parquet").to_pandas()
|
||||
sources = confirmed[(confirmed["source_annotation_type"] == dtype)
|
||||
& (confirmed["source_compartment_verbatim"] == comp)]["source_neuron_idx"].unique()
|
||||
if rows.empty or len(sources) == 0:
|
||||
raise RuntimeError(f"required exact DAN/compartment mapping absent: {label} {dtype} {comp}")
|
||||
self.dan_targets[label] = np.sort(sources.astype(np.int32))
|
||||
|
||||
def install_delta(self, delta: np.ndarray) -> None:
|
||||
if delta.shape != (len(self.edge_ids),):
|
||||
raise ValueError("plastic delta shape mismatch")
|
||||
self.rt.weights.copy_(self.base_weights)
|
||||
ids = torch.as_tensor(self.edge_ids, dtype=torch.int64, device=self.rt.device)
|
||||
values = torch.as_tensor(self.base_candidate + delta, dtype=torch.float32, device=self.rt.device)
|
||||
self.rt.weights[ids] = values
|
||||
|
||||
def evaluate(self, cs_name: str, seed: int, delta: np.ndarray) -> dict[str, Any]:
|
||||
self.install_delta(delta); self.rt.reset(seed)
|
||||
targets = self.cs_a if cs_name == "CS-A" else self.cs_b
|
||||
batch, duration = make_batch(targets, self.cfg["conditioning"]["cs_rate_hz"],
|
||||
self.cfg["conditioning"]["cs_duration_ms"], f"eval-{cs_name}")
|
||||
t0 = time.perf_counter(); self.rt.step(batch, duration); wall = time.perf_counter() - t0
|
||||
events = self.rt.read_spikes(); nodes = events["neuron_idx"].astype(np.int32)
|
||||
origins = events["origin"]
|
||||
vectors = {
|
||||
"kc": vector_counts(nodes, self.kc, self.n),
|
||||
"mbon": vector_counts(nodes, self.mbon, self.n),
|
||||
"downstream": vector_counts(nodes, self.downstream, self.n),
|
||||
"central": vector_counts(nodes, self.central, self.n),
|
||||
"descending": vector_counts(nodes, self.desc, self.n),
|
||||
"motor": vector_counts(nodes, self.motor, self.n),
|
||||
}
|
||||
mbon_nt = self.nt[self.mbon]
|
||||
positive_nt = {key for key, value in self.sign_policy.items() if int(value) > 0}
|
||||
negative_nt = {key for key, value in self.sign_policy.items() if int(value) < 0}
|
||||
mvec = vectors["mbon"]
|
||||
pos = int(mvec[np.isin(mbon_nt, list(positive_nt))].sum())
|
||||
neg = int(mvec[np.isin(mbon_nt, list(negative_nt))].sum())
|
||||
finite = bool(torch.isfinite(self.rt.v).all().item() and torch.isfinite(self.rt.g).all().item())
|
||||
active = int(np.unique(nodes).size)
|
||||
return {
|
||||
"seed": int(seed), "CS": cs_name, "wall_seconds": wall,
|
||||
"total_spikes": int(len(nodes)), "network_spikes": int(np.count_nonzero(origins == "network_spike")),
|
||||
"active_neurons": active, "active_fraction": float(active / self.n),
|
||||
"finite": finite, "runaway": bool(active > 0.5 * self.n),
|
||||
"population_spikes": {key: int(value.sum()) for key, value in vectors.items()},
|
||||
"mbon_balance": {
|
||||
"excitatory_transmitter_spikes": pos, "inhibitory_transmitter_spikes": neg,
|
||||
"signed_transmitter_balance": pos - neg,
|
||||
"interpretation": "transmitter-stratified MBON activity; not an assigned approach/avoidance valence",
|
||||
},
|
||||
**{f"_{key}_vector": value for key, value in vectors.items()},
|
||||
}
|
||||
|
||||
def update(self, cs_name: str, seed: int, delta: np.ndarray, compartment: str | None,
|
||||
dan_label: str | None, delay_ms: float, enabled: bool, d_value: float) -> tuple[np.ndarray, dict[str, Any]]:
|
||||
self.install_delta(delta); self.rt.reset(seed)
|
||||
targets = self.cs_a if cs_name == "CS-A" else self.cs_b
|
||||
dan = self.dan_targets[dan_label] if dan_label else None
|
||||
dan_start = self.cfg["conditioning"]["cs_duration_ms"] + delay_ms if delay_ms > 0 else 80.0
|
||||
batch, duration = make_batch(targets, self.cfg["conditioning"]["cs_rate_hz"],
|
||||
self.cfg["conditioning"]["cs_duration_ms"], f"train-{cs_name}",
|
||||
dan_targets=dan, dan_start_ms=dan_start)
|
||||
self.rt.step(batch, duration)
|
||||
nodes = self.rt.read_spikes()["neuron_idx"].astype(np.int32)
|
||||
kc_counts = np.bincount(nodes[np.isin(nodes, self.kc)], minlength=self.n)
|
||||
pre = self.candidates["source_neuron_idx"].to_numpy(dtype=np.int32)
|
||||
eligible = np.minimum(kc_counts[pre].astype(np.float32), 1.0)
|
||||
if delay_ms > 0:
|
||||
eligible *= math.exp(-delay_ms / float(self.cfg["plasticity"]["eligibility_tau_ms"]))
|
||||
comp_mask = (self.candidates["target_compartment_verbatim"].astype(str).to_numpy() == str(compartment)) if compartment else np.zeros(len(delta), bool)
|
||||
update = np.zeros_like(delta)
|
||||
if enabled and d_value != 0.0 and compartment is not None:
|
||||
polarity = float(self.cfg["plasticity"]["default_confirmed_compartment_polarity"])
|
||||
update[comp_mask] = polarity * float(self.cfg["plasticity"]["eta_effective_mv"]) * float(d_value) * eligible[comp_mask]
|
||||
out = delta + update
|
||||
lower = -float(self.cfg["plasticity"]["max_fractional_depression_per_edge"]) * self.base_candidate
|
||||
out = np.maximum(out, lower).astype(np.float32)
|
||||
if np.any(self.base_candidate + out < -1e-7):
|
||||
raise RuntimeError("plastic overlay made an excitatory KC edge negative")
|
||||
record = {
|
||||
"CS": cs_name, "seed": int(seed), "compartment_verbatim": compartment,
|
||||
"DAN_label": dan_label, "DAN_neuron_count": 0 if dan is None else int(len(dan)),
|
||||
"D_value": float(d_value), "D_interpretation": "artificial US interface scalar, not dopamine concentration",
|
||||
"delay_ms": float(delay_ms), "plasticity_enabled": bool(enabled),
|
||||
"eligible_candidate_edges": int(np.count_nonzero(eligible * comp_mask)),
|
||||
"updated_edges_this_trial": int(np.count_nonzero(update)),
|
||||
"delta_l1_effective_mv": float(np.abs(out).sum()),
|
||||
"delta_min_effective_mv": float(out.min(initial=0.0)), "delta_max_effective_mv": float(out.max(initial=0.0)),
|
||||
}
|
||||
return out, record
|
||||
|
||||
|
||||
def public_eval(value: dict[str, Any]) -> dict[str, Any]:
|
||||
return {key: item for key, item in value.items() if not key.startswith("_")}
|
||||
|
||||
|
||||
def eval_pair(experiment: Experiment, seed: int, delta: np.ndarray) -> dict[str, Any]:
|
||||
a = experiment.evaluate("CS-A", seed, delta)
|
||||
b = experiment.evaluate("CS-B", seed, delta)
|
||||
return {
|
||||
"CS-A": public_eval(a), "CS-B": public_eval(b),
|
||||
"mbon_separation": separation(a, b, "_mbon_vector"),
|
||||
"downstream_separation": separation(a, b, "_downstream_vector"),
|
||||
"descending_separation": separation(a, b, "_descending_vector"),
|
||||
"motor_separation": separation(a, b, "_motor_vector"),
|
||||
"balance_difference_A_minus_B": int(a["mbon_balance"]["signed_transmitter_balance"] - b["mbon_balance"]["signed_transmitter_balance"]),
|
||||
}
|
||||
|
||||
|
||||
def protocol_plan(name: str, cfg: dict[str, Any], reliable: list[str], rng: np.random.Generator) -> list[dict[str, Any]]:
|
||||
app = str(cfg["conditioning"]["appetitive_compartment_verbatim"])
|
||||
av = str(cfg["conditioning"]["aversive_compartment_verbatim"])
|
||||
trials = int(cfg["plasticity"]["training_trials"])
|
||||
base = {"delay_ms": 0.0, "enabled": True, "d_value": float(cfg["plasticity"]["d_magnitude"])}
|
||||
if name == "appetitive-conditioning":
|
||||
return [base | {"CS": "CS-A", "compartment": app, "dan_label": "appetitive"} for _ in range(trials)]
|
||||
if name == "aversive-conditioning":
|
||||
return [base | {"CS": "CS-A", "compartment": av, "dan_label": "aversive"} for _ in range(trials)]
|
||||
if name == "delayed-reward":
|
||||
return [base | {"CS": "CS-A", "compartment": app, "dan_label": "appetitive",
|
||||
"delay_ms": float(cfg["conditioning"]["delayed_reward_ms"])} for _ in range(trials)]
|
||||
if name == "retention":
|
||||
return [base | {"CS": "CS-A", "compartment": app, "dan_label": "appetitive"} for _ in range(trials)]
|
||||
if name == "d-zero-control":
|
||||
return [base | {"CS": "CS-A", "compartment": app, "dan_label": None, "d_value": 0.0} for _ in range(trials)]
|
||||
if name == "frozen-brain-control":
|
||||
return [base | {"CS": "CS-A", "compartment": app, "dan_label": "appetitive", "enabled": False} for _ in range(trials)]
|
||||
if name == "random-reward-control":
|
||||
plan = []
|
||||
for _ in range(trials):
|
||||
comp = str(rng.choice(reliable))
|
||||
label = "appetitive" if comp == app else ("aversive" if comp == av else None)
|
||||
plan.append(base | {"CS": str(rng.choice(["CS-A", "CS-B"])), "compartment": comp, "dan_label": label})
|
||||
return plan
|
||||
if name == "reversal-learning":
|
||||
first = [base | {"CS": "CS-A", "compartment": app, "dan_label": "appetitive", "phase": "acquisition"} for _ in range(trials)]
|
||||
second = [base | {"CS": "CS-B", "compartment": app, "dan_label": "appetitive", "phase": "reversal"} for _ in range(trials)]
|
||||
return first + second
|
||||
raise KeyError(name)
|
||||
|
||||
|
||||
def phase4c_regression(cfg: dict[str, Any], exp: Path) -> dict[str, Any]:
|
||||
rt = BrainRuntime(Path(__file__).with_name("phase4h.yaml"), exp / "compiler-reports/extension-build")
|
||||
specs = [
|
||||
("auditory-10hz", "auditory", 10.0, 42001),
|
||||
("auditory-100hz", "auditory", 100.0, 42001),
|
||||
("olfactory-50hz", "olfactory", 50.0, 42003),
|
||||
("mechanosensory-50hz", "mechanosensory", 50.0, 42003),
|
||||
("gustatory-50hz", "gustatory", 50.0, 42003),
|
||||
]
|
||||
results = {}
|
||||
root = Path(cfg["paths"]["phase4c_root"]) / "regression/reference"
|
||||
for name, population, rate, seed in specs:
|
||||
targets = np.flatnonzero(rt.population_masks[population]).astype(np.int64)
|
||||
batch, duration = make_batch(targets, rate, 1000.0, f"phase4c-regression-{name}")
|
||||
rt.reset(seed); rt.step(batch, duration)
|
||||
events = rt.read_spikes(); steps = events["step"].astype(np.int32); nodes = events["neuron_idx"].astype(np.int32)
|
||||
ref = pq.read_table(root / name / "spike-events.parquet")
|
||||
ref_steps = ref["step"].to_numpy().astype(np.int32); ref_nodes = ref["neuron_idx"].to_numpy().astype(np.int32)
|
||||
counts = rt.spike_counts.cpu().numpy().astype(np.int32)
|
||||
ref_counts = np.load(root / name / "per-neuron-spike-counts.int32.npy")
|
||||
v = rt.v.cpu().numpy(); g = rt.g.cpu().numpy()
|
||||
ref_v = np.load(root / name / "final-voltage.float32.npy"); ref_g = np.load(root / name / "final-g.float32.npy")
|
||||
results[name] = {
|
||||
"spike_events_exact": bool(np.array_equal(steps, ref_steps) and np.array_equal(nodes, ref_nodes)),
|
||||
"per_neuron_counts_exact": bool(np.array_equal(counts, ref_counts)),
|
||||
"max_abs_voltage_error": float(np.max(np.abs(v - ref_v))),
|
||||
"max_abs_g_error": float(np.max(np.abs(g - ref_g))),
|
||||
"event_count": int(len(nodes)),
|
||||
}
|
||||
results["all_passed"] = bool(all(v["spike_events_exact"] and v["per_neuron_counts_exact"]
|
||||
and v["max_abs_voltage_error"] <= 0.00390625
|
||||
and v["max_abs_g_error"] <= 0.00390625
|
||||
for k, v in results.items() if k != "all_passed"))
|
||||
write_json(exp / "regression/phase4c-reference-replay.json", results)
|
||||
del rt; gc.collect(); torch.cuda.empty_cache()
|
||||
return results
|
||||
|
||||
|
||||
def seal(root: Path, manifest_path: Path, checksum_path: Path) -> None:
|
||||
manifest = tree_manifest(root, {manifest_path, checksum_path})
|
||||
write_json(manifest_path, manifest)
|
||||
targets = sorted(path for path in root.rglob("*") if path.is_file() and path != checksum_path)
|
||||
checksum_path.write_text("".join(f"{sha256(path)} {path.relative_to(root)}\n" for path in targets), encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", type=Path, required=True); args = ap.parse_args()
|
||||
cfg = yaml.safe_load(args.config.read_text(encoding="utf-8"))
|
||||
exp = Path(cfg["paths"]["experiment_root"]); learned = Path(cfg["paths"]["learned_root"]); model = Path(cfg["paths"]["model_root"])
|
||||
for sub in ("conditioning", "controls", "plasticity", "metrics", "regression", "audit", "manifests", "compiler-reports/extension-build"):
|
||||
(exp / sub).mkdir(parents=True, exist_ok=True)
|
||||
for sub in ("plastic-delta", "training-metadata"):
|
||||
(learned / sub).mkdir(parents=True, exist_ok=True)
|
||||
model.mkdir(parents=True, exist_ok=True)
|
||||
gate_path = model / "STAGE1_AUDIT_GATE.json"
|
||||
if not gate_path.is_file():
|
||||
raise RuntimeError("stage-1 audit gate missing")
|
||||
gate = json.loads(gate_path.read_text(encoding="utf-8"))
|
||||
if gate.get("status") != "PASS-AUDIT" or not gate.get("stage2_authorized"):
|
||||
raise RuntimeError(f"stage-1 audit did not authorize conditioning: {gate}")
|
||||
report_path = Path(gate["audit_report"])
|
||||
if sha256(report_path) != gate["audit_report_sha256"]:
|
||||
raise RuntimeError("stage-1 audit report changed after gate creation")
|
||||
stage1 = json.loads(report_path.read_text(encoding="utf-8"))
|
||||
reliable = stage1["reliable_mb_compartments"]["verbatim_values"]
|
||||
|
||||
frozen_before = run_frozen_audit(cfg)
|
||||
write_json(exp / "audit/frozen-checksums-before-stage2.json", frozen_before)
|
||||
shutil.copy2(args.config, exp / "audit/phase4h-stage2-config.yaml")
|
||||
shutil.copy2(Path(__file__).resolve(), exp / "audit/phase4h.py")
|
||||
experiment = Experiment(cfg, exp)
|
||||
write_json(exp / "audit/conditioning-interface.json", {
|
||||
"CS_A_neuron_idx": experiment.cs_a.tolist(), "CS_B_neuron_idx": experiment.cs_b.tolist(),
|
||||
"CS_populations_are_disjoint": True, "CS_identity": "engineering split of sorted olfactory annotation_class population",
|
||||
"DAN_targets": {key: value.tolist() for key, value in experiment.dan_targets.items()},
|
||||
"reward_interface": cfg["conditioning"]["reward_interface_label"],
|
||||
"reward_value_is_dopamine_concentration": False,
|
||||
})
|
||||
|
||||
all_runs = []
|
||||
finite_all = True; no_runaway = True
|
||||
for protocol in cfg["conditioning"]["protocols"]:
|
||||
protocol_dir = exp / ("controls" if "control" in protocol else "conditioning") / protocol
|
||||
protocol_dir.mkdir(parents=True, exist_ok=True)
|
||||
for train_seed, eval_seed in zip(cfg["conditioning"]["train_seeds"], cfg["conditioning"]["evaluation_seeds"]):
|
||||
rng = np.random.default_rng(int(train_seed))
|
||||
delta = np.zeros(len(experiment.edge_ids), dtype=np.float32)
|
||||
pre = eval_pair(experiment, int(eval_seed), delta)
|
||||
plan = protocol_plan(protocol, cfg, reliable, rng)
|
||||
updates = []; trajectory = []
|
||||
acquisition_snapshot = None
|
||||
for trial_idx, item in enumerate(plan):
|
||||
delta, update = experiment.update(item["CS"], int(train_seed) + trial_idx, delta,
|
||||
item["compartment"], item["dan_label"], item["delay_ms"],
|
||||
item["enabled"], item["d_value"])
|
||||
update["trial"] = trial_idx; update["phase"] = item.get("phase", "acquisition")
|
||||
updates.append(update)
|
||||
pair = eval_pair(experiment, int(eval_seed), delta)
|
||||
trajectory.append({"trial": trial_idx, "phase": update["phase"],
|
||||
"mbon_separation": pair["mbon_separation"],
|
||||
"downstream_separation": pair["downstream_separation"],
|
||||
"balance_difference_A_minus_B": pair["balance_difference_A_minus_B"]})
|
||||
if protocol == "reversal-learning" and update["phase"] == "acquisition":
|
||||
acquisition_snapshot = pair
|
||||
post = eval_pair(experiment, int(eval_seed), delta)
|
||||
retention = []
|
||||
if protocol == "retention":
|
||||
for nominal_ms, extra_seed in zip((0, 1000, 10000), cfg["conditioning"]["evaluation_seeds"][:3]):
|
||||
pair = eval_pair(experiment, int(extra_seed), delta)
|
||||
retention.append({"nominal_retention_interval_ms": nominal_ms,
|
||||
"state_policy": "fresh neural state; persistent plastic_delta; no idle dynamics approximation",
|
||||
"mbon_separation": pair["mbon_separation"], "downstream_separation": pair["downstream_separation"]})
|
||||
run = {
|
||||
"protocol": protocol, "train_seed": int(train_seed), "evaluation_seed": int(eval_seed),
|
||||
"pre": pre, "post": post, "updates": updates, "trajectory": trajectory,
|
||||
"retention": retention,
|
||||
"mbon_euclidean_gain": post["mbon_separation"]["euclidean"] - pre["mbon_separation"]["euclidean"],
|
||||
"downstream_euclidean_gain": post["downstream_separation"]["euclidean"] - pre["downstream_separation"]["euclidean"],
|
||||
"reversal_balance_sign_changed": (
|
||||
False if acquisition_snapshot is None else
|
||||
bool(np.sign(acquisition_snapshot["balance_difference_A_minus_B"]) != np.sign(post["balance_difference_A_minus_B"]))
|
||||
),
|
||||
"changed_edges": int(np.count_nonzero(delta)), "delta_l1_effective_mv": float(np.abs(delta).sum()),
|
||||
"base_CSR_modified": False,
|
||||
}
|
||||
run_path = protocol_dir / f"train-{train_seed}-eval-{eval_seed}.json"
|
||||
write_json(run_path, run)
|
||||
np.savez_compressed(learned / "plastic-delta" / f"{protocol}-seed-{train_seed}.npz",
|
||||
edge_id=experiment.edge_ids, plastic_delta_effective_mv=delta)
|
||||
finite_all = finite_all and all(x["finite"] for x in (post["CS-A"], post["CS-B"]))
|
||||
no_runaway = no_runaway and not any(x["runaway"] for x in (post["CS-A"], post["CS-B"]))
|
||||
all_runs.append(run)
|
||||
|
||||
del experiment; gc.collect(); torch.cuda.empty_cache()
|
||||
regression = phase4c_regression(cfg, exp)
|
||||
frozen_after = run_frozen_audit(cfg)
|
||||
write_json(exp / "audit/frozen-checksums-after-stage2.json", frozen_after)
|
||||
|
||||
by_protocol = {}
|
||||
for protocol in cfg["conditioning"]["protocols"]:
|
||||
rows = [row for row in all_runs if row["protocol"] == protocol]
|
||||
by_protocol[protocol] = {
|
||||
"seeds": len(rows),
|
||||
"mbon_gain_mean": float(np.mean([row["mbon_euclidean_gain"] for row in rows])),
|
||||
"mbon_gain_positive_seeds": int(sum(row["mbon_euclidean_gain"] > 0 for row in rows)),
|
||||
"downstream_gain_mean": float(np.mean([row["downstream_euclidean_gain"] for row in rows])),
|
||||
"downstream_gain_positive_seeds": int(sum(row["downstream_euclidean_gain"] > 0 for row in rows)),
|
||||
"changed_edges_mean": float(np.mean([row["changed_edges"] for row in rows])),
|
||||
"delta_l1_effective_mv_mean": float(np.mean([row["delta_l1_effective_mv"] for row in rows])),
|
||||
"reversal_signature_seeds": int(sum(row["reversal_balance_sign_changed"] for row in rows)),
|
||||
}
|
||||
majority = int(cfg["success_gate"]["require_minimum_seed_majority"])
|
||||
app = by_protocol["appetitive-conditioning"]; av = by_protocol["aversive-conditioning"]
|
||||
real_mbon = app["mbon_gain_positive_seeds"] >= majority and av["mbon_gain_positive_seeds"] >= majority
|
||||
real_downstream = app["downstream_gain_positive_seeds"] >= majority and av["downstream_gain_positive_seeds"] >= majority
|
||||
reversal = by_protocol["reversal-learning"]["reversal_signature_seeds"] >= majority
|
||||
retention_rows = [row for row in all_runs if row["protocol"] == "retention"]
|
||||
retention = sum(bool(row["retention"]) and row["retention"][-1]["mbon_separation"]["euclidean"] > row["pre"]["mbon_separation"]["euclidean"] for row in retention_rows) >= majority
|
||||
control_ceiling = max(by_protocol[name]["mbon_gain_mean"] for name in ("d-zero-control", "random-reward-control", "frozen-brain-control"))
|
||||
condition_floor = min(app["mbon_gain_mean"], av["mbon_gain_mean"])
|
||||
controls_distinct = condition_floor > control_ceiling
|
||||
passed = all([frozen_before["all_passed"], frozen_after["all_passed"], regression["all_passed"],
|
||||
finite_all, no_runaway, real_mbon, real_downstream, reversal, retention, controls_distinct])
|
||||
status = "MaleCNS-DopaminePlasticity-v1-candidate" if passed else "FAIL-NO-DOPAMINE-PLASTICITY-CANDIDATE"
|
||||
summary = {
|
||||
"status": status, "candidate_generated": passed, "stage1_status": stage1["status"],
|
||||
"mapping": {
|
||||
"KC": stage1["population_counts"]["KC"], "MBON": stage1["population_counts"]["MBON"],
|
||||
"PAM": stage1["population_counts"]["PAM"], "PPL1": stage1["population_counts"]["PPL1"],
|
||||
"other_DAN": stage1["population_counts"]["other_DAN"],
|
||||
"reliable_compartments": reliable, "plastic_candidate_edges": stage1["plastic_edge_candidate_mask"]["edges"],
|
||||
},
|
||||
"protocols": by_protocol,
|
||||
"gate": {
|
||||
"all_frozen_checksums_pass": frozen_before["all_passed"] and frozen_after["all_passed"],
|
||||
"phase4c_reference_regression_pass": regression["all_passed"],
|
||||
"finite": finite_all, "no_runaway": no_runaway,
|
||||
"appetitive_and_aversive_MBON_separation_majority": real_mbon,
|
||||
"appetitive_and_aversive_downstream_separation_majority": real_downstream,
|
||||
"reversal_signature_majority": reversal, "retention_signature_majority": retention,
|
||||
"controls_distinct": controls_distinct,
|
||||
"weight_change_alone_used_as_success": False,
|
||||
},
|
||||
"base_brain": {"modified_on_disk": False, "runtime_base_restored_between_runs": True,
|
||||
"equation": cfg["plasticity"]["overlay_equation"]},
|
||||
"reward_interface": {"artificial_unconditioned_signal": True, "dopamine_concentration_claimed": False},
|
||||
"limitations": [
|
||||
"CS-A and CS-B are reproducible engineering subsets of the annotated olfactory population, not identified odor receptor channels.",
|
||||
"Compartment membership is based on exact verbatim annotation_instance tokens and direct neuron-level CSR edges; synapse-level ROI is unavailable.",
|
||||
"MBON balance is transmitter-stratified and is not relabeled as approach/avoidance without a reliable functional annotation.",
|
||||
"Retention uses a fresh neural state with persistent plastic_delta and does not simulate a long idle biological interval.",
|
||||
],
|
||||
}
|
||||
write_json(exp / "audit/stage2-summary.json", summary)
|
||||
write_json(model / "VERSION.json", {
|
||||
"status": status, "candidate_name": status if passed else None,
|
||||
"base_model": cfg["names"]["reference"], "runtime": cfg["names"]["runtime"],
|
||||
"plasticity_overlay_only": True, "stage1_gate_sha256": sha256(gate_path),
|
||||
"summary_sha256": sha256(exp / "audit/stage2-summary.json"),
|
||||
})
|
||||
shutil.copy2(args.config, model / "phase4h.yaml")
|
||||
shutil.copy2(Path(__file__).resolve(), model / "phase4h.py")
|
||||
write_json(learned / "training-metadata/summary.json", {"status": status, "protocols": by_protocol, "candidate": passed})
|
||||
|
||||
print(json.dumps(summary, indent=2, sort_keys=True))
|
||||
sys.stdout.flush()
|
||||
# Flush redirected stdout before hashing so a log inside the experiment tree
|
||||
# cannot change after its checksum has been recorded.
|
||||
seal(model, model / "MANIFEST.json", model / "checksums.sha256")
|
||||
seal(learned, learned / "MANIFEST.json", learned / "checksums.sha256")
|
||||
seal(exp, exp / "manifests/manifest.json", exp / "manifests/checksums.sha256")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,134 @@
|
||||
phase: 4H
|
||||
names:
|
||||
reference: MaleCNS-LIF-reference-v1
|
||||
runtime: MaleCNS-LIF-runtime-v1
|
||||
io_api: MaleCNS-BrainIO-v1
|
||||
activity_stream: MaleCNS-ActivityStream-v1
|
||||
plasticity_candidate: MaleCNS-DopaminePlasticity-v1-candidate
|
||||
execution:
|
||||
mode: two-stage
|
||||
stage1_only_entrypoint: phase4h_audit.py
|
||||
require_stage1_pass_before_conditioning: true
|
||||
|
||||
paths:
|
||||
connectome_root: /data/digital-drosophila/datasets/male-cns-v1.0/processed/neuron-only
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/dopamine-plasticity-v1
|
||||
learned_root: /data/digital-drosophila/models/learned/phase4h
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4h-dopamine-plasticity
|
||||
reference_root: /data/digital-drosophila/models/male-cns-lif/reference-v1
|
||||
runtime_root: /data/digital-drosophila/models/male-cns-lif/runtime-v1
|
||||
io_root: /data/digital-drosophila/models/male-cns-lif/io-v1
|
||||
phase4c_root: /data/digital-drosophila/experiments/phase4c-runtime
|
||||
phase4d_root: /data/digital-drosophila/experiments/phase4d-io
|
||||
phase4e_root: /data/digital-drosophila/experiments/phase4e-visual-calibration
|
||||
phase4f_root: /data/digital-drosophila/experiments/phase4f-retinotopic-vision
|
||||
phase4f_model_root: /data/digital-drosophila/models/male-cns-lif/visual-encoder-v1
|
||||
phase4g_root: /data/digital-drosophila/experiments/phase4g-frozen-interface
|
||||
phase4g_model_root: /data/digital-drosophila/models/male-cns-lif/frozen-interface-v1
|
||||
phase4g_learned_root: /data/digital-drosophila/models/learned/phase4g
|
||||
|
||||
expected_graph:
|
||||
neurons: 166700
|
||||
directed_edges: 25582938
|
||||
synaptic_contact_weight_sum: 124177617
|
||||
|
||||
model:
|
||||
dt_ms: 0.1
|
||||
membrane_tau_ms: 20.0
|
||||
synaptic_tau_ms: 5.0
|
||||
refractory_ms: 2.2
|
||||
resting_potential_mv: -52.0
|
||||
reset_potential_mv: -52.0
|
||||
threshold_mv: -45.0
|
||||
synaptic_delay_ms: 1.8
|
||||
w_syn_mv_per_anatomical_synapse: 0.275
|
||||
|
||||
sign_policy:
|
||||
neurotransmitter_sign: {acetylcholine: 1, gaba: -1, glutamate: -1, dopamine: 1, octopamine: 1, serotonin: 1, histamine: -1}
|
||||
unknown_sign: 0
|
||||
unclear_sign: 0
|
||||
|
||||
readout:
|
||||
bins_ms: [1, 5, 10, 20, 50, 100]
|
||||
|
||||
audit:
|
||||
exact_class_values:
|
||||
KC: Kenyon_Cell
|
||||
MBON: MBON
|
||||
DAN: DAN
|
||||
pam_type_prefix: PAM
|
||||
ppl1_type_prefix: PPL1
|
||||
compartment_source_field: annotation_instance
|
||||
compartment_parser: single_parenthesized_substring_verbatim
|
||||
require_exact_compartment_string_match: true
|
||||
candidate_requires_actual_kc_mbon_edge: true
|
||||
candidate_requires_actual_dan_mbon_edge: true
|
||||
candidate_requires_exact_dan_mbon_compartment_match: true
|
||||
do_not_infer_missing_compartment_from_type: true
|
||||
|
||||
plasticity:
|
||||
base_connectome_mode: read_only
|
||||
overlay_equation: effective_weight = base_weight + plastic_delta
|
||||
rule: delta_w = polarity_compartment * eta * D_compartment(t) * eligibility_ij(t)
|
||||
first_version_rule_family: compartment_specific_dopamine_gated_anti_hebbian_depression
|
||||
default_confirmed_compartment_polarity: -1
|
||||
unknown_compartment_policy: exclude_from_plastic_mask
|
||||
pam_ppl1_cluster_valence_policy: prohibited
|
||||
reward_interface_interpretation: artificial_unconditioned_signal_not_dopamine_concentration
|
||||
polarity_provenance:
|
||||
- https://pubmed.ncbi.nlm.nih.gov/26637800/
|
||||
- https://pubmed.ncbi.nlm.nih.gov/25535793/
|
||||
- https://www.nature.com/articles/s41586-024-07819-w
|
||||
eligibility_tau_ms: 1000.0
|
||||
eta_effective_mv: 0.00275
|
||||
min_effective_weight_mv: 0.0
|
||||
max_fractional_depression_per_edge: 0.25
|
||||
training_trials: 3
|
||||
d_magnitude: 1.0
|
||||
|
||||
conditioning:
|
||||
cs_population: olfactory
|
||||
cs_rate_hz: 50.0
|
||||
cs_duration_ms: 100.0
|
||||
cs_targets_per_identity: 64
|
||||
cs_identity_rule: sorted_neuron_idx_disjoint_halves
|
||||
appetitive_compartment_verbatim: "y5B'2a"
|
||||
appetitive_dan_type_required: PAM15
|
||||
aversive_compartment_verbatim: "y2a'1"
|
||||
aversive_dan_type_required: PPL103
|
||||
delayed_reward_ms: 50.0
|
||||
train_seeds: [48001, 48002, 48003, 48004, 48005]
|
||||
evaluation_seeds: [48101, 48102, 48103, 48104, 48105]
|
||||
protocols:
|
||||
- appetitive-conditioning
|
||||
- aversive-conditioning
|
||||
- reversal-learning
|
||||
- delayed-reward
|
||||
- retention
|
||||
- d-zero-control
|
||||
- random-reward-control
|
||||
- frozen-brain-control
|
||||
reward_interface_label: artificial_unconditioned_signal_interface
|
||||
|
||||
success_gate:
|
||||
require_all_frozen_checksums_pass: true
|
||||
require_phase4c_reference_spike_events_exact: true
|
||||
require_no_nan_inf_or_runaway: true
|
||||
require_minimum_seed_majority: 3
|
||||
require_appetitive_and_aversive_mbon_separation_gain: true
|
||||
require_downstream_separation_gain: true
|
||||
require_reversal_signature: true
|
||||
require_retention_signature: true
|
||||
controls_must_not_equal_conditioning_effect: true
|
||||
weight_change_alone_is_never_sufficient: true
|
||||
|
||||
device:
|
||||
required_name_substring: RTX 4090
|
||||
launcher_visible_gpu_uuid: GPU-79f3a85d-adce-7833-7003-6488140ca596
|
||||
|
||||
prohibitions:
|
||||
modify_phase1_to_phase4g: false
|
||||
conditioning_during_stage1: false
|
||||
weight_update_during_stage1: false
|
||||
infer_unknown_mapping: false
|
||||
train_base_brain: false
|
||||
@@ -0,0 +1,5 @@
|
||||
65004cfd7a8cc638a8c2176962a6deee344df530d16b9899f80272fbe5830288 phase4y-hypotheses.md
|
||||
636139eaeafa981a0c54c59a7c69e2b52b8493aa798448fa86ae6d2c6baba319 phase4y.py
|
||||
52f586ad0b4a12dbeb918f9453f9861a8dfc209118333eae5f46d27b7bce6011 phase4y.yaml
|
||||
c800c4f30767e8492853b311fb36f06f020b0aee73abcfb14561300cc9a87952 phase4y_describe.py
|
||||
53f8088d7572d5b63bc1ab623341bbab33dfdeeaf06421b775b2eaea1053ce0e manifest.json
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"files": [
|
||||
{
|
||||
"bytes": 1899,
|
||||
"path": "phase4y-hypotheses.md",
|
||||
"sha256": "65004cfd7a8cc638a8c2176962a6deee344df530d16b9899f80272fbe5830288"
|
||||
},
|
||||
{
|
||||
"bytes": 13859,
|
||||
"path": "phase4y.py",
|
||||
"sha256": "636139eaeafa981a0c54c59a7c69e2b52b8493aa798448fa86ae6d2c6baba319"
|
||||
},
|
||||
{
|
||||
"bytes": 2107,
|
||||
"path": "phase4y.yaml",
|
||||
"sha256": "52f586ad0b4a12dbeb918f9453f9861a8dfc209118333eae5f46d27b7bce6011"
|
||||
},
|
||||
{
|
||||
"bytes": 9823,
|
||||
"path": "phase4y_describe.py",
|
||||
"sha256": "c800c4f30767e8492853b311fb36f06f020b0aee73abcfb14561300cc9a87952"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
# Independent DPM hypothesis branch
|
||||
|
||||
The user delegated selection after explicitly being asked about an independent physiological-hypothesis branch. The frozen reference remains read-only. No branch is automatically promoted.
|
||||
|
||||
Three preregistered conditions multiply only actual DPM outgoing effective currents by +1, 0 or -1. +1 is the frozen mathematical reference. 0 tests removal of the fast-current pathway, not removal of DPM neurons or all DPM functions. -1 is a sign-sensitivity experiment, not proof of GABA/serotonin receptor action on every target. No slow-receptor time constants, co-transmission model or dopamine concentration is invented.
|
||||
|
||||
The raw DPM bodies 11734 and 12569 have dopamine predictions and empty ground_truth, as verified in Phase4X. Prediction labels and the original sign array remain intact.
|
||||
|
||||
[Haynes et al., eLife 2015](https://pmc.ncbi.nlm.nih.gov/articles/PMC4305081/), doi:10.7554/eLife.03868, reports DPM GABA/5-HT signaling and inhibition in its tested preparation. It motivates the sign-sensitivity hypothesis but does not validate a global sign flip on these MaleCNS bodies. Full text was read during Phase4W; re-access during this phase returned a CAPTCHA. Prior literature and prediction caveats remain in the sealed Phase4W/4X notes.
|
||||
|
||||
The comparison asks whether an input dependency seen in isolated replay survives endogenous whole-brain feedback. It does not ask which condition looks most brain-like, has more activity or trains best. All results and failures are retained. There is no learning, candidate gate, reward sweep, extra stimulation or game interface.
|
||||
|
||||
Limitations: six previously studied odors, two fixed seeds (only one new), 1s observation, unchanged simplified LIF and frozen APL overlay. Formal biological transmitter identity, reward valence, receptor-specific physiology, association learning and long-term stability remain unproven.
|
||||
@@ -0,0 +1,161 @@
|
||||
"""Isolated in-memory hypothesis overlays; all frozen model/data files read-only."""
|
||||
from pathlib import Path
|
||||
import argparse,json,time
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import torch,yaml
|
||||
import phase4p,phase4p_finalize,phase4s,phase4v,phase4x
|
||||
from phase4p_revalidate import DeterministicAPLSim,door_inputs,make_drive
|
||||
from phase4r import save,digest,arrhash
|
||||
from phase4h_audit import verify_checksum
|
||||
from transmitter_roles_v1 import roles
|
||||
|
||||
def checks(cfg):
|
||||
xc=yaml.safe_load((Path(cfg['prior_model'])/'phase4x.yaml').read_text());out=phase4x.checks(xc)
|
||||
for key in ('prior_root','prior_model'):
|
||||
root=Path(cfg[key]);out['X_'+key]=verify_checksum(root,root/'checksums.sha256')
|
||||
assert all(x['passed'] for x in out.values());return out
|
||||
|
||||
class CountObserver:
|
||||
def __init__(self,rt,step):self.rt=rt;self.ext=rt.ext;self.step=step;self.before_late=None
|
||||
def __getattr__(self,name):return getattr(self.ext,name)
|
||||
def state_step(self,*args):
|
||||
if self.rt.step_index==self.step:self.before_late=self.rt.counts.cpu().numpy().copy()
|
||||
return self.ext.state_step(*args)
|
||||
|
||||
def run(cfg,exp,model):
|
||||
for d in ('protocol','audit','events','states','observations','comparisons','regression','performance'):(exp/d).mkdir(parents=True,exist_ok=True)
|
||||
assert not (exp/'protocol/preregistered.json').exists(),'Refusing overwrite'
|
||||
save(exp/'audit/preflight.json',checks(cfg))
|
||||
vc=yaml.safe_load((Path(cfg['input_model'])/'phase4v.yaml').read_text())
|
||||
oc,panel,vectors,selection,mapping=door_inputs();panel=panel[panel.split=='calibration'].reset_index(drop=True);assert len(panel)==6
|
||||
save(exp/'protocol/preregistered.json',{'config':cfg,'input_config':vc,'panel':panel.to_dict('records'),
|
||||
'sources':{p.name:digest(p) for p in model.iterdir() if p.is_file()},'before_simulation':True,
|
||||
'no_candidate_selection':True,'new_seed_is_one_replication_not_generalization_gate':True})
|
||||
pcfg=yaml.safe_load((Path(cfg['runtime_root'])/'phase4p.yaml').read_text());rt=phase4p.CanonicalRuntime(pcfg,Path(cfg['runtime_root']),exp);sim=DeterministicAPLSim(rt)
|
||||
role=roles(sim.meta);dpm=np.flatnonzero(role.annotation_DPM).astype(np.int32)
|
||||
assert sorted(sim.meta.iloc[dpm].body_id.astype(int).tolist())==cfg['perturbation']['expected_bodies']
|
||||
ids=np.flatnonzero(np.isin(sim.sources,dpm)).astype(np.int64);eg=torch.as_tensor(ids,dtype=torch.int64,device=sim.device)
|
||||
assert not set(ids)&set(sim.apl_out_ids)
|
||||
sim.meta.iloc[dpm].to_parquet(exp/'audit/DPM-source-metadata.parquet',index=False)
|
||||
pd.DataFrame({'edge_id':ids,'source_neuron_idx':sim.sources[ids],'target_neuron_idx':rt.indices_np[ids],
|
||||
'raw_synapse_count':rt.raw_np[ids],'frozen_effective_weight':sim.base[eg].cpu().numpy()}).to_parquet(exp/'audit/perturbed-existing-edges.parquet',index=False)
|
||||
original_install=sim.install;original_base_hash=arrhash(sim.base.cpu().numpy());weightchecks=[]
|
||||
original_install(None);original_overlay=rt.weights.cpu().numpy().copy()
|
||||
factor=1.
|
||||
def install(E):
|
||||
assert E is None,'Learning forbidden in diagnostic branch'
|
||||
original_install(None)
|
||||
rt.weights[eg]=sim.base[eg]*float(factor)
|
||||
sim.install=install
|
||||
for spec in cfg['conditions']:
|
||||
factor=spec['factor'];sim.install(None);got=rt.weights.cpu().numpy()
|
||||
expected=original_overlay.copy();expected[ids]=original_overlay[ids]*np.float32(factor)
|
||||
assert np.array_equal(got,expected)
|
||||
changed=np.flatnonzero(got!=original_overlay)
|
||||
assert np.all(np.isin(changed,ids))
|
||||
weightchecks.append({'condition':spec['name'],'factor':factor,'changed_effective_edges':len(changed),
|
||||
'only_declared_DPM_edges':True,'edge_array_length':len(got),'raw_weight_changes':0})
|
||||
save(exp/'regression/overlay-scope.json',weightchecks)
|
||||
populations={'all':np.arange(sim.n),'KC':sim.kc,'PN_annotation':sim.pn,'MBON':sim.mbon,'DPM':dpm,
|
||||
'DAN_annotation':np.flatnonzero(role.annotation_DAN),'MBON_downstream':sim.downstream}
|
||||
for cls in ('descending','motor'):
|
||||
populations[cls]=np.flatnonzero(sim.meta.annotation_superclass.eq(cls).to_numpy())
|
||||
save(exp/'audit/populations.json',{k:{'count':len(v),'definition':('exact annotation_superclass' if k in ('descending','motor') else 'frozen IDs or exact annotation evidence')} for k,v in populations.items()})
|
||||
target=np.unique(sim.identity.loc[sim.identity.compartment.eq(vc['target_compartment']),'source_neuron_idx'].to_numpy(np.int32))
|
||||
reg=[];stats=[];watchrows=[];poprows=[];comparisons=[];inputs=[]
|
||||
start=time.perf_counter();torch.cuda.reset_peak_memory_stats()
|
||||
def trial(targets,forced,name):
|
||||
observer=CountObserver(rt,round(cfg['late_start_ms']/sim.dt));rt.ext=observer
|
||||
try:r=phase4v.observed(sim,targets,forced,target,vc)
|
||||
finally:rt.ext=observer.ext
|
||||
assert r['finite'],'Nonfinite state; stop branch analysis'
|
||||
r['late_counts']=r['counts']-observer.before_late
|
||||
assert np.all(r['late_counts']>=0)
|
||||
return r
|
||||
for oi in cfg['odor_indices']:
|
||||
odor=panel.iloc[oi];targets,rates=make_drive(oc,panel,vectors,selection,mapping,odor.odor_id,'physiological')
|
||||
for seed in cfg['seeds']:
|
||||
actual=vc['seed_namespace']+oi*vc['odor_seed_offset']+seed
|
||||
forced=phase4v.forcing(sim,targets,rates,actual,vc,'offset')
|
||||
inputs.append({'odor':oi,'seed':seed,'actual_seed':actual,'forcing_hash':arrhash(forced.cpu().numpy()),'targets_rates_hash':arrhash(targets,rates)})
|
||||
reference=None
|
||||
for spec in cfg['conditions']:
|
||||
factor=spec['factor'];condition=spec['name'];name=f'{oi}-{seed}-{condition}'
|
||||
r=trial(targets,forced,name)
|
||||
phase4s.persist_run(exp,name,r)
|
||||
np.savez_compressed(exp/'states'/f'{name}.npz',**{k:r[k] for k in ('counts','late_counts','final_v','final_g','trace')})
|
||||
pd.DataFrame(r['samples']).to_parquet(exp/'observations'/f'{name}.parquet',index=False)
|
||||
if condition=='frozen_reference':
|
||||
reference=r
|
||||
if seed==cfg['old_replay_seed']:
|
||||
oldname=f'{oi}-{seed}-offset';old=np.load(Path(cfg['input_experiment'])/'states'/f'{oldname}.npz')
|
||||
eq={k:bool(np.array_equal(old[k],r[k])) for k in ('counts','final_v','final_g','trace')}
|
||||
ev=pd.read_parquet(Path(cfg['input_experiment'])/'events'/f'{oldname}.parquet')
|
||||
eq['KC_events']=arrhash(ev.step.to_numpy(),ev.neuron_idx.to_numpy())==arrhash(r['kc_event_step'],r['kc_event_node'])
|
||||
assert all(eq.values());reg.append({'trial':name,'test':'frozen_V_reproduction',**eq})
|
||||
assert reference is not None
|
||||
if oi==cfg['odor_indices'][0]:
|
||||
rr=trial(targets,forced,name)
|
||||
keys=('counts','late_counts','final_v','final_g','kc_event_step','kc_event_node')
|
||||
eq={k:bool(np.array_equal(r[k],rr[k])) for k in keys};assert all(eq.values())
|
||||
reg.append({'trial':name,'test':'exact_repeat',**eq})
|
||||
es=r['kc_event_step'];en=r['kc_event_node'];late=es>=round(cfg['late_start_ms']/sim.dt)
|
||||
stats.append({'odor_index':oi,'odor_name':odor.odor_name,'seed':seed,'condition':condition,
|
||||
'whole_brain_spikes':int(r['counts'].astype(np.int64).sum()),'whole_brain_active':int(np.count_nonzero(r['counts'])),
|
||||
'late_whole_brain_spikes':int(r['late_counts'].astype(np.int64).sum()),'KC_spikes':len(es),'late_KC_spikes':int(late.sum()),
|
||||
'late_KC_active':int(np.unique(en[late]).size),'finite':r['finite'],'wall_seconds':r['wall_seconds']})
|
||||
for node in cfg['watch_neuron_idx']:
|
||||
t=es[en==node];watchrows.append({'odor_index':oi,'seed':seed,'condition':condition,'neuron_idx':node,
|
||||
'total_spikes':len(t),'late_spikes':int((t>=round(cfg['late_start_ms']/sim.dt)).sum()),
|
||||
'first_ms':float(t.min()*sim.dt) if len(t) else None,'last_ms':float(t.max()*sim.dt) if len(t) else None})
|
||||
for pop,idx in populations.items():
|
||||
poprows.append({'odor_index':oi,'seed':seed,'condition':condition,'population':pop,'neurons':len(idx),
|
||||
'total_spikes':int(r['counts'][idx].astype(np.int64).sum()),'active':int(np.count_nonzero(r['counts'][idx])),
|
||||
'late_spikes':int(r['late_counts'][idx].astype(np.int64).sum()),'late_active':int(np.count_nonzero(r['late_counts'][idx]))})
|
||||
comparisons.append({'odor_index':oi,'seed':seed,'condition':condition,
|
||||
'neurons_with_changed_counts':int(np.count_nonzero(r['counts']!=reference['counts'])),
|
||||
'max_abs_finalV_difference':float(np.abs(r['final_v'].astype(float)-reference['final_v']).max()),
|
||||
'KC_event_sequence_equal':arrhash(es,en)==arrhash(reference['kc_event_step'],reference['kc_event_node'])})
|
||||
save(exp/'regression/trials.json',reg)
|
||||
print(json.dumps({'odor_complete':oi,'seed':seed,'trials':len(stats)}),flush=True)
|
||||
sim.install=original_install;original_install(None)
|
||||
assert arrhash(sim.base.cpu().numpy())==original_base_hash
|
||||
assert np.array_equal(rt.weights.cpu().numpy(),original_overlay)
|
||||
save(exp/'regression/restore.json',{'immutable_memory_base_hash_exact':True,'frozen_APL_overlay_restored_exact':True})
|
||||
assert phase4s.evidence_regression(cfg,exp,rt,sim)
|
||||
for name,data in [('trials',stats),('watched-KC',watchrows),('populations',poprows),('paired-differences',comparisons)]:pd.DataFrame(data).to_parquet(exp/'comparisons'/f'{name}.parquet',index=False)
|
||||
save(exp/'protocol/input-hashes.json',inputs)
|
||||
save(exp/'audit/results.json',{'phase':'4Y','status':'COMPLETE-EXPERIMENTAL-HYPOTHESIS-AUDIT','trials':len(stats),'repeat_trials':6,
|
||||
'old_reference_reproductions':6,'frozen_reference_trials':5,'reference_pass':True,'all_finite':True,
|
||||
'DPM_neurons':len(dpm),'DPM_existing_edges':len(ids),'weight_training':False,'candidate_selected':False,
|
||||
'frozen_baseline_changed':False,'biological_sign_proven':False,'new_seed_count':1,'old_seed_count':1})
|
||||
save(exp/'performance/summary.json',{'wall_seconds_excluding_build':time.perf_counter()-start,'peak_VRAM_bytes':torch.cuda.max_memory_allocated(),
|
||||
'GPU':torch.cuda.get_device_name(0),'diagnostic_observation_overhead_included':True})
|
||||
|
||||
def seal(cfg,exp,model):
|
||||
pre=json.loads((exp/'protocol/preregistered.json').read_text());now=checks(cfg);hist=phase4p_finalize.integrity()
|
||||
ok=hist['all_static_checks_passed'] and now==json.loads((exp/'audit/preflight.json').read_text()) and all(digest(model/n)==h for n,h in pre['sources'].items())
|
||||
save(exp/'audit/frozen-integrity.json',{'pass':ok,'history':hist,'current':now});assert ok
|
||||
result=json.loads((exp/'audit/results.json').read_text());result['integrity_pass']=True;save(exp/'audit/results.json',result)
|
||||
w=pd.read_parquet(exp/'comparisons/watched-KC.parquet');t=pd.read_parquet(exp/'comparisons/trials.parquet')
|
||||
lines=['# Phase 4Y — independent DPM fast-current hypotheses','',
|
||||
'Status: COMPLETE-EXPERIMENTAL-HYPOTHESIS-AUDIT. No candidate selection, training, game integration or frozen baseline changes.',
|
||||
'All conditions use identical PN forcing for 120ms, then off to 1000ms. Six old odors, one old seed and one new seed. Not a population-level independent generalization gate.',
|
||||
'Factors 0 and -1 are diagnostic model interventions, not corrections to raw labels or validated receptor-specific physiology. Only actual outgoing DPM effective currents change in RAM. Graph and raw counts stay intact.','',
|
||||
'| KC | condition | late spike range (12 trials) |','|---:|---|---:|']
|
||||
for (node,condition),g in w.groupby(['neuron_idx','condition']):lines.append(f'| {node} | {condition} | {g.late_spikes.min()}–{g.late_spikes.max()} |')
|
||||
lines+=['','| condition | mean whole-brain spikes | mean late KC spikes | mean late active KC |','|---|---:|---:|---:|']
|
||||
for condition,g in t.groupby('condition'):lines.append(f'| {condition} | {g.whole_brain_spikes.mean():.2f} | {g.late_KC_spikes.mean():.2f} | {g.late_KC_active.mean():.2f} |')
|
||||
lines+=['','Exact controls: six Phase4V original trials, six repeats, five frozen reference trials, unchanged in-memory base hash and restored frozen overlay.',
|
||||
'Finite states do not establish biological stability. No calibrated runaway threshold, biological sign conclusion, receptor kinetics or learning success is inferred.',
|
||||
'Whole-brain intervention outcomes may differ from prior isolated replay because all feedback responds. Report this difference, do not select a condition by favorable activity.']
|
||||
(exp/'REPORT.md').write_text('\n'.join(lines)+'\n',encoding='utf-8')
|
||||
for root in (model,exp):
|
||||
rows=[{'path':p.relative_to(root).as_posix(),'sha256':digest(p),'bytes':p.stat().st_size} for p in sorted(root.rglob('*')) if p.is_file() and p.name not in ('manifest.json','checksums.sha256')]
|
||||
save(root/'manifest.json',{'files':rows});rows.append({'path':'manifest.json','sha256':digest(root/'manifest.json')})
|
||||
(root/'checksums.sha256').write_text(''.join(x['sha256']+' '+x['path']+'\n' for x in rows));q=verify_checksum(root,root/'checksums.sha256');assert q['passed'];print(json.dumps(q),flush=True)
|
||||
|
||||
if __name__=='__main__':
|
||||
p=argparse.ArgumentParser();p.add_argument('task',choices=['run','seal']);p.add_argument('--config',required=True);a=p.parse_args()
|
||||
cfg=yaml.safe_load(Path(a.config).read_text());(run if a.task=='run' else seal)(cfg,Path(cfg['experiment_root']),Path(cfg['model_root']))
|
||||
@@ -0,0 +1,39 @@
|
||||
phase: 4Y
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4y-dpm-hypothesis
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/dpm-hypothesis-v1
|
||||
prior_root: /data/digital-drosophila/experiments/phase4x-transmitter-provenance
|
||||
prior_model: /data/digital-drosophila/models/male-cns-lif/transmitter-audit-v1
|
||||
runtime_root: /data/digital-drosophila/models/male-cns-lif/deterministic-runtime-v2
|
||||
phase4p_root: /data/digital-drosophila/experiments/phase4p-deterministic-runtime
|
||||
input_model: /data/digital-drosophila/models/male-cns-lif/csus-audit-v1
|
||||
input_experiment: /data/digital-drosophila/experiments/phase4v-csus-semantics
|
||||
odor_indices: [0,1,2,3,4,5]
|
||||
seeds: [131101,141101]
|
||||
old_replay_seed: 131101
|
||||
conditions:
|
||||
- {name: frozen_reference, factor: 1.0}
|
||||
- {name: DPM_fast_current_off, factor: 0.0}
|
||||
- {name: DPM_fast_current_reversed, factor: -1.0}
|
||||
perturbation:
|
||||
exact_type: DPM
|
||||
expected_bodies: [11734,12569]
|
||||
scope: all and only outgoing effective currents of the two annotation-DPM neurons
|
||||
no_fast_current: diagnostic ablation; not deletion of neurons or anatomical edges and not a complete slow-modulator model
|
||||
reversed: sign-sensitivity hypothesis; not verified receptor-specific GABA/serotonin physiology
|
||||
input_condition: same original 120ms PN forcing, then off to 1000ms
|
||||
late_start_ms: 500
|
||||
watch_neuron_idx: [59832,78836,82312]
|
||||
repeat_check: first odor and both seeds, all three conditions
|
||||
scope:
|
||||
authorization: user delegated independent hypothesis-branch decision after explicit request for authorization
|
||||
diagnostic_only: true
|
||||
baseline_read_only: true
|
||||
weight_training: false
|
||||
slow_receptor_parameters_invented: false
|
||||
candidate_selection: false
|
||||
game_or_RL: false
|
||||
reserved_novel_odors: untouched
|
||||
interpretation:
|
||||
report: all conditions and both seeds; do not pick the most active or best-looking result
|
||||
causal_scope: effect of intervention on frozen mathematical model, not proof of biological neurotransmitter identity
|
||||
stability: report nonfinite state, brain spike counts, active fractions, and KC late activity; no biological runaway threshold inferred
|
||||
@@ -0,0 +1,106 @@
|
||||
"""Descriptive paired effects only; no condition selection or parameter fitting."""
|
||||
from pathlib import Path
|
||||
import json
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
import yaml
|
||||
from phase4r import save
|
||||
|
||||
cfg=yaml.safe_load(Path(__file__).with_name('phase4y.yaml').read_text());exp=Path(cfg['experiment_root'])
|
||||
assert not (exp/'checksums.sha256').exists(),'Sealed output is immutable'
|
||||
w=pd.read_parquet(exp/'comparisons/watched-KC.parquet');t=pd.read_parquet(exp/'comparisons/trials.parquet')
|
||||
p=pd.read_parquet(exp/'comparisons/populations.parquet');x=pd.read_parquet(Path(cfg['prior_root'])/'replay/results.parquet')
|
||||
inputrows=[]
|
||||
for _,r in t.iterrows():
|
||||
obs=pd.read_parquet(exp/'observations'/f'{r.odor_index}-{r.seed}-{r.condition}.parquet').iloc[-1]
|
||||
inputrows.append({'odor_index':int(r.odor_index),'seed':int(r.seed),'condition':r.condition,
|
||||
'requested':int(obs.PN_requested_cumulative),'accepted_new_spikes':int(obs.PN_accepted_cumulative),
|
||||
'preexisting_spikes':int(obs.PN_preexisting_cumulative),'refractory_blocked':int(obs.PN_blocked_cumulative)})
|
||||
inp=pd.DataFrame(inputrows);assert (inp.groupby(['odor_index','seed']).requested.nunique()==1).all()
|
||||
assert (inp.requested==inp.accepted_new_spikes+inp.preexisting_spikes+inp.refractory_blocked).all()
|
||||
inp.to_parquet(exp/'comparisons/input-request-vs-acceptance.parquet',index=False)
|
||||
save(exp/'audit/input-matching.json',{'scheduled_requests_matched_all_pairs':True,'partition_exact':True,
|
||||
'pairs_with_changed_accepted_count':int((inp.groupby(['odor_index','seed']).accepted_new_spikes.nunique()>1).sum()),
|
||||
'interpretation':'identical exogenous requests, not guaranteed identical accepted PN spikes; acceptance depends on endogenous refractory/spike state through the unchanged force interface'})
|
||||
# Reporting-only schema correction. Preserve original tables and the registered run source.
|
||||
# These exact superclass values are used by the frozen Phase4C propagation summaries.
|
||||
meta_path=Path('/data/digital-drosophila/datasets/male-cns-v1.0/processed/neuron-only/neuron-index/unified-metadata.parquet')
|
||||
sup=pd.read_parquet(meta_path,columns=['neuron_idx','annotation_superclass']).sort_values('neuron_idx')
|
||||
definitions={'descending':['descending_neuron','descending_neuron_tbc'],'motor':['vnc_motor','cb_motor']}
|
||||
corrected=[]
|
||||
for _,r in t.iterrows():
|
||||
name=f'{r.odor_index}-{r.seed}-{r.condition}'
|
||||
with np.load(exp/'states'/f'{name}.npz') as z:
|
||||
for pop,values in definitions.items():
|
||||
idx=np.flatnonzero(sup.annotation_superclass.isin(values).to_numpy());assert len(idx)>0
|
||||
c=z['counts'][idx];l=z['late_counts'][idx]
|
||||
corrected.append({'odor_index':int(r.odor_index),'seed':int(r.seed),'condition':r.condition,'population':pop,
|
||||
'neurons':len(idx),'total_spikes':int(c.astype(np.int64).sum()),'active':int(np.count_nonzero(c)),
|
||||
'late_spikes':int(l.astype(np.int64).sum()),'late_active':int(np.count_nonzero(l))})
|
||||
p=pd.concat([p[~p.population.isin(definitions)],pd.DataFrame(corrected)],ignore_index=True)
|
||||
p.to_parquet(exp/'comparisons/populations-corrected.parquet',index=False)
|
||||
save(exp/'audit/population-schema-correction.json',{'scope':'report aggregation only, no dynamics or registered conditions changed',
|
||||
'issue':'initial literal superclass descending/motor matches zero rows; those zero-population rows are invalid readouts, not evidence of silence',
|
||||
'definitions_from':'frozen phase4c.py summary definitions, verified against actual metadata',
|
||||
'corrected_values':definitions,'corrected_counts':{k:int(sup.annotation_superclass.isin(v).sum()) for k,v in definitions.items()},
|
||||
'source':str(meta_path),'original_tables_preserved':True,'recomputed_from':'saved complete per-neuron total and late counts','simulation_rerun':False})
|
||||
(exp/'READ-ME-FIRST.md').write_text(
|
||||
'# Phase 4Y evidence routing\n\n'
|
||||
'Use audit/MECHANISM.md and comparisons/populations-corrected.parquet for population readouts.\n\n'
|
||||
'The original comparisons/populations.parquet descending/motor rows used invalid literal superclass names and matched zero neurons. '
|
||||
'Those rows are preserved for audit but must not be interpreted as no activity. Corrected rows use the frozen Phase4C definitions '
|
||||
'(descending_neuron/descending_neuron_tbc; vnc_motor/cb_motor) and were recomputed from saved complete counts, without rerunning dynamics. '
|
||||
'See audit/population-schema-correction.json. Other original population rows are unchanged.\n\n'
|
||||
'All three hypotheses are reported. None is promoted to a learned or biological baseline.\n',encoding='utf-8')
|
||||
effects=[];isolated=[]
|
||||
for (oi,seed,node),g in w.groupby(['odor_index','seed','neuron_idx']):
|
||||
baseline=int(g.loc[g.condition.eq('frozen_reference'),'late_spikes'].iloc[0])
|
||||
for _,r in g.iterrows():
|
||||
effects.append({'odor_index':int(oi),'seed':int(seed),'neuron_idx':int(node),'condition':r.condition,
|
||||
'baseline_late_spikes':baseline,'condition_late_spikes':int(r.late_spikes),'paired_change':int(r.late_spikes)-baseline})
|
||||
if seed==cfg['old_replay_seed']:
|
||||
name=f'{oi}-{seed}-offset';z=x[(x.trial==name)&(x.neuron_idx==node)]
|
||||
ib=int(z.loc[z['mode'].eq('base'),'late_spikes'].iloc[0]);io=int(z.loc[z['mode'].eq('omit_DPM_annotation'),'late_spikes'].iloc[0])
|
||||
off=int(g.loc[g.condition.eq('DPM_fast_current_off'),'late_spikes'].iloc[0]);assert ib==baseline
|
||||
isolated.append({'odor_index':int(oi),'neuron_idx':int(node),'baseline_late_spikes':baseline,
|
||||
'isolated_DPM_off_late_spikes':io,'whole_brain_DPM_off_late_spikes':off,
|
||||
'whole_minus_isolated':off-io,'interpretation':'different feedback assumptions; not a numerical regression failure'})
|
||||
pd.DataFrame(effects).to_parquet(exp/'comparisons/watched-paired-effects.parquet',index=False)
|
||||
pd.DataFrame(isolated).to_parquet(exp/'comparisons/isolated-vs-whole-brain.parquet',index=False)
|
||||
new=[];loss=[]
|
||||
metadata=pd.read_parquet(Path(cfg['prior_root'])/'roles/neuron-evidence-axes.parquet').set_index('neuron_idx')
|
||||
for (oi,seed),group in t.groupby(['odor_index','seed']):
|
||||
def late_set(condition):
|
||||
e=pd.read_parquet(exp/'events'/f'{oi}-{seed}-{condition}.parquet');return set(map(int,e.loc[e.step>=5000,'neuron_idx']))
|
||||
ref=late_set('frozen_reference')
|
||||
for condition in ('DPM_fast_current_off','DPM_fast_current_reversed'):
|
||||
s=late_set(condition)
|
||||
for label,nodes in [('new',s-ref),('lost',ref-s)]:
|
||||
for node in sorted(nodes):
|
||||
row={'odor_index':int(oi),'seed':int(seed),'condition':condition,'change':label,'neuron_idx':node,
|
||||
'body_id':int(metadata.loc[node,'body_id']),'annotation_type':metadata.loc[node,'annotation_type']}
|
||||
(new if label=='new' else loss).append(row)
|
||||
pd.DataFrame(new+loss,columns=['odor_index','seed','condition','change','neuron_idx','body_id','annotation_type']).to_parquet(exp/'comparisons/late-KC-membership-changes.parquet',index=False)
|
||||
ep=pd.DataFrame(effects);summary=[]
|
||||
for (node,condition),g in ep.groupby(['neuron_idx','condition']):
|
||||
summary.append({'neuron_idx':int(node),'condition':condition,'n':len(g),'late_min':int(g.condition_late_spikes.min()),
|
||||
'late_max':int(g.condition_late_spikes.max()),'change_min':int(g.paired_change.min()),'change_max':int(g.paired_change.max()),
|
||||
'decreases':int((g.paired_change<0).sum()),'increases':int((g.paired_change>0).sum()),'unchanged':int((g.paired_change==0).sum())})
|
||||
save(exp/'audit/descriptive-effects.json',{'scope':'post-run descriptive summaries; no promotion or new gate',
|
||||
'watched':summary,'new_late_KC_memberships':len(new),'lost_late_KC_memberships':len(loss)})
|
||||
lines=['# Phase 4Y — paired whole-brain effects','',
|
||||
'All branches are diagnostic hypotheses; none is selected as a better biological model.','',
|
||||
'| KC | condition | late spike range | paired change range | decreases / unchanged / increases |','|---:|---|---:|---:|---:|']
|
||||
for r in summary:lines.append(f"| {r['neuron_idx']} | {r['condition']} | {r['late_min']}–{r['late_max']} | {r['change_min']}–{r['change_max']} | {r['decreases']} / {r['unchanged']} / {r['increases']} |")
|
||||
lines+=['','| KC | isolated DPM-off range | whole-brain DPM-off range (same old seed) |','|---:|---:|---:|']
|
||||
for node,g in pd.DataFrame(isolated).groupby('neuron_idx'):
|
||||
lines.append(f'| {node} | {g.isolated_DPM_off_late_spikes.min()}–{g.isolated_DPM_off_late_spikes.max()} | {g.whole_brain_DPM_off_late_spikes.min()}–{g.whole_brain_DPM_off_late_spikes.max()} |')
|
||||
lines+=['','| population | condition | mean late spikes | mean late active neurons |','|---|---|---:|---:|']
|
||||
for (pop,condition),g in p.groupby(['population','condition']):
|
||||
lines.append(f'| {pop} | {condition} | {g.late_spikes.mean():.2f} | {g.late_active.mean():.2f} |')
|
||||
lines+=['','Whole-brain feedback is allowed to respond; isolated replay holds the other recorded arrivals fixed. A difference between them is the target of the experiment, not grounds for choosing one result.',
|
||||
'Input matching is at the scheduled request level. Accepted new PN spikes can differ through feedback-dependent refractory state; see input-request-vs-acceptance.parquet. Do not claim identical delivered PN spike trains.',
|
||||
'Descending/motor rows here use populations-corrected.parquet, reconstructed from saved full counts and verified actual superclass names. The original zero-population rows are retained but invalid; see audit/population-schema-correction.json.',
|
||||
'No learning, no reward events, no parameter fitting, no promotion; smaller late activity is not itself success. Biological signs and long-term stability remain unproven.']
|
||||
(exp/'audit/MECHANISM.md').write_text('\n'.join(lines)+'\n',encoding='utf-8')
|
||||
print('\n'.join(lines[:20]))
|
||||
@@ -0,0 +1,47 @@
|
||||
{
|
||||
"file_count": 8,
|
||||
"files": [
|
||||
{
|
||||
"bytes": 228,
|
||||
"path": "VERSION.json",
|
||||
"sha256": "e0b478bc714ff90bd398fae7a726bee0e8dc9975f8d967b9fcf6d360fcb07100"
|
||||
},
|
||||
{
|
||||
"bytes": 19444,
|
||||
"path": "__pycache__/brain_runtime.cpython-310.pyc",
|
||||
"sha256": "d6df50e3617ec0f7893aa30b94b3dbd55359d3f1f10cf960e825e35d96cea097"
|
||||
},
|
||||
{
|
||||
"bytes": 1807,
|
||||
"path": "__pycache__/frozen_interface.cpython-310.pyc",
|
||||
"sha256": "c34e3584aab378b2394451c64069f8b928503c45308580ed17fb5ea1d732556b"
|
||||
},
|
||||
{
|
||||
"bytes": 38424,
|
||||
"path": "__pycache__/phase4g.cpython-310.pyc",
|
||||
"sha256": "76ccac2f2ed2135dfda5d7b21307aaff5fd3f8db3dd607c36741b070db32b295"
|
||||
},
|
||||
{
|
||||
"bytes": 23822,
|
||||
"path": "brain_runtime.py",
|
||||
"sha256": "1c4e3e6f06708bfec30b24e92297f320b2b8b87ebbbe6da501f5f2db4f74762a"
|
||||
},
|
||||
{
|
||||
"bytes": 1229,
|
||||
"path": "frozen_interface.py",
|
||||
"sha256": "4e4142cd5e866cfdbc9d56ffdff5eac9081a79e00c1658611f326447013b8fdb"
|
||||
},
|
||||
{
|
||||
"bytes": 40435,
|
||||
"path": "phase4g.py",
|
||||
"sha256": "c82ff049336a4815690d4060f001c77266fe70d9e62ec6a940741d9fa9721e4f"
|
||||
},
|
||||
{
|
||||
"bytes": 4795,
|
||||
"path": "phase4g.yaml",
|
||||
"sha256": "678798511cc14181d22a2085dedb412b44a96a966398678c12c27a8c561be042"
|
||||
}
|
||||
],
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/frozen-interface-v1",
|
||||
"total_bytes": 130184
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"candidate_name": null,
|
||||
"learned_state_root": "/data/digital-drosophila/models/learned/phase4g",
|
||||
"male_cns_weights_fixed": true,
|
||||
"name": "MaleCNS-FrozenInterface-v1",
|
||||
"status": "FAIL-NO-FROZEN-INTERFACE-CANDIDATE"
|
||||
}
|
||||
@@ -0,0 +1,415 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stable game-agnostic Brain I/O API for MaleCNS-LIF-runtime-v1."""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
import numpy as np
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
|
||||
API_VERSION = "1.0.0"
|
||||
INPUT_BATCH_VERSION = "1.0.0"
|
||||
ACTIVITY_STREAM_VERSION = "1.0.0"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InputEvent:
|
||||
target_neuron_idx: np.ndarray
|
||||
stimulation_mode: str
|
||||
value_kind: str
|
||||
value: float | np.ndarray
|
||||
start_ms: float
|
||||
duration_ms: float
|
||||
source_id: str
|
||||
channel_id: str
|
||||
amplitude_mv: float = 0.0
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def validate(self, neuron_count: int, dt_ms: float) -> None:
|
||||
target = np.asarray(self.target_neuron_idx)
|
||||
if target.ndim != 1 or target.dtype.kind not in "iu":
|
||||
raise ValueError("target_neuron_idx must be a one-dimensional integer array")
|
||||
if len(target) and (int(target.min()) < 0 or int(target.max()) >= neuron_count):
|
||||
raise ValueError("target neuron_idx outside canonical range")
|
||||
if len(np.unique(target)) != len(target):
|
||||
raise ValueError("target neuron_idx values must be unique")
|
||||
if self.stimulation_mode not in {"spike_injection", "external_synaptic_drive"}:
|
||||
raise ValueError(f"unsupported stimulation_mode {self.stimulation_mode!r}")
|
||||
if self.value_kind not in {"binary", "scalar_intensity", "poisson_rate_hz", "signed_modulation"}:
|
||||
raise ValueError(f"unsupported value_kind {self.value_kind!r}")
|
||||
if self.start_ms < 0 or self.duration_ms <= 0:
|
||||
raise ValueError("timestamps/duration must be non-negative/positive")
|
||||
for x in (self.start_ms, self.duration_ms):
|
||||
if not math.isclose(x / dt_ms, round(x / dt_ms), abs_tol=1e-8):
|
||||
raise ValueError(f"{x} ms is not aligned to dt={dt_ms} ms")
|
||||
value = np.asarray(self.value)
|
||||
if value.ndim > 1 or (value.ndim == 1 and len(value) not in (1, len(target))):
|
||||
raise ValueError("value must be scalar or match target count")
|
||||
if self.value_kind == "poisson_rate_hz" and np.any(value < 0):
|
||||
raise ValueError("Poisson rate cannot be negative")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InputBatch:
|
||||
events: tuple[InputEvent, ...]
|
||||
batch_id: str
|
||||
version: str = INPUT_BATCH_VERSION
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def validate(self, neuron_count: int, dt_ms: float) -> None:
|
||||
if self.version != INPUT_BATCH_VERSION:
|
||||
raise ValueError(f"InputBatch version must be {INPUT_BATCH_VERSION}")
|
||||
for event in self.events:
|
||||
event.validate(neuron_count, dt_ms)
|
||||
|
||||
|
||||
class VisualAdapterV1:
|
||||
"""Maps abstract channels to audited visual hierarchy populations."""
|
||||
|
||||
def __init__(self, config: dict[str, Any], phase4c_root: Path):
|
||||
self.config = config
|
||||
self.phase4c_root = phase4c_root
|
||||
self.modes: dict[str, np.ndarray] = {}
|
||||
for mode, spec in config["visual_adapter"]["modes"].items():
|
||||
population = spec["hierarchy_population"]
|
||||
path = phase4c_root / "visual-audit" / f"hierarchy-{population}.parquet"
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
self.modes[mode] = pq.read_table(path, columns=["neuron_idx"])["neuron_idx"].to_numpy().astype(np.int64)
|
||||
|
||||
def population(self, mode: str) -> np.ndarray:
|
||||
if mode not in self.modes:
|
||||
raise KeyError(mode)
|
||||
return self.modes[mode].copy()
|
||||
|
||||
def encode(
|
||||
self, mode: str, channel_values: np.ndarray, start_ms: float, duration_ms: float,
|
||||
source_id: str, channel_id: str, value_kind: str = "poisson_rate_hz",
|
||||
) -> InputBatch:
|
||||
if mode not in self.modes:
|
||||
raise KeyError(mode)
|
||||
spec = self.config["visual_adapter"]["modes"][mode]
|
||||
targets = self.modes[mode]
|
||||
values = np.asarray(channel_values, dtype=np.float32)
|
||||
if values.shape != (len(targets),):
|
||||
raise ValueError(f"visual channel must have shape {(len(targets),)}, got {values.shape}")
|
||||
selected = np.flatnonzero(values != 0).astype(np.int64)
|
||||
target = targets[selected]
|
||||
selected_values = values[selected]
|
||||
event = InputEvent(
|
||||
target_neuron_idx=target,
|
||||
stimulation_mode=spec["input_mode"],
|
||||
value_kind=value_kind,
|
||||
value=selected_values,
|
||||
amplitude_mv=float(self.config["visual_adapter"]["external_drive_mv"]),
|
||||
start_ms=start_ms,
|
||||
duration_ms=duration_ms,
|
||||
source_id=source_id,
|
||||
channel_id=channel_id,
|
||||
metadata={
|
||||
"adapter": self.config["names"]["visual_adapter"],
|
||||
"adapter_mode": mode,
|
||||
"biological_fidelity": spec["biological_fidelity"],
|
||||
"connectome_weights_modified": False,
|
||||
},
|
||||
)
|
||||
return InputBatch(events=(event,), batch_id=f"{source_id}:{channel_id}:{start_ms}", metadata={"adapter_mode": mode})
|
||||
|
||||
|
||||
class BrainRuntime:
|
||||
"""Persistent exact-state BrainRuntime API. It has no game-specific dependency."""
|
||||
|
||||
def __init__(self, config_path: Path, extension_build_dir: Path):
|
||||
self.config_path = config_path.resolve()
|
||||
self.cfg = yaml.safe_load(self.config_path.read_text(encoding="utf-8"))
|
||||
self.dt = float(self.cfg["model"]["dt_ms"])
|
||||
self.n = 166700
|
||||
self.model_root = Path(self.cfg["paths"]["model_root"])
|
||||
self.runtime_root = Path(self.cfg["paths"]["runtime_root"])
|
||||
self.input_root = Path(self.cfg["paths"]["connectome_root"])
|
||||
runtime_module_dir = str(self.runtime_root)
|
||||
if runtime_module_dir not in sys.path:
|
||||
sys.path.insert(0, runtime_module_dir)
|
||||
import phase4c as runtime_module
|
||||
self.runtime_module = runtime_module
|
||||
p4cfg = yaml.safe_load((self.runtime_root / "phase4c.yaml").read_text(encoding="utf-8"))
|
||||
self.paths = runtime_module.exact_paths(p4cfg)
|
||||
self.indptr_np, self.indices_np, self.raw_np = runtime_module.p4b.validate_graph(p4cfg, self.paths)
|
||||
self.table = runtime_module.full_metadata(self.paths["metadata"])
|
||||
self.population_masks = runtime_module.populations(self.table)
|
||||
if not torch.cuda.is_available() or torch.cuda.device_count() != 1:
|
||||
raise RuntimeError("BrainRuntime requires exactly one visible CUDA GPU")
|
||||
self.device = torch.device("cuda:0")
|
||||
props = torch.cuda.get_device_properties(self.device)
|
||||
if self.cfg["device"]["required_name_substring"] not in props.name:
|
||||
raise RuntimeError(f"required RTX 4090, got {props.name}")
|
||||
extension_build_dir.mkdir(parents=True, exist_ok=True)
|
||||
build_experiment_root = extension_build_dir.parent.parent
|
||||
(build_experiment_root / "compiler-reports/extension-build").mkdir(parents=True, exist_ok=True)
|
||||
self.extension, report = runtime_module.build_extension(self.runtime_root, build_experiment_root)
|
||||
if self.extension is None:
|
||||
raise RuntimeError(f"runtime-v1 extension unavailable: {report}")
|
||||
base = runtime_module.Runtime(p4cfg, self.table, self.indptr_np, self.indices_np, self.raw_np, self.device, self.extension)
|
||||
self.indptr, self.indices, self.weights = base.indptr, base.indices, base.weights
|
||||
self.signs, self.superclass = base.signs, base.superclass
|
||||
self.neuron_type = np.asarray(self.table["annotation_type"].to_pylist(), dtype=object)
|
||||
self.neuron_class = np.asarray(self.table["annotation_class"].to_pylist(), dtype=object)
|
||||
self.body_id = self.table["body_id"].to_numpy().astype(np.uint64)
|
||||
m = self.cfg["model"]
|
||||
self.delay_steps = int(round(float(m["synaptic_delay_ms"]) / self.dt))
|
||||
self.queue_len = self.delay_steps + 1
|
||||
self.refractory_steps = int(round(float(m["refractory_ms"]) / self.dt))
|
||||
self.alpha = self.dt / float(m["membrane_tau_ms"])
|
||||
self.g_decay = 1.0 - self.dt / float(m["synaptic_tau_ms"])
|
||||
self.rest = float(m["resting_potential_mv"])
|
||||
self.reset_value = float(m["reset_potential_mv"])
|
||||
self.threshold = float(m["threshold_mv"])
|
||||
self.output_populations = self._output_populations()
|
||||
self.reset(0)
|
||||
|
||||
def _output_populations(self) -> dict[str, np.ndarray]:
|
||||
s = self.superclass
|
||||
return {
|
||||
"descending": np.flatnonzero(np.isin(s, ["descending_neuron", "descending_neuron_tbc"])).astype(np.int32),
|
||||
"motor_related": np.flatnonzero(np.isin(s, ["vnc_motor", "cb_motor"])).astype(np.int32),
|
||||
"ascending": np.flatnonzero(np.isin(s, ["ascending_neuron", "sensory_ascending", "sensory_ascending_tbc", "efferent_ascending"])).astype(np.int32),
|
||||
}
|
||||
|
||||
def reset(self, seed: int, initial_state: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
self.seed = int(seed)
|
||||
self.generator = torch.Generator(device=self.device); self.generator.manual_seed(self.seed)
|
||||
self.v = torch.full((self.n,), self.rest, dtype=torch.float32, device=self.device)
|
||||
self.g = torch.zeros(self.n, dtype=torch.float32, device=self.device)
|
||||
self.refractory = torch.zeros(self.n, dtype=torch.int32, device=self.device)
|
||||
self.delay_queue = torch.zeros((self.queue_len, self.n), dtype=torch.float32, device=self.device)
|
||||
self.spike_counts = torch.zeros(self.n, dtype=torch.int32, device=self.device)
|
||||
self.simulation_step = 0; self.sequence_id = 0
|
||||
self.spike_steps: list[np.ndarray] = []; self.spike_nodes: list[np.ndarray] = []
|
||||
self.spike_origins: list[np.ndarray] = []
|
||||
self.drive_steps: list[np.ndarray] = []; self.drive_nodes: list[np.ndarray] = []
|
||||
self.drive_values: list[np.ndarray] = []; self.input_audit: list[dict[str, Any]] = []
|
||||
if initial_state is not None:
|
||||
self.restore_state(initial_state)
|
||||
return self.get_state_summary()
|
||||
|
||||
@property
|
||||
def simulation_time_ms(self) -> float:
|
||||
return self.simulation_step * self.dt
|
||||
|
||||
def _event_values(self, event: InputEvent, targets: torch.Tensor) -> torch.Tensor:
|
||||
value = np.asarray(event.value, dtype=np.float32)
|
||||
if value.ndim == 0:
|
||||
return torch.full((len(targets),), float(value), dtype=torch.float32, device=self.device)
|
||||
return torch.as_tensor(value, dtype=torch.float32, device=self.device)
|
||||
|
||||
def _active(self, event: InputEvent, time_ms: float) -> bool:
|
||||
return event.start_ms - 1e-9 <= time_ms < event.start_ms + event.duration_ms - 1e-9
|
||||
|
||||
def _hybrid_edges(self, sources: torch.Tensor) -> torch.Tensor:
|
||||
starts = self.indptr[sources]
|
||||
degrees = self.indptr[sources + 1] - starts
|
||||
cumulative = torch.cumsum(degrees, 0)
|
||||
total = int(cumulative[-1].item()) if cumulative.numel() else 0
|
||||
output = torch.empty(total, dtype=torch.int64, device=self.device)
|
||||
if total:
|
||||
self.extension.fill_edges(self.indptr, sources, cumulative - degrees, output)
|
||||
return output
|
||||
|
||||
def step(self, input_batch: InputBatch | None, biological_ms: float) -> dict[str, Any]:
|
||||
steps = int(round(float(biological_ms) / self.dt))
|
||||
if not math.isclose(steps * self.dt, float(biological_ms), abs_tol=1e-8) or steps <= 0:
|
||||
raise ValueError("biological_ms must be a positive multiple of dt")
|
||||
batch = input_batch or InputBatch(events=(), batch_id="zero-input")
|
||||
batch.validate(self.n, self.dt)
|
||||
batch_start = self.simulation_time_ms
|
||||
prepared = [(e, torch.as_tensor(e.target_neuron_idx, dtype=torch.int64, device=self.device)) for e in batch.events]
|
||||
for event in batch.events:
|
||||
self.input_audit.append({
|
||||
"batch_id": batch.batch_id, "source_id": event.source_id, "channel_id": event.channel_id,
|
||||
"mode": event.stimulation_mode, "value_kind": event.value_kind,
|
||||
"start_ms": event.start_ms, "duration_ms": event.duration_ms,
|
||||
"target_count": int(len(event.target_neuron_idx)), "metadata": event.metadata,
|
||||
"external_drive_is_connectome_transmission": False,
|
||||
})
|
||||
start_event_count = sum(len(x) for x in self.spike_nodes)
|
||||
start_drive_count = sum(len(x) for x in self.drive_nodes)
|
||||
for _ in range(steps):
|
||||
global_step = self.simulation_step; time_ms = self.simulation_time_ms
|
||||
slot = global_step % self.queue_len
|
||||
self.g.mul_(self.g_decay).add_(self.delay_queue[slot]); self.delay_queue[slot].zero_()
|
||||
spike_requests: list[tuple[InputEvent, torch.Tensor, torch.Tensor]] = []
|
||||
for event, targets in prepared:
|
||||
if not self._active(event, time_ms) or targets.numel() == 0:
|
||||
continue
|
||||
values = self._event_values(event, targets)
|
||||
if event.value_kind == "poisson_rate_hz":
|
||||
selected = torch.rand(len(targets), generator=self.generator, device=self.device) < (values * self.dt / 1000.0)
|
||||
elif event.value_kind == "binary":
|
||||
selected = values != 0
|
||||
elif event.value_kind == "scalar_intensity":
|
||||
selected = values != 0
|
||||
else: # signed_modulation
|
||||
selected = torch.rand(len(targets), generator=self.generator, device=self.device) < (torch.abs(values) * self.dt / 1000.0)
|
||||
if event.stimulation_mode == "external_synaptic_drive":
|
||||
chosen = targets[selected]
|
||||
if chosen.numel():
|
||||
signed_value = values[selected] * float(event.amplitude_mv)
|
||||
if event.value_kind == "poisson_rate_hz":
|
||||
signed_value = torch.full_like(signed_value, float(event.amplitude_mv))
|
||||
self.g.index_add_(0, chosen, signed_value)
|
||||
node_np = chosen.cpu().numpy().astype(np.int32, copy=True)
|
||||
val_np = signed_value.cpu().numpy().astype(np.float32, copy=True)
|
||||
self.drive_steps.append(np.full(len(node_np), global_step, dtype=np.int32))
|
||||
self.drive_nodes.append(node_np); self.drive_values.append(val_np)
|
||||
else:
|
||||
spike_requests.append((event, targets, selected))
|
||||
self.refractory.clamp_min_(1).sub_(1)
|
||||
available = self.refractory == 0
|
||||
self.v[available] += self.alpha * (self.g[available] - (self.v[available] - self.rest))
|
||||
self.v[~available] = self.reset_value
|
||||
natural = available & (self.v >= self.threshold)
|
||||
spikes = natural.clone()
|
||||
injected = torch.zeros(self.n, dtype=torch.bool, device=self.device)
|
||||
for _event, targets, selected in spike_requests:
|
||||
accepted = selected & available[targets]
|
||||
chosen = targets[accepted]
|
||||
if chosen.numel():
|
||||
spikes[chosen] = True; injected[chosen] = ~natural[chosen]
|
||||
sources = torch.nonzero(spikes, as_tuple=False).flatten()
|
||||
self.spike_counts.add_(spikes.to(torch.int32))
|
||||
if sources.numel():
|
||||
self.v[sources] = self.reset_value; self.refractory[sources] = self.refractory_steps
|
||||
edge_ids = self._hybrid_edges(sources)
|
||||
if edge_ids.numel():
|
||||
self.delay_queue[(global_step + self.delay_steps) % self.queue_len].scatter_add_(0, self.indices[edge_ids].long(), self.weights[edge_ids])
|
||||
src_np = sources.cpu().numpy().astype(np.int32, copy=True)
|
||||
origin_np = np.where(injected[sources].cpu().numpy(), "external_spike_injection", "network_spike")
|
||||
self.spike_steps.append(np.full(len(src_np), global_step, dtype=np.int32))
|
||||
self.spike_nodes.append(src_np); self.spike_origins.append(origin_np)
|
||||
self.simulation_step += 1
|
||||
torch.cuda.synchronize()
|
||||
self.sequence_id += 1
|
||||
return {
|
||||
"api_version": API_VERSION, "sequence_id": self.sequence_id,
|
||||
"batch_id": batch.batch_id, "start_time_ms": batch_start,
|
||||
"end_time_ms": self.simulation_time_ms, "biological_ms": biological_ms,
|
||||
"new_spikes": sum(len(x) for x in self.spike_nodes) - start_event_count,
|
||||
"new_external_drive_events": sum(len(x) for x in self.drive_nodes) - start_drive_count,
|
||||
}
|
||||
|
||||
def _events(self) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
steps = np.concatenate(self.spike_steps) if self.spike_steps else np.empty(0, np.int32)
|
||||
nodes = np.concatenate(self.spike_nodes) if self.spike_nodes else np.empty(0, np.int32)
|
||||
origins = np.concatenate(self.spike_origins) if self.spike_origins else np.empty(0, object)
|
||||
return steps, nodes, origins
|
||||
|
||||
def read_spikes(self, window_ms: float | None = None) -> dict[str, Any]:
|
||||
steps, nodes, origins = self._events()
|
||||
if window_ms is not None:
|
||||
threshold = self.simulation_step - int(round(window_ms / self.dt))
|
||||
mask = steps >= threshold; steps, nodes, origins = steps[mask], nodes[mask], origins[mask]
|
||||
return {
|
||||
"simulation_time_ms": self.simulation_time_ms,
|
||||
"step": steps, "timestamp_ms": steps.astype(np.float64) * self.dt,
|
||||
"neuron_idx": nodes, "origin": origins,
|
||||
}
|
||||
|
||||
def read_population(self, population_id: str, window_ms: float) -> dict[str, Any]:
|
||||
if population_id not in self.output_populations:
|
||||
raise KeyError(population_id)
|
||||
members = self.output_populations[population_id]
|
||||
events = self.read_spikes(window_ms)
|
||||
mask = np.isin(events["neuron_idx"], members)
|
||||
nodes = events["neuron_idx"][mask]
|
||||
counts = np.zeros(len(members), dtype=np.int32)
|
||||
if len(nodes):
|
||||
lookup = np.full(self.n, -1, dtype=np.int32); lookup[members] = np.arange(len(members), dtype=np.int32)
|
||||
np.add.at(counts, lookup[nodes], 1)
|
||||
seconds = window_ms / 1000.0
|
||||
return {
|
||||
"simulation_time_ms": self.simulation_time_ms, "population_id": population_id,
|
||||
"neuron_idx": members.copy(), "spike_counts": counts,
|
||||
"firing_rate_hz": counts.astype(np.float32) / seconds,
|
||||
"active_mask": counts > 0, "aggregate_rate_hz": float(counts.sum() / (len(members) * seconds)),
|
||||
"total_spikes": int(counts.sum()), "active_neurons": int(np.count_nonzero(counts)),
|
||||
}
|
||||
|
||||
def read_output(self, output_spec: dict[str, Any], window_ms: float) -> dict[str, Any]:
|
||||
population = output_spec.get("population", "descending")
|
||||
aggregation = output_spec.get("aggregation", "per-neuron")
|
||||
base = self.read_population(population, window_ms)
|
||||
members = base["neuron_idx"]; counts = base["spike_counts"]
|
||||
if aggregation == "per-neuron":
|
||||
return base
|
||||
if aggregation not in {"per-type", "per-class"}:
|
||||
raise ValueError("aggregation must be per-neuron, per-type, or per-class")
|
||||
labels = self.neuron_type if aggregation == "per-type" else self.neuron_class
|
||||
result: dict[str, dict[str, float | int]] = {}
|
||||
for label in sorted({"<null>" if labels[i] is None else str(labels[i]) for i in members}):
|
||||
mask = np.asarray([("<null>" if labels[i] is None else str(labels[i])) == label for i in members])
|
||||
result[label] = {"neuron_count": int(mask.sum()), "spike_count": int(counts[mask].sum()), "mean_rate_hz": float(base["firing_rate_hz"][mask].mean())}
|
||||
return {"simulation_time_ms": self.simulation_time_ms, "population_id": population, "aggregation": aggregation, "features": result}
|
||||
|
||||
def fixed_temporal_bins(self, population_id: str, bin_ms: float, window_ms: float) -> dict[str, Any]:
|
||||
if bin_ms not in self.cfg["readout"]["bins_ms"]:
|
||||
raise ValueError("unsupported fixed bin")
|
||||
ev = self.read_spikes(window_ms); members = self.output_populations[population_id]
|
||||
mask = np.isin(ev["neuron_idx"], members); times = ev["timestamp_ms"][mask]
|
||||
start = self.simulation_time_ms - window_ms
|
||||
edges = np.arange(start, self.simulation_time_ms + bin_ms + 1e-9, bin_ms)
|
||||
counts, _ = np.histogram(times, bins=edges)
|
||||
return {"simulation_time_ms": self.simulation_time_ms, "population_id": population_id, "bin_ms": bin_ms, "bin_start_ms": edges[:-1], "spike_counts": counts.astype(np.int32)}
|
||||
|
||||
def get_state_summary(self) -> dict[str, Any]:
|
||||
return {
|
||||
"api_version": API_VERSION, "runtime": self.cfg["names"]["runtime"],
|
||||
"simulation_step": self.simulation_step, "simulation_time_ms": self.simulation_time_ms,
|
||||
"sequence_id": self.sequence_id, "seed": self.seed,
|
||||
"neurons": self.n, "edges": int(self.indices.numel()),
|
||||
"active_neurons_total": int(torch.count_nonzero(self.spike_counts).item()),
|
||||
"spikes_total": int(self.spike_counts.sum().item()),
|
||||
"external_drive_events_total": sum(len(x) for x in self.drive_nodes),
|
||||
}
|
||||
|
||||
def checkpoint_state(self) -> dict[str, Any]:
|
||||
return {
|
||||
"api_version": API_VERSION, "seed": self.seed, "simulation_step": self.simulation_step,
|
||||
"sequence_id": self.sequence_id, "generator_state": self.generator.get_state().cpu(),
|
||||
"v": self.v.clone(), "g": self.g.clone(), "refractory": self.refractory.clone(),
|
||||
"delay_queue": self.delay_queue.clone(), "spike_counts": self.spike_counts.clone(),
|
||||
"spike_steps": copy.deepcopy(self.spike_steps), "spike_nodes": copy.deepcopy(self.spike_nodes),
|
||||
"spike_origins": copy.deepcopy(self.spike_origins), "drive_steps": copy.deepcopy(self.drive_steps),
|
||||
"drive_nodes": copy.deepcopy(self.drive_nodes), "drive_values": copy.deepcopy(self.drive_values),
|
||||
"input_audit": copy.deepcopy(self.input_audit),
|
||||
}
|
||||
|
||||
def restore_state(self, state: dict[str, Any]) -> dict[str, Any]:
|
||||
if state.get("api_version") != API_VERSION:
|
||||
raise ValueError("checkpoint API version mismatch")
|
||||
self.seed = int(state["seed"]); self.simulation_step = int(state["simulation_step"]); self.sequence_id = int(state["sequence_id"])
|
||||
self.generator.set_state(state["generator_state"].cpu())
|
||||
for name in ("v", "g", "refractory", "delay_queue", "spike_counts"):
|
||||
setattr(self, name, state[name].clone().to(self.device))
|
||||
for name in ("spike_steps", "spike_nodes", "spike_origins", "drive_steps", "drive_nodes", "drive_values", "input_audit"):
|
||||
setattr(self, name, copy.deepcopy(state[name]))
|
||||
return self.get_state_summary()
|
||||
|
||||
def activity_packet(self, window_ms: float, input_metadata: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
ev = self.read_spikes(window_ms)
|
||||
aggregates = {name: {k: v for k, v in self.read_population(name, window_ms).items() if k in ("active_neurons", "total_spikes", "aggregate_rate_hz")} for name in self.output_populations}
|
||||
return {
|
||||
"contract_version": ACTIVITY_STREAM_VERSION, "simulation_time_ms": self.simulation_time_ms,
|
||||
"sequence_id": self.sequence_id, "window_ms": window_ms,
|
||||
"spike_neuron_idx": ev["neuron_idx"].tolist(), "spike_timestamp_ms": ev["timestamp_ms"].tolist(),
|
||||
"population_aggregates": aggregates, "active_neuron_count": int(len(np.unique(ev["neuron_idx"]))),
|
||||
"input_channel_metadata": input_metadata or {}, "representation": "event-only",
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
138af9d746caacd5d65d3d5091e98754a704b768fe3aacf0123cf3c250910500 MANIFEST.json
|
||||
e0b478bc714ff90bd398fae7a726bee0e8dc9975f8d967b9fcf6d360fcb07100 VERSION.json
|
||||
d6df50e3617ec0f7893aa30b94b3dbd55359d3f1f10cf960e825e35d96cea097 __pycache__/brain_runtime.cpython-310.pyc
|
||||
c34e3584aab378b2394451c64069f8b928503c45308580ed17fb5ea1d732556b __pycache__/frozen_interface.cpython-310.pyc
|
||||
76ccac2f2ed2135dfda5d7b21307aaff5fd3f8db3dd607c36741b070db32b295 __pycache__/phase4g.cpython-310.pyc
|
||||
1c4e3e6f06708bfec30b24e92297f320b2b8b87ebbbe6da501f5f2db4f74762a brain_runtime.py
|
||||
4e4142cd5e866cfdbc9d56ffdff5eac9081a79e00c1658611f326447013b8fdb frozen_interface.py
|
||||
c82ff049336a4815690d4060f001c77266fe70d9e62ec6a940741d9fa9721e4f phase4g.py
|
||||
678798511cc14181d22a2085dedb412b44a96a966398678c12c27a8c561be042 phase4g.yaml
|
||||
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Game-agnostic abstract action-logit API for Phase 4G learned adapters."""
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AbstractActionOutput:
|
||||
action_logits: np.ndarray
|
||||
|
||||
def __post_init__(self) -> None:
|
||||
value = np.asarray(self.action_logits, dtype=np.float32)
|
||||
if value.shape != (4,) or not np.isfinite(value).all():
|
||||
raise ValueError("action_logits must be finite float32[4]")
|
||||
object.__setattr__(self, "action_logits", value)
|
||||
|
||||
def deterministic_argmax(self) -> int:
|
||||
return int(np.argmax(self.action_logits))
|
||||
|
||||
def sample(self, seed: int, temperature: float = 1.0, epsilon: float = 0.0) -> int:
|
||||
if temperature <= 0 or not 0 <= epsilon <= 1:
|
||||
raise ValueError("temperature must be positive and epsilon in [0,1]")
|
||||
rng = np.random.default_rng(seed)
|
||||
if rng.random() < epsilon:
|
||||
return int(rng.integers(0, 4))
|
||||
z = self.action_logits.astype(np.float64) / temperature
|
||||
z -= z.max(); p = np.exp(z); p /= p.sum()
|
||||
return int(rng.choice(4, p=p))
|
||||
|
||||
|
||||
ACTION_NAMES = ("ACTION_0", "ACTION_1", "ACTION_2", "ACTION_3")
|
||||
|
||||
@@ -0,0 +1,465 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4G: learned interfaces around an immutable MaleCNS-LIF brain core."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse, gc, hashlib, json, math, platform, shutil, subprocess, sys, time
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow as pa
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import torch.nn as nn
|
||||
import yaml
|
||||
from scipy.spatial import cKDTree
|
||||
|
||||
from brain_runtime import BrainRuntime, InputBatch, InputEvent
|
||||
from frozen_interface import AbstractActionOutput
|
||||
|
||||
|
||||
def np_json(x: Any) -> Any:
|
||||
if isinstance(x, np.generic): return x.item()
|
||||
if isinstance(x, np.ndarray): return x.tolist()
|
||||
if isinstance(x, Path): return str(x)
|
||||
raise TypeError(type(x).__name__)
|
||||
|
||||
|
||||
def write_json(path: Path, value: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, indent=2, sort_keys=True, default=np_json)+"\n", encoding="utf-8")
|
||||
|
||||
|
||||
def sha256(path: Path, chunk: int=8<<20) -> str:
|
||||
h=hashlib.sha256()
|
||||
with path.open("rb") as f:
|
||||
while data:=f.read(chunk): h.update(data)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def verify_checksum(root: Path, control: Path) -> dict[str,Any]:
|
||||
checked=0; failures=[]
|
||||
for line in control.read_text().splitlines():
|
||||
if not line.strip(): continue
|
||||
expected,rel=line.split(None,1); path=root/rel.strip().lstrip("*"); checked+=1
|
||||
if not path.is_file() or sha256(path)!=expected: failures.append(str(path))
|
||||
return {"control":str(control),"checked":checked,"failures":failures,"passed":not failures}
|
||||
|
||||
|
||||
def frozen_audit(cfg:dict[str,Any])->tuple[list[Path],dict[str,str],dict[str,Any]]:
|
||||
p=cfg["paths"]
|
||||
controls=[Path(p["reference_root"])/"checksums.sha256",Path(p["runtime_root"])/"checksums.sha256",
|
||||
Path(p["io_root"])/"checksums.sha256",Path(p["phase4c_root"])/"manifests/checksums.sha256",
|
||||
Path(p["phase4d_root"])/"manifests/checksums.sha256",Path(p["phase4e_root"])/"manifests/checksums.sha256",
|
||||
Path(p["phase4f_root"])/"manifests/checksums.sha256",Path(p["phase4f_model_root"])/"checksums.sha256"]
|
||||
if missing:=[str(x) for x in controls if not x.is_file()]: raise FileNotFoundError(missing)
|
||||
checks=[]
|
||||
for x in controls:
|
||||
root=x.parent.parent if x.parent.name=="manifests" else x.parent; checks.append(verify_checksum(root,x))
|
||||
if not all(x["passed"] for x in checks): raise RuntimeError(checks)
|
||||
before={str(x):sha256(x) for x in controls}
|
||||
return controls,before,{"all_passed":True,"controls":checks}
|
||||
|
||||
|
||||
def stable_seed(base:int,text:str)->int:
|
||||
return base+int.from_bytes(hashlib.sha256(text.encode()).digest()[:4],"little")%1_000_000
|
||||
|
||||
|
||||
def make_samples(cfg:dict[str,Any])->pd.DataFrame:
|
||||
rows=[]
|
||||
for split,key in (("train","train_stimulus_seeds"),("held-out","evaluation_stimulus_seeds")):
|
||||
for seed in cfg["dataset"][key]:
|
||||
rng=np.random.default_rng(int(seed))
|
||||
if split=="train":
|
||||
offset=rng.uniform(-.08,.08); speed=rng.uniform(.85,1.15); size=rng.uniform(.9,1.1); curve=rng.uniform(-.08,.08); phase=rng.uniform(-.05,.05)
|
||||
else:
|
||||
sign=-1 if int(seed)%2 else 1
|
||||
offset=sign*rng.uniform(.16,.26); speed=rng.choice([rng.uniform(.62,.78),rng.uniform(1.25,1.42)])
|
||||
size=rng.choice([rng.uniform(.68,.82),rng.uniform(1.22,1.40)]); curve=sign*rng.uniform(.16,.26); phase=sign*rng.uniform(.10,.20)
|
||||
for pattern in cfg["dataset"]["patterns"]:
|
||||
rows.append({"sample_id":len(rows),"split":split,"stimulus_seed":int(seed),"pattern":pattern,
|
||||
"position_offset":offset,"speed_scale":speed,"size_scale":size,"trajectory_curve":curve,"temporal_phase":phase})
|
||||
return pd.DataFrame(rows)
|
||||
|
||||
|
||||
def frames_variant(pattern:str,x:np.ndarray,y:np.ndarray,segments:int,v:dict[str,float])->np.ndarray:
|
||||
out=[]; off=v["position_offset"]
|
||||
for i in range(segments):
|
||||
z=np.clip((i/max(segments-1,1)+v["temporal_phase"])*v["speed_scale"],0,1)
|
||||
s0=.22*v["size_scale"]
|
||||
if pattern=="full-field": f=np.ones_like(x)
|
||||
elif pattern in {"left-spot","right-spot","upper-spot","lower-spot"}:
|
||||
loc={"left-spot":(-.65+off,0),"right-spot":(.65+off,0),"upper-spot":(off,.65),"lower-spot":(off,-.65)}[pattern]
|
||||
f=np.exp(-((x-loc[0])**2+(y-loc[1])**2)/(2*s0*s0))
|
||||
elif pattern in {"left-to-right","right-to-left","up-to-down","down-to-up"}:
|
||||
p=-.8+1.6*z; p=-p if pattern in {"right-to-left","down-to-up"} else p; bend=v["trajectory_curve"]*math.sin(math.pi*z)
|
||||
cx,cy=((p,bend+off) if pattern in {"left-to-right","right-to-left"} else (bend+off,-p))
|
||||
f=np.exp(-((x-cx)**2+(y-cy)**2)/(2*(.18*v["size_scale"])**2))
|
||||
elif pattern=="looming":
|
||||
s=(.08+.48*z)*v["size_scale"]; f=np.exp(-((x-off)**2+y*y)/(2*s*s))
|
||||
elif pattern=="sparse-moving-object":
|
||||
cx=-.75+1.5*z; cy=.55-1.1*z+v["trajectory_curve"]*math.sin(math.pi*z)+off
|
||||
f=np.exp(-((x-cx)**2+(y-cy)**2)/(2*(.10*v["size_scale"])**2))
|
||||
elif pattern=="static-object": f=np.exp(-((x+.25-off)**2+(y-.20)**2)/(2*(.10*v["size_scale"])**2))
|
||||
else: raise KeyError(pattern)
|
||||
out.append(f.astype(np.float32))
|
||||
return np.asarray(out)
|
||||
|
||||
|
||||
def channels(frames:np.ndarray,coords:np.ndarray)->dict[str,np.ndarray]:
|
||||
prev=np.vstack([np.zeros((1,frames.shape[1]),np.float32),frames[:-1]]); delta=frames-prev
|
||||
nbr=cKDTree(coords).query(coords,k=min(7,len(coords)))[1][:,1:]
|
||||
dx=np.abs(coords[:,None,0]-coords[nbr,0]); dy=np.abs(coords[:,None,1]-coords[nbr,1]); hs=[];vs=[]
|
||||
for f in frames:
|
||||
d=np.abs(f[:,None]-f[nbr]); hs.append((d*dx).sum(1)/np.maximum(dx.sum(1),1e-8)); vs.append((d*dy).sum(1)/np.maximum(dy.sum(1),1e-8))
|
||||
return {"luminance":frames,"on":np.maximum(delta,0),"off":np.maximum(-delta,0),"temporal":np.abs(delta),"horizontal":np.asarray(hs),"vertical":np.asarray(vs)}
|
||||
|
||||
|
||||
def hex_grid(count:int)->tuple[np.ndarray,np.ndarray]:
|
||||
e=0
|
||||
while 1+3*e*(e+1)<count:e+=1
|
||||
u=[];v=[]
|
||||
for q in range(-e,e+1):
|
||||
for r in range(max(-e,-e-q),min(e,e-q)+1):u.append(q);v.append(r)
|
||||
u=np.asarray(u);v=np.asarray(v);x=1.5*v;y=-np.sqrt(3)*(u+v/2);x/=max(abs(x).max(),1);y/=max(abs(y).max(),1)
|
||||
return x,y
|
||||
|
||||
|
||||
def encode(cfg:dict[str,Any],route:str,sample:pd.Series,pool:pd.DataFrame,fly_xy:tuple[np.ndarray,np.ndarray])->tuple[InputBatch,np.ndarray,float,dict[str,float]]:
|
||||
t0=time.perf_counter(); seg=int(cfg["dataset"]["temporal_segments"]);k=int(cfg["dataset"]["targets_per_segment"])
|
||||
xy=pool[["retinotopic_x","retinotopic_y"]].to_numpy(float); var={x:float(sample[x]) for x in ("position_offset","speed_scale","size_scale","trajectory_curve","temporal_phase")}
|
||||
if route=="route-a": c=channels(frames_variant(sample.pattern,xy[:,0],xy[:,1],seg,var),xy)
|
||||
else:
|
||||
fx,fy=fly_xy; fxy=np.c_[fx,fy]; fc=channels(frames_variant(sample.pattern,fx,fy,seg,var),fxy); near=cKDTree(fxy).query(xy,k=1)[1]
|
||||
c={name:value[:,near] for name,value in fc.items()}
|
||||
score=c["luminance"]+c["on"]+c["off"]+.5*c["temporal"]+c["horizontal"]+c["vertical"]
|
||||
dt=100/seg;events=[];union=[]
|
||||
for i in range(seg):
|
||||
order=np.lexsort((pool.neuron_idx.to_numpy(),-score[i])); target=np.sort(pool.neuron_idx.to_numpy(np.int64)[order[:k]]);union.append(target)
|
||||
events.append(InputEvent(target,"external_synaptic_drive","poisson_rate_hz",float(cfg["dataset"]["poisson_rate_hz"]),i*dt,dt,
|
||||
"phase4g-learnable-encoder",f"{route}:{sample.pattern}:{i}",float(cfg["dataset"]["external_drive_mv"]),
|
||||
{"route":route,"pattern":sample.pattern,"adapter_only":True,"brain_state_direct_write":False}))
|
||||
descriptor={"mean_luminance":float(c["luminance"].mean()),"on":float(c["on"].mean()),"off":float(c["off"].mean()),
|
||||
"temporal":float(c["temporal"].mean()),"horizontal":float(c["horizontal"].mean()),"vertical":float(c["vertical"].mean()),**var}
|
||||
return InputBatch(tuple(events),f"phase4g:{route}:{sample.sample_id}"),np.unique(np.concatenate(union)),time.perf_counter()-t0,descriptor
|
||||
|
||||
|
||||
def groups_and_maps(cfg:dict[str,Any],rt:BrainRuntime,bins:int=128)->tuple[dict[str,np.ndarray],dict[str,np.ndarray]]:
|
||||
root=Path(cfg["paths"]["phase4c_root"])/"visual-audit";s=np.asarray(rt.superclass,object)
|
||||
groups={"visual-projection":np.flatnonzero(np.isin(s,["visual_projection","visual_projection_tbc"])).astype(np.int32),
|
||||
"central-downstream":pq.read_table(root/"hierarchy-central_brain.parquet",columns=["neuron_idx"])["neuron_idx"].to_numpy().astype(np.int32),
|
||||
"descending":rt.output_populations["descending"],"motor-related":rt.output_populations["motor_related"]}
|
||||
maps={}
|
||||
for name,members in groups.items():
|
||||
a=np.full(rt.n,-1,np.int16);a[members]=((members.astype(np.int64)*2654435761)%bins).astype(np.int16);maps[name]=a
|
||||
return groups,maps
|
||||
|
||||
|
||||
def graph_controls(cfg:dict[str,Any],rt:BrainRuntime,exp:Path)->dict[str,np.ndarray]:
|
||||
out=exp/"random-connectome";out.mkdir(parents=True,exist_ok=True);seed=int(cfg["controls"]["graph_seed"]);rng=np.random.default_rng(seed)
|
||||
real=np.asarray(rt.indices_np,dtype=np.int32); shuffled_path=out/"shuffled-indices.int32.npy"; degree_path=out/"degree-preserving-indices.int32.npy"
|
||||
if not shuffled_path.is_file(): np.save(shuffled_path,rng.integers(0,rt.n,size=len(real),dtype=np.int32))
|
||||
if not degree_path.is_file():
|
||||
degree=real.copy();rng.shuffle(degree);np.save(degree_path,degree)
|
||||
shuffled=np.load(shuffled_path,mmap_mode="r");degree=np.load(degree_path,mmap_mode="r")
|
||||
real_in=np.bincount(real,minlength=rt.n);degree_in=np.bincount(degree,minlength=rt.n)
|
||||
audit={"edge_count":len(real),"neurons":rt.n,"seed":seed,"real_indices_sha256":sha256(Path(cfg["paths"]["connectome_root"])/"gpu/csr-indices.int32.npy"),
|
||||
"shuffled":{"sha256":sha256(shuffled_path),"edge_count_equal":len(shuffled)==len(real),"source_out_degree_exact":True,"target_in_degree_preserved":False},
|
||||
"degree_preserving":{"sha256":sha256(degree_path),"edge_count_equal":len(degree)==len(real),"source_out_degree_exact":True,
|
||||
"target_in_degree_exact":bool(np.array_equal(real_in,degree_in))},"weights_changed":False,"brain_artifacts_written":False}
|
||||
write_json(out/"audit.json",audit);return {"real-malecns":real,"shuffled-edges":shuffled,"degree-preserving":degree}
|
||||
|
||||
|
||||
def hash_counts(nodes:np.ndarray,map_:np.ndarray,bins:int=128)->np.ndarray:
|
||||
x=map_[nodes];x=x[x>=0];return np.bincount(x,minlength=bins).astype(np.float32)
|
||||
|
||||
|
||||
def bypass_features(descriptors:list[dict[str,float]],route:str)->np.ndarray:
|
||||
names=sorted(descriptors[0]);x=np.asarray([[d[n] for n in names] for d in descriptors],np.float32)
|
||||
rng=np.random.default_rng(47901 if route=="route-a" else 47902);w=rng.normal(size=(x.shape[1],128)).astype(np.float32);b=rng.uniform(0,2*np.pi,128).astype(np.float32)
|
||||
return np.sqrt(2/128)*np.cos(x@w+b)
|
||||
|
||||
|
||||
def run_dataset(cfg:dict[str,Any],rt:BrainRuntime,samples:pd.DataFrame,pool:pd.DataFrame,exp:Path)->tuple[dict[str,Any],list[dict[str,Any]]]:
|
||||
controls=graph_controls(cfg,rt,exp);groups,maps=groups_and_maps(cfg,rt)
|
||||
official_coords=pq.read_table(Path(cfg["paths"]["phase4f_root"])/"retinotopy/visual-retinotopy.parquet",columns=["official_hex1","official_hex2","coordinate_reliability"]).to_pandas()
|
||||
unique_cols=int(official_coords.loc[official_coords.coordinate_reliability=="official-direct",["official_hex1","official_hex2"]].drop_duplicates().shape[0]);fly=hex_grid(unique_cols)
|
||||
windows=list(map(int,cfg["dataset"]["biological_windows_ms"]));routes=("route-a","route-b");features={};records=[];descriptors={r:[] for r in routes}
|
||||
cached=[exp/"datasets"/f"{graph}-{route}.npz" for graph in controls for route in routes]
|
||||
cached += [exp/"bypass"/f"brain-bypass-{route}.npz" for route in routes]
|
||||
record_path=exp/"datasets/generation-records.json"
|
||||
if record_path.is_file() and all(path.is_file() for path in cached):
|
||||
for graph in controls:
|
||||
features[graph]={}
|
||||
for route in routes:
|
||||
z=np.load(exp/"datasets"/f"{graph}-{route}.npz");features[graph][route]={k:z[k] for k in z.files if k.startswith("w")}
|
||||
features["brain-bypass"]={}
|
||||
for route in routes:
|
||||
z=np.load(exp/"bypass"/f"brain-bypass-{route}.npz");features["brain-bypass"][route]={k:z[k] for k in z.files}
|
||||
return features,json.loads(record_path.read_text())
|
||||
for graph,indices in controls.items():
|
||||
rt.indices_np=np.asarray(indices);rt.indices=torch.as_tensor(np.asarray(indices),dtype=torch.int32,device=rt.device);features[graph]={}
|
||||
for route in routes:
|
||||
store={f"w{w}_{g}":np.zeros((len(samples),128),np.float32) for w in windows for g in groups};enc_times=np.zeros(len(samples));brain_times=np.zeros((len(samples),len(windows)));drive=np.zeros((len(samples),len(windows)),np.int32);amps=np.zeros((len(samples),len(windows)),np.float64)
|
||||
local_desc=[]
|
||||
for row in samples.itertuples(index=False):
|
||||
batch,targets,enc_time,desc=encode(cfg,route,pd.Series(row._asdict()),pool,fly);enc_times[row.sample_id]=enc_time
|
||||
if graph=="real-malecns": descriptors[route].append(desc)
|
||||
rt.reset(int(row.stimulus_seed));last=0
|
||||
for wi,w in enumerate(windows):
|
||||
t0=time.perf_counter();rt.step(batch,float(w-last));brain_times[row.sample_id,wi]=time.perf_counter()-t0;last=w
|
||||
ev=rt.read_spikes();mask=ev["timestamp_ms"]<w+1e-9;nodes=ev["neuron_idx"][mask]
|
||||
for g in groups:store[f"w{w}_{g}"][row.sample_id]=hash_counts(nodes,maps[g])
|
||||
drive_mask=np.concatenate(rt.drive_steps)<int(round(w/rt.dt)) if rt.drive_steps else np.empty(0,bool)
|
||||
drive[row.sample_id,wi]=int(np.count_nonzero(drive_mask))
|
||||
vals=np.concatenate(rt.drive_values) if rt.drive_values else np.empty(0,np.float32);amps[row.sample_id,wi]=float(vals[drive_mask].sum()) if len(vals) else 0
|
||||
ev=rt.read_spikes();active=len(np.unique(ev["neuron_idx"]));rate=len(ev["neuron_idx"])/(rt.n*.1);finite=bool(torch.isfinite(rt.v).all() and torch.isfinite(rt.g).all())
|
||||
records.append({"graph":graph,"route":route,"sample_id":row.sample_id,"pattern":row.pattern,"split":row.split,"stimulus_seed":row.stimulus_seed,
|
||||
"encoder_wall_seconds":enc_time,"brain_wall_seconds_100ms":float(brain_times[row.sample_id].sum()),"active_neurons":active,"mean_rate_hz":rate,
|
||||
"runaway":active/rt.n>=.8 or rate>=100,"finite":finite})
|
||||
store["encoder_wall_seconds"]=enc_times;store["brain_increment_wall_seconds"]=brain_times;store["external_event_counts"]=drive;store["external_amplitudes_mv"]=amps
|
||||
path=exp/"datasets"/f"{graph}-{route}.npz";np.savez_compressed(path,**store);features[graph][route]={k:v for k,v in store.items() if k.startswith("w")}
|
||||
# Bypass uses identical external visual descriptors but no MaleCNS execution.
|
||||
features["brain-bypass"]={}
|
||||
for route in routes:
|
||||
base=bypass_features(descriptors[route],route);features["brain-bypass"][route]={f"w{w}_{g}":base.copy() for w in windows for g in groups}
|
||||
np.savez_compressed(exp/"bypass"/f"brain-bypass-{route}.npz",**features["brain-bypass"][route])
|
||||
write_json(exp/"datasets/generation-records.json",records)
|
||||
# Exact matched-energy audit by graph, route, split seed and window.
|
||||
matched={}
|
||||
for graph in controls:
|
||||
matched[graph]={}
|
||||
for route in routes:
|
||||
z=np.load(exp/"datasets"/f"{graph}-{route}.npz");entry={}
|
||||
for split,seed in samples[["split","stimulus_seed"]].drop_duplicates().itertuples(index=False):
|
||||
ids=samples.index[(samples.split==split)&(samples.stimulus_seed==seed)].to_numpy();entry[f"{split}-{seed}"]={}
|
||||
for wi,w in enumerate(windows):
|
||||
counts=z["external_event_counts"][ids,wi];amps=z["external_amplitudes_mv"][ids,wi]
|
||||
entry[f"{split}-{seed}"][str(w)]={"event_counts":sorted(set(map(int,counts))),"amplitude_mv":sorted(set(map(float,amps))),"matched":len(set(counts))==1 and len(set(amps))==1}
|
||||
matched[graph][route]=entry
|
||||
write_json(exp/"audit/matched-input.json",matched)
|
||||
return features,records
|
||||
|
||||
|
||||
def reduce_bins(x:np.ndarray,bins:int)->np.ndarray:
|
||||
if bins==128:return x.astype(np.float32)
|
||||
out=np.zeros((len(x),bins),np.float32)
|
||||
for i in range(128):out[:,i%bins]+=x[:,i]
|
||||
return out
|
||||
|
||||
|
||||
class Decoder(nn.Module):
|
||||
def __init__(self,d:int,capacity:str,hidden:int=16):
|
||||
super().__init__();self.net=nn.Linear(d,4) if capacity=="Tiny" else nn.Sequential(nn.Linear(d,hidden),nn.ReLU(),nn.Linear(hidden,4))
|
||||
def forward(self,x):return self.net(x)
|
||||
|
||||
|
||||
def fit_model(x:np.ndarray,y:np.ndarray,test_x:np.ndarray,test_y:np.ndarray,capacity:str,classes:int,seed:int,cfg:dict[str,Any])->tuple[dict[str,Any],dict[str,Any]]:
|
||||
torch.manual_seed(seed);mean=x.mean(0);std=x.std(0);std[std<1e-6]=1
|
||||
xt=torch.tensor((x-mean)/std);yt=torch.tensor(y,dtype=torch.long);xx=torch.tensor((test_x-mean)/std)
|
||||
hidden=int(cfg["training"]["capacities"][capacity].get("hidden",16));m=Decoder(x.shape[1],capacity,hidden);opt=torch.optim.Adam(m.parameters(),lr=float(cfg["training"]["learning_rate"]));t0=time.perf_counter()
|
||||
for _ in range(int(cfg["training"]["epochs"])):
|
||||
opt.zero_grad();loss=nn.functional.cross_entropy(m(xt)[:,:classes],yt);loss.backward();opt.step()
|
||||
train_wall=time.perf_counter()-t0
|
||||
with torch.no_grad():
|
||||
tr=m(xt)[:,:classes].numpy();te=m(xx)[:,:classes].numpy();pred=te.argmax(1);acc=float(np.mean(pred==test_y));train_acc=float(np.mean(tr.argmax(1)==y))
|
||||
params=sum(p.numel() for p in m.parameters());flops=(2*x.shape[1]*4 if capacity=="Tiny" else 2*x.shape[1]*hidden+2*hidden*4)
|
||||
state={k:v.detach().cpu() for k,v in m.state_dict().items()}
|
||||
bundle={"state_dict":state,"mean":mean,"std":std,"capacity":capacity,"input_dim":x.shape[1],"hidden":hidden,"classes":classes,"optimizer_state":opt.state_dict()}
|
||||
result={"accuracy":acc,"train_accuracy":train_acc,"predictions":pred.tolist(),"truth":test_y.tolist(),"confusion":np.bincount(test_y*classes+pred,minlength=classes*classes).reshape(classes,classes).tolist(),
|
||||
"train_wall_seconds":train_wall,"trainable_parameters":params,"approx_flops_per_inference":flops}
|
||||
return bundle,result
|
||||
|
||||
|
||||
def predict(bundle:dict[str,Any],x:np.ndarray)->np.ndarray:
|
||||
m=Decoder(bundle["input_dim"],bundle["capacity"],bundle["hidden"]);m.load_state_dict(bundle["state_dict"]);m.eval()
|
||||
with torch.no_grad():return m(torch.tensor((x-bundle["mean"])/bundle["std"]))[:,:bundle["classes"]].numpy()
|
||||
|
||||
|
||||
def task_data(samples:pd.DataFrame,spec:dict[str,Any])->tuple[np.ndarray,np.ndarray,np.ndarray,np.ndarray]:
|
||||
label=dict(zip(spec["patterns"],spec["labels"]));sel=samples.pattern.isin(spec["patterns"]);ids=samples.index[sel].to_numpy();y=np.asarray([label[x] for x in samples.loc[sel,"pattern"]],np.int64)
|
||||
train=samples.loc[sel,"split"].to_numpy()=="train";return ids[train],y[train],ids[~train],y[~train]
|
||||
|
||||
|
||||
def selected_matrix(a:np.ndarray,b:np.ndarray,samples:pd.DataFrame,choice:dict[str,str])->np.ndarray:
|
||||
return np.asarray([a[i] if choice.get(samples.iloc[i].pattern,"route-a")=="route-a" else b[i] for i in range(len(samples))],np.float32)
|
||||
|
||||
|
||||
def train_experiments(cfg:dict[str,Any],features:dict[str,Any],samples:pd.DataFrame,exp:Path)->tuple[dict[str,Any],dict[str,Any]]:
|
||||
capacities=cfg["training"]["capacities"];seeds=list(map(int,cfg["training"]["model_seeds"]));windows=list(map(int,cfg["dataset"]["biological_windows_ms"]));readouts=["visual-projection","central-downstream","descending","motor-related"]
|
||||
A={};selected_configs={}
|
||||
for graph in cfg["controls"]["graphs"]:
|
||||
A[graph]={}
|
||||
for route in ("route-a","route-b"):
|
||||
A[graph][route]={}
|
||||
for cap,cspec in capacities.items():
|
||||
A[graph][route][cap]={}
|
||||
for task,spec in cfg["dataset"]["tasks"].items():
|
||||
tr,ytr,te,yte=task_data(samples,spec);pilot_train=tr[samples.iloc[tr].stimulus_seed.to_numpy()!=cfg["dataset"]["train_stimulus_seeds"][-1]];pilot_y=ytr[samples.iloc[tr].stimulus_seed.to_numpy()!=cfg["dataset"]["train_stimulus_seeds"][-1]]
|
||||
val=tr[samples.iloc[tr].stimulus_seed.to_numpy()==cfg["dataset"]["train_stimulus_seeds"][-1]];val_y=ytr[samples.iloc[tr].stimulus_seed.to_numpy()==cfg["dataset"]["train_stimulus_seeds"][-1]]
|
||||
pilot=[]
|
||||
for w in windows:
|
||||
for read in readouts:
|
||||
x=reduce_bins(features[graph][route][f"w{w}_{read}"],int(cspec["feature_bins"]));_,r=fit_model(x[pilot_train],pilot_y,x[val],val_y,cap,int(spec["classes"]),seeds[0],cfg);pilot.append((r["accuracy"],w,read))
|
||||
readout_priority={"descending":3,"motor-related":2,"central-downstream":1,"visual-projection":0}
|
||||
_,w,read=max(pilot,key=lambda z:(z[0],readout_priority[z[2]],-z[1]));x=reduce_bins(features[graph][route][f"w{w}_{read}"],int(cspec["feature_bins"]));runs=[]
|
||||
for seed in seeds:
|
||||
_,r=fit_model(x[tr],ytr,x[te],yte,cap,int(spec["classes"]),seed,cfg);r["seed"]=seed;runs.append(r)
|
||||
result={"selected_window_ms":w,"selected_readout":read,"held_out_accuracy_mean":float(np.mean([z["accuracy"] for z in runs])),"held_out_accuracy_std":float(np.std([z["accuracy"] for z in runs])),"runs":runs,"chance":1/int(spec["classes"])}
|
||||
A[graph][route][cap][task]=result
|
||||
if graph=="real-malecns":selected_configs[(cap,task,route)]=(w,read)
|
||||
write_json(exp/"training/experiment-a-decoder-only.json",A)
|
||||
B={};C={};best_bundles={}
|
||||
for graph in cfg["controls"]["graphs"]:
|
||||
B[graph]={};C[graph]={}
|
||||
for cap,cspec in capacities.items():
|
||||
B[graph][cap]={};C[graph][cap]={}
|
||||
for task,spec in cfg["dataset"]["tasks"].items():
|
||||
# A common anatomy/readout configuration selected on real Route A training-validation only.
|
||||
w,read=selected_configs[(cap,task,"route-a")];a=reduce_bins(features[graph]["route-a"][f"w{w}_{read}"],int(cspec["feature_bins"]));b=reduce_bins(features[graph]["route-b"][f"w{w}_{read}"],int(cspec["feature_bins"]));tr,ytr,te,yte=task_data(samples,spec)
|
||||
bruns=[];cruns=[]
|
||||
for seed in seeds:
|
||||
fixed,base=fit_model(a[tr],ytr,a[te],yte,cap,int(spec["classes"]),seed,cfg)
|
||||
if cap=="Tiny":
|
||||
la=nn.functional.cross_entropy(torch.tensor(predict(fixed,a[tr])),torch.tensor(ytr)).item();lb=nn.functional.cross_entropy(torch.tensor(predict(fixed,b[tr])),torch.tensor(ytr)).item();choice={p:("route-a" if la<=lb else "route-b") for p in spec["patterns"]}
|
||||
else:
|
||||
choice={}
|
||||
for p in spec["patterns"]:
|
||||
ids=tr[samples.iloc[tr].pattern.to_numpy()==p];yy=ytr[samples.iloc[tr].pattern.to_numpy()==p]
|
||||
la=nn.functional.cross_entropy(torch.tensor(predict(fixed,a[ids])),torch.tensor(yy)).item();lb=nn.functional.cross_entropy(torch.tensor(predict(fixed,b[ids])),torch.tensor(yy)).item();choice[p]="route-a" if la<=lb else "route-b"
|
||||
xm=selected_matrix(a,b,samples,choice);logits=predict(fixed,xm[te]);pred=logits.argmax(1);bruns.append({"seed":seed,"accuracy":float(np.mean(pred==yte)),"route_choice":choice,"decoder_parameters":base["trainable_parameters"],"encoder_parameters":int(cspec["encoder_parameters"])})
|
||||
# Coordinate descent: update only route-selection adapter and decoder; brain feature bank is immutable.
|
||||
cchoice={p:"route-a" for p in spec["patterns"]};joint=None
|
||||
for _ in range(3):
|
||||
xm=selected_matrix(a,b,samples,cchoice);joint,_=fit_model(xm[tr],ytr,xm[te],yte,cap,int(spec["classes"]),seed,cfg)
|
||||
if cap=="Tiny":
|
||||
choices=[]
|
||||
for route,xr in (("route-a",a),("route-b",b)):
|
||||
loss=nn.functional.cross_entropy(torch.tensor(predict(joint,xr[tr])),torch.tensor(ytr)).item();choices.append((loss,route))
|
||||
cchoice={p:min(choices)[1] for p in spec["patterns"]}
|
||||
else:
|
||||
for p in spec["patterns"]:
|
||||
ids=tr[samples.iloc[tr].pattern.to_numpy()==p];yy=ytr[samples.iloc[tr].pattern.to_numpy()==p]
|
||||
la=nn.functional.cross_entropy(torch.tensor(predict(joint,a[ids])),torch.tensor(yy)).item();lb=nn.functional.cross_entropy(torch.tensor(predict(joint,b[ids])),torch.tensor(yy)).item();cchoice[p]="route-a" if la<=lb else "route-b"
|
||||
xm=selected_matrix(a,b,samples,cchoice);joint,jr=fit_model(xm[tr],ytr,xm[te],yte,cap,int(spec["classes"]),seed,cfg);jr.update({"seed":seed,"route_choice":cchoice,"encoder_parameters":int(cspec["encoder_parameters"])});cruns.append(jr)
|
||||
best_bundles[(graph,cap,task,seed)]={"bundle":joint,"choice":cchoice,"window":w,"readout":read,"accuracy":jr["accuracy"]}
|
||||
B[graph][cap][task]={"window_ms":w,"readout":read,"held_out_accuracy_mean":float(np.mean([z["accuracy"] for z in bruns])),"held_out_accuracy_std":float(np.std([z["accuracy"] for z in bruns])),"runs":bruns,"chance":1/int(spec["classes"])}
|
||||
C[graph][cap][task]={"window_ms":w,"readout":read,"held_out_accuracy_mean":float(np.mean([z["accuracy"] for z in cruns])),"held_out_accuracy_std":float(np.std([z["accuracy"] for z in cruns])),"runs":cruns,"chance":1/int(spec["classes"])}
|
||||
write_json(exp/"training/experiment-b-encoder-only.json",B);write_json(exp/"training/experiment-c-joint.json",C)
|
||||
return {"experiment_a":A,"experiment_b":B,"experiment_c":C},best_bundles
|
||||
|
||||
|
||||
def pair_metrics(a:np.ndarray,b:np.ndarray)->dict[str,float]:
|
||||
an=np.linalg.norm(a);bn=np.linalg.norm(b);cos=1-(float(a@b/(an*bn)) if an and bn else (1 if an==bn else 0));aa=a>0;bb=b>0;u=np.count_nonzero(aa|bb)
|
||||
corr=0 if np.array_equal(a,b) else (1-float(np.corrcoef(a,b)[0,1]) if np.std(a) and np.std(b) else 1)
|
||||
return {"cosine_distance":cos,"euclidean_distance":float(np.linalg.norm(a-b)),"correlation_distance":corr,"jaccard_distance":1-float(np.count_nonzero(aa&bb)/u) if u else 0}
|
||||
|
||||
|
||||
def separability_audit(cfg:dict[str,Any],features:dict[str,Any],samples:pd.DataFrame,exp:Path)->dict[str,Any]:
|
||||
out={};windows=list(map(int,cfg["dataset"]["biological_windows_ms"]));pairs=[("left-spot","right-spot"),("upper-spot","lower-spot"),("left-to-right","right-to-left"),("looming","left-to-right"),("sparse-moving-object","static-object")]
|
||||
for read in ("visual-projection","central-downstream","descending","motor-related"):
|
||||
out[read]={}
|
||||
for w in windows:
|
||||
x=features["real-malecns"]["route-a"][f"w{w}_{read}"];entry={}
|
||||
for a,b in pairs:
|
||||
xa=x[(samples.pattern==a)&(samples.split=="held-out")];xb=x[(samples.pattern==b)&(samples.split=="held-out")]
|
||||
between=[pair_metrics(i,j) for i in xa for j in xb];within=[pair_metrics(z[i],z[j]) for z in (xa,xb) for i in range(len(z)) for j in range(i+1,len(z))]
|
||||
entry[f"{a}__{b}"]={"between_mean":{k:float(np.mean([q[k] for q in between])) for k in between[0]},"within_mean":{k:float(np.mean([q[k] for q in within])) for k in within[0]}}
|
||||
out[read][str(w)]=entry
|
||||
write_json(exp/"separability/summary.json",out);return out
|
||||
|
||||
|
||||
def robustness(cfg:dict[str,Any],features:dict[str,Any],samples:pd.DataFrame,bundles:dict[Any,Any],training:dict[str,Any],exp:Path)->dict[str,Any]:
|
||||
task=cfg["robustness"]["task"];spec=cfg["dataset"]["tasks"][task];tr,ytr,te,yte=task_data(samples,spec);fraction=float(cfg["robustness"]["perturbation_fraction"]);out={}
|
||||
for graph in cfg["controls"]["graphs"]:
|
||||
cap="Small";seed=int(cfg["training"]["model_seeds"][0]);item=bundles[(graph,cap,task,seed)];w=item["window"];read=item["readout"]
|
||||
a=reduce_bins(features[graph]["route-a"][f"w{w}_{read}"],128);b=reduce_bins(features[graph]["route-b"][f"w{w}_{read}"],128);x=selected_matrix(a,b,samples,item["choice"]);base=x[te];rng=np.random.default_rng(47400);conditions={}
|
||||
for condition in cfg["robustness"]["conditions"]:
|
||||
z=base.copy()
|
||||
if condition=="input-noise":z+=rng.normal(0,fraction*max(float(z.std()),1e-3),z.shape)
|
||||
elif condition=="event-dropout":z=rng.binomial(np.maximum(np.rint(z),0).astype(np.int32),1-fraction).astype(np.float32)
|
||||
elif condition=="neuron-dropout":z[:,rng.choice(z.shape[1],int(z.shape[1]*fraction),replace=False)]=0
|
||||
elif condition=="adapter-noise":z*=rng.normal(1,fraction,z.shape)
|
||||
elif condition=="partial-occlusion":z[:,:int(z.shape[1]*fraction)]=0
|
||||
elif condition=="temporal-jitter":
|
||||
available=list(map(int,cfg["dataset"]["biological_windows_ms"]));alt=max([q for q in available if q<w],default=w)
|
||||
aa=reduce_bins(features[graph]["route-a"][f"w{alt}_{read}"],128);bb=reduce_bins(features[graph]["route-b"][f"w{alt}_{read}"],128);z=selected_matrix(aa,bb,samples,item["choice"])[te]
|
||||
pred=predict(item["bundle"],z).argmax(1);conditions[condition]=float(np.mean(pred==yte))
|
||||
conditions["clean"]=float(np.mean(predict(item["bundle"],base).argmax(1)==yte));out[graph]={"window_ms":w,"readout":read,"accuracy":conditions,
|
||||
"method_note":"feature-domain stress of frozen neural readout; temporal jitter uses adjacent earlier biological window; no brain retraining"}
|
||||
write_json(exp/"robustness/summary.json",out);return out
|
||||
|
||||
|
||||
def regression(cfg:dict[str,Any],rt:BrainRuntime,exp:Path)->dict[str,Any]:
|
||||
sys.path.insert(0,cfg["paths"]["io_root"]);import phase4d as p4d
|
||||
trials=p4d.regression(exp,cfg,rt);api=p4d.api_tests(exp,rt);stream=p4d.stream_contract(exp,rt)
|
||||
result={"phase4c_trials":all(x["passed"] for x in trials.values()),"phase4d_brainio":api["all_passed"],"activity_stream":stream["replay_exact"],
|
||||
"phase4e_checksum":True,"phase4f_checksum":True,"all_passed":all(x["passed"] for x in trials.values()) and api["all_passed"] and stream["replay_exact"]}
|
||||
write_json(exp/"regression/summary.json",result);return result
|
||||
|
||||
|
||||
def tree_manifest(root:Path,output:Path,excludes:set[Path])->dict[str,Any]:
|
||||
files=[]
|
||||
for p in sorted(root.rglob("*")):
|
||||
if p.is_file() and p not in excludes:files.append({"path":str(p.relative_to(root)),"bytes":p.stat().st_size,"sha256":sha256(p)})
|
||||
value={"root":str(root),"file_count":len(files),"total_bytes":sum(x["bytes"] for x in files),"files":files};write_json(output,value);return value
|
||||
|
||||
|
||||
def main()->None:
|
||||
ap=argparse.ArgumentParser();ap.add_argument("--config",type=Path,required=True);args=ap.parse_args();cfg=yaml.safe_load(args.config.read_text());exp=Path(cfg["paths"]["experiment_root"]);learned=Path(cfg["paths"]["learned_root"]);interface=Path(cfg["paths"]["interface_root"])
|
||||
for sub in ("datasets","encoder","decoder","training","controls","random-connectome","bypass","separability","robustness","generalization","latency","regression","audit","manifests","compiler-reports/extension-build"):(exp/sub).mkdir(parents=True,exist_ok=True)
|
||||
for sub in ("encoder","decoder","checkpoints","optimizer-state","training-metadata"):(learned/sub).mkdir(parents=True,exist_ok=True)
|
||||
torch.set_num_threads(1);torch.set_num_interop_threads(1)
|
||||
controls,before,frozen=frozen_audit(cfg);write_json(exp/"audit/frozen-before.json",frozen);shutil.copy2(args.config,exp/"training/phase4g.yaml")
|
||||
samples=make_samples(cfg);pq.write_table(pa.Table.from_pandas(samples,preserve_index=False),exp/"datasets/sample-index.parquet",compression="zstd");write_json(exp/"generalization/held-out-policy.json",{"train_ranges":"small position/speed/size/trajectory/time variations","held_out_ranges":"disjoint larger position, unseen speed/size, curved trajectory and temporal phase","train_test_overlap":False})
|
||||
sys.path.insert(0,cfg["paths"]["model_root"]);rt=BrainRuntime(args.config,exp/"compiler-reports/extension-build");pool=pq.read_table(Path(cfg["paths"]["phase4f_root"])/"retinotopy/visual-projection-entry-pool.parquet").to_pandas()
|
||||
features,records=run_dataset(cfg,rt,samples,pool,exp);del rt;gc.collect();torch.cuda.empty_cache()
|
||||
training,bundles=train_experiments(cfg,features,samples,exp);sep=separability_audit(cfg,features,samples,exp);rob=robustness(cfg,features,samples,bundles,training,exp)
|
||||
# Select the formal best real joint result by cross-seed mean, never by a single best run.
|
||||
candidates=[]
|
||||
for cap,tasks in training["experiment_c"]["real-malecns"].items():
|
||||
for task,v in tasks.items():candidates.append((v["held_out_accuracy_mean"],cap,task,v))
|
||||
_,best_cap,best_task,best=max(candidates,key=lambda z:z[0]);mean_accuracy=best["held_out_accuracy_mean"]
|
||||
best_seed=min(cfg["training"]["model_seeds"],key=lambda s:abs(bundles[("real-malecns",best_cap,best_task,int(s))]["accuracy"]-mean_accuracy));best_bundle=bundles[("real-malecns",best_cap,best_task,int(best_seed))]
|
||||
ckpt={"decoder":best_bundle["bundle"],"encoder_route_choice":best_bundle["choice"],"window_ms":best_bundle["window"],"readout":best_bundle["readout"],"task":best_task,"capacity":best_cap,"brain_parameters_in_optimizer":False}
|
||||
torch.save(ckpt,learned/"checkpoints/best-frozen-interface.pt");torch.save(best_bundle["bundle"]["optimizer_state"],learned/"optimizer-state/best-decoder-optimizer.pt")
|
||||
write_json(learned/"training-metadata/best.json",{k:v for k,v in ckpt.items() if k not in {"decoder"}})
|
||||
# Reload checkpoint and exercise abstract action API.
|
||||
loaded=torch.load(learned/"checkpoints/best-frozen-interface.pt",weights_only=False);spec=cfg["dataset"]["tasks"][best_task];tr,ytr,te,yte=task_data(samples,spec);w=loaded["window_ms"];read=loaded["readout"];a=reduce_bins(features["real-malecns"]["route-a"][f"w{w}_{read}"],int(cfg["training"]["capacities"][best_cap]["feature_bins"]));b=reduce_bins(features["real-malecns"]["route-b"][f"w{w}_{read}"],int(cfg["training"]["capacities"][best_cap]["feature_bins"]));x=selected_matrix(a,b,samples,loaded["encoder_route_choice"]);logit=predict(loaded["decoder"],x[te[:1]])[0];padded=np.full(4,-1e9,np.float32);padded[:len(logit)]=logit;api=AbstractActionOutput(padded);api_audit={"checkpoint_reload_prediction":api.deterministic_argmax(),"stochastic_sample":api.sample(47500,temperature=.8),"epsilon_sample":api.sample(47501,epsilon=.1),"abstract_only":True,"game_semantics_assigned":False};write_json(exp/"audit/abstract-action-api.json",api_audit)
|
||||
# Latency audit includes real measured encoder/brain times and decoder microbenchmark.
|
||||
real_records=[x for x in records if x["graph"]=="real-malecns"];decoder_t0=time.perf_counter()
|
||||
for _ in range(1000):predict(loaded["decoder"],x[te[:1]])
|
||||
decoder_s=(time.perf_counter()-decoder_t0)/1000;latency={}
|
||||
for w in cfg["dataset"]["biological_windows_ms"]:
|
||||
# Brain increments were measured separately; cumulative fractions are directly available from saved NPZ.
|
||||
z=np.load(exp/"datasets/real-malecns-route-a.npz");wi=list(cfg["dataset"]["biological_windows_ms"]).index(w);brain=float(z["brain_increment_wall_seconds"][:,:wi+1].sum(1).mean());enc=float(z["encoder_wall_seconds"].mean());total=enc+brain+decoder_s
|
||||
latency[str(w)]={"encoder_seconds":enc,"brain_seconds":brain,"decoder_seconds":decoder_s,"total_seconds":total,"samples_per_hour":3600/total}
|
||||
write_json(exp/"latency/summary.json",latency)
|
||||
# Fresh frozen runtime regression.
|
||||
reg_rt=BrainRuntime(args.config,exp/"compiler-reports/extension-build");reg=regression(cfg,reg_rt,exp);del reg_rt;gc.collect();torch.cuda.empty_cache()
|
||||
after={str(x):sha256(x) for x in controls};unchanged=before==after
|
||||
# Formal gate.
|
||||
chance=1/int(cfg["dataset"]["tasks"][best_task]["classes"]);runs=best["runs"];margin=float(cfg["success_gate"]["stable_accuracy_margin_over_chance"]);success_seeds=sum(r["accuracy"]>=chance+margin for r in runs)
|
||||
downstream=best["readout"] in {"descending","motor-related"};matched=json.loads((exp/"audit/matched-input.json").read_text());matched_pass=all(vv["matched"] for g in matched.values() for r in g.values() for s in r.values() for vv in s.values())
|
||||
stable=success_seeds>=int(cfg["success_gate"]["successful_model_seed_majority"]);nonrun=all(not x["runaway"] and x["finite"] for x in records)
|
||||
control_means={g:training["experiment_c"][g][best_cap][best_task]["held_out_accuracy_mean"] for g in cfg["controls"]["graphs"]};control_difference=any(abs(control_means["real-malecns"]-v)>=.05 for g,v in control_means.items() if g!="real-malecns")
|
||||
bypass_too_easy=any(training["experiment_c"]["brain-bypass"][cap][task]["held_out_accuracy_mean"]>=float(cfg["success_gate"]["bypass_too_easy_accuracy"]) for cap in cfg["training"]["capacities"] for task in cfg["dataset"]["tasks"])
|
||||
passed=all([unchanged,reg["all_passed"],best["held_out_accuracy_mean"]>chance,stable,downstream,matched_pass,nonrun,control_difference])
|
||||
status="PASS-CANDIDATE" if passed else "FAIL-NO-FROZEN-INTERFACE-CANDIDATE"
|
||||
summary={"status":status,"candidate_generated":passed,"best":{"capacity":best_cap,"task":best_task,"window_ms":best["window_ms"],"readout":best["readout"],"held_out_accuracy_mean":best["held_out_accuracy_mean"],"held_out_accuracy_std":best["held_out_accuracy_std"],"chance":chance,"successful_seeds":success_seeds},
|
||||
"control_means":control_means,"control_difference":control_difference,"matched_input_pass":matched_pass,"no_runaway_nan_inf":nonrun,"frozen_brain_unchanged":unchanged,"regression":reg,"checkpoint_reload_pass":True,"bypass_too_easy":bypass_too_easy,
|
||||
"task_too_easy_for_brain_contribution_analysis":bypass_too_easy,"trainable_encoder_parameters":cfg["training"]["capacities"][best_cap]["encoder_parameters"],"trainable_decoder_parameters":best["runs"][0]["trainable_parameters"],"surrogate_note":"discrete route-selection coordinate descent is an engineering optimization mechanism, not a biological claim","prohibitions_respected":cfg["prohibitions"]}
|
||||
write_json(exp/"audit/summary.json",summary);interface.mkdir(parents=True,exist_ok=True)
|
||||
for src in (Path(__file__).resolve(),args.config.resolve(),Path(__file__).with_name("frozen_interface.py"),Path(__file__).with_name("brain_runtime.py")):
|
||||
dst=interface/src.name
|
||||
if src.resolve()!=dst.resolve():shutil.copy2(src,dst)
|
||||
write_json(interface/"VERSION.json",{"name":cfg["names"]["interface"],"status":status,"candidate_name":cfg["names"]["candidate"] if passed else None,"male_cns_weights_fixed":True,"learned_state_root":str(learned)})
|
||||
im=tree_manifest(interface,interface/"MANIFEST.json",{interface/"MANIFEST.json",interface/"checksums.sha256"});targets=sorted(p for p in interface.rglob("*") if p.is_file() and p.name!="checksums.sha256");(interface/"checksums.sha256").write_text("".join(f"{sha256(p)} {p.relative_to(interface)}\n" for p in targets))
|
||||
lm=tree_manifest(learned,learned/"MANIFEST.json",{learned/"MANIFEST.json",learned/"checksums.sha256"});targets=sorted(p for p in learned.rglob("*") if p.is_file() and p.name!="checksums.sha256");(learned/"checksums.sha256").write_text("".join(f"{sha256(p)} {p.relative_to(learned)}\n" for p in targets))
|
||||
em=tree_manifest(exp,exp/"manifests/manifest.json",{exp/"manifests/manifest.json",exp/"manifests/checksums.sha256"});targets=sorted(p for p in exp.rglob("*") if p.is_file() and p!=exp/"manifests/checksums.sha256");(exp/"manifests/checksums.sha256").write_text("".join(f"{sha256(p)} {p.relative_to(exp)}\n" for p in targets))
|
||||
print(json.dumps({"status":status,"best":summary["best"],"controls":control_means,"matched":matched_pass,"regression":reg["all_passed"],"frozen":unchanged},indent=2))
|
||||
|
||||
|
||||
if __name__=="__main__":main()
|
||||
@@ -0,0 +1,114 @@
|
||||
phase: 4G
|
||||
names:
|
||||
reference: MaleCNS-LIF-reference-v1
|
||||
runtime: MaleCNS-LIF-runtime-v1
|
||||
io_api: MaleCNS-BrainIO-v1
|
||||
activity_stream: MaleCNS-ActivityStream-v1
|
||||
interface: MaleCNS-FrozenInterface-v1
|
||||
candidate: MaleCNS-FrozenInterface-v1-candidate
|
||||
|
||||
paths:
|
||||
connectome_root: /data/digital-drosophila/datasets/male-cns-v1.0/processed/neuron-only
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif
|
||||
reference_root: /data/digital-drosophila/models/male-cns-lif/reference-v1
|
||||
runtime_root: /data/digital-drosophila/models/male-cns-lif/runtime-v1
|
||||
io_root: /data/digital-drosophila/models/male-cns-lif/io-v1
|
||||
phase4c_root: /data/digital-drosophila/experiments/phase4c-runtime
|
||||
phase4d_root: /data/digital-drosophila/experiments/phase4d-io
|
||||
phase4e_root: /data/digital-drosophila/experiments/phase4e-visual-calibration
|
||||
phase4f_root: /data/digital-drosophila/experiments/phase4f-retinotopic-vision
|
||||
phase4f_model_root: /data/digital-drosophila/models/male-cns-lif/visual-encoder-v1
|
||||
interface_root: /data/digital-drosophila/models/male-cns-lif/frozen-interface-v1
|
||||
learned_root: /data/digital-drosophila/models/learned/phase4g
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4g-frozen-interface
|
||||
|
||||
device:
|
||||
required_name_substring: RTX 4090
|
||||
launcher_visible_gpu_uuid: GPU-79f3a85d-adce-7833-7003-6488140ca596
|
||||
|
||||
model:
|
||||
dt_ms: 0.1
|
||||
membrane_tau_ms: 20.0
|
||||
synaptic_tau_ms: 5.0
|
||||
refractory_ms: 2.2
|
||||
resting_potential_mv: -52.0
|
||||
reset_potential_mv: -52.0
|
||||
threshold_mv: -45.0
|
||||
synaptic_delay_ms: 1.8
|
||||
w_syn_mv_per_anatomical_synapse: 0.275
|
||||
|
||||
sign_policy:
|
||||
neurotransmitter_sign: {acetylcholine: 1, gaba: -1, glutamate: -1, dopamine: 1, octopamine: 1, serotonin: 1, histamine: -1}
|
||||
unknown_sign: 0
|
||||
unclear_sign: 0
|
||||
|
||||
dataset:
|
||||
patterns: [left-spot, right-spot, upper-spot, lower-spot, left-to-right, right-to-left, up-to-down, down-to-up, looming, sparse-moving-object, static-object, full-field]
|
||||
train_stimulus_seeds: [47001, 47002, 47003, 47004, 47005]
|
||||
evaluation_stimulus_seeds: [47101, 47102, 47103]
|
||||
biological_windows_ms: [10, 20, 30, 50, 100]
|
||||
temporal_segments: 5
|
||||
targets_per_segment: 460
|
||||
poisson_rate_hz: 50.0
|
||||
external_drive_mv: 8.8
|
||||
tasks:
|
||||
left-right: {patterns: [left-spot, right-spot], labels: [0, 1], classes: 2}
|
||||
up-down: {patterns: [upper-spot, lower-spot], labels: [0, 1], classes: 2}
|
||||
moving-static: {patterns: [sparse-moving-object, static-object], labels: [0, 1], classes: 2}
|
||||
looming-translation: {patterns: [looming, left-to-right], labels: [0, 1], classes: 2}
|
||||
spatial-4way: {patterns: [left-spot, right-spot, upper-spot, lower-spot], labels: [0, 1, 2, 3], classes: 4}
|
||||
motion-4way: {patterns: [left-to-right, right-to-left, up-to-down, down-to-up], labels: [0, 1, 2, 3], classes: 4}
|
||||
|
||||
training:
|
||||
model_seeds: [47201, 47202, 47203, 47204, 47205]
|
||||
evaluation_seeds: [47101, 47102, 47103]
|
||||
epochs: 120
|
||||
learning_rate: 0.02
|
||||
capacities:
|
||||
Tiny: {feature_bins: 32, decoder: linear, encoder_parameters: 2}
|
||||
Small: {feature_bins: 128, decoder: mlp, hidden: 16, encoder_parameters: 24}
|
||||
abstract_actions: 4
|
||||
|
||||
controls:
|
||||
graph_seed: 47300
|
||||
graphs: [real-malecns, shuffled-edges, degree-preserving, brain-bypass]
|
||||
shuffled_definition: independent uniform target per existing CSR edge slot; source degrees and edge count preserved
|
||||
degree_preserving_definition: global permutation of original target array; exact source out-degree and target in-degree preserved
|
||||
|
||||
robustness:
|
||||
task: spatial-4way
|
||||
conditions: [input-noise, event-dropout, neuron-dropout, adapter-noise, partial-occlusion, temporal-jitter]
|
||||
perturbation_fraction: 0.15
|
||||
|
||||
readout:
|
||||
bins_ms: [1, 5, 10, 20, 50]
|
||||
|
||||
regression:
|
||||
float_absolute_tolerance: 0.00390625
|
||||
reference_trials:
|
||||
- {name: auditory-10hz, population: auditory, rate_hz: 10, seed: 42001, duration_ms: 1000.0}
|
||||
- {name: auditory-100hz, population: auditory, rate_hz: 100, seed: 42001, duration_ms: 1000.0}
|
||||
- {name: olfactory-50hz, population: olfactory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
- {name: mechanosensory-50hz, population: mechanosensory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
- {name: gustatory-50hz, population: gustatory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
|
||||
success_gate:
|
||||
stable_accuracy_margin_over_chance: 0.10
|
||||
successful_model_seed_majority: 3
|
||||
require_descending_or_motor_information: true
|
||||
require_control_difference: true
|
||||
bypass_too_easy_accuracy: 0.95
|
||||
|
||||
prohibitions:
|
||||
brain_backpropagation: false
|
||||
connectome_changes: false
|
||||
transmitter_sign_changes: false
|
||||
brain_gain_changes: false
|
||||
internal_weight_changes: false
|
||||
lif_parameter_changes: false
|
||||
game_rom: false
|
||||
stable_retro: false
|
||||
reinforcement_learning: false
|
||||
dopamine_stdp: false
|
||||
web_server: false
|
||||
three_d_frontend: false
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"artifact": "io-v1",
|
||||
"files": [
|
||||
{
|
||||
"bytes": 1564,
|
||||
"path": "VERSION.json",
|
||||
"sha256": "2ad17a93bf22339db56ae3869af9a2a7c26ca7359d8923c693d71dde63927812"
|
||||
},
|
||||
{
|
||||
"bytes": 23822,
|
||||
"path": "brain_runtime.py",
|
||||
"sha256": "1c4e3e6f06708bfec30b24e92297f320b2b8b87ebbbe6da501f5f2db4f74762a"
|
||||
},
|
||||
{
|
||||
"bytes": 34442,
|
||||
"path": "phase4d.py",
|
||||
"sha256": "9ed8e4246252bdfc590c88e2993b7e105f278c6c4943e1c688f387c1cf54aeb7"
|
||||
},
|
||||
{
|
||||
"bytes": 3175,
|
||||
"path": "phase4d.yaml",
|
||||
"sha256": "e94301362e630e59fa248f883ae09d369dce720bce970d0d2f47ef6a6ed36a2a"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,81 @@
|
||||
{
|
||||
"activity_stream_version": "1.0.0",
|
||||
"api_methods": [
|
||||
"reset",
|
||||
"step",
|
||||
"read_spikes",
|
||||
"read_population",
|
||||
"read_output",
|
||||
"get_state_summary",
|
||||
"checkpoint_state",
|
||||
"restore_state"
|
||||
],
|
||||
"api_tests": {
|
||||
"all_passed": true,
|
||||
"checkpoint_restore_spikes_exact": true,
|
||||
"checkpoint_restore_voltage_exact": true,
|
||||
"continuous_step_does_not_reset_clock": true,
|
||||
"fixed_temporal_bins_tested": {
|
||||
"1": [
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
0
|
||||
],
|
||||
"10": [
|
||||
0,
|
||||
1,
|
||||
0
|
||||
],
|
||||
"20": [
|
||||
1,
|
||||
0
|
||||
],
|
||||
"5": [
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
1,
|
||||
0
|
||||
]
|
||||
},
|
||||
"per_class_aggregation_labels": 1,
|
||||
"per_neuron_readout_shape": 1316,
|
||||
"per_type_aggregation_labels": 481,
|
||||
"readout_modes_passed": true,
|
||||
"reset_same_seed_continuous_chunk_spikes_exact": true,
|
||||
"single_vs_chunk_voltage_max_abs_error": 0.0,
|
||||
"timestamps_present": true
|
||||
},
|
||||
"api_version": "1.0.0",
|
||||
"continuous_gpu_state": true,
|
||||
"game_agnostic": true,
|
||||
"input_batch_version": "1.0.0",
|
||||
"name": "MaleCNS-BrainIO-v1",
|
||||
"prohibitions": {
|
||||
"connectome_or_sign_changes": false,
|
||||
"controller_buttons": false,
|
||||
"rom_access": false,
|
||||
"stable_retro": false,
|
||||
"training": false,
|
||||
"web_server": false
|
||||
},
|
||||
"runtime_dependency": "MaleCNS-LIF-runtime-v1"
|
||||
}
|
||||
@@ -0,0 +1,415 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stable game-agnostic Brain I/O API for MaleCNS-LIF-runtime-v1."""
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import json
|
||||
import math
|
||||
import sys
|
||||
from dataclasses import dataclass, field
|
||||
from pathlib import Path
|
||||
from typing import Any, Iterable
|
||||
|
||||
import numpy as np
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
|
||||
API_VERSION = "1.0.0"
|
||||
INPUT_BATCH_VERSION = "1.0.0"
|
||||
ACTIVITY_STREAM_VERSION = "1.0.0"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InputEvent:
|
||||
target_neuron_idx: np.ndarray
|
||||
stimulation_mode: str
|
||||
value_kind: str
|
||||
value: float | np.ndarray
|
||||
start_ms: float
|
||||
duration_ms: float
|
||||
source_id: str
|
||||
channel_id: str
|
||||
amplitude_mv: float = 0.0
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def validate(self, neuron_count: int, dt_ms: float) -> None:
|
||||
target = np.asarray(self.target_neuron_idx)
|
||||
if target.ndim != 1 or target.dtype.kind not in "iu":
|
||||
raise ValueError("target_neuron_idx must be a one-dimensional integer array")
|
||||
if len(target) and (int(target.min()) < 0 or int(target.max()) >= neuron_count):
|
||||
raise ValueError("target neuron_idx outside canonical range")
|
||||
if len(np.unique(target)) != len(target):
|
||||
raise ValueError("target neuron_idx values must be unique")
|
||||
if self.stimulation_mode not in {"spike_injection", "external_synaptic_drive"}:
|
||||
raise ValueError(f"unsupported stimulation_mode {self.stimulation_mode!r}")
|
||||
if self.value_kind not in {"binary", "scalar_intensity", "poisson_rate_hz", "signed_modulation"}:
|
||||
raise ValueError(f"unsupported value_kind {self.value_kind!r}")
|
||||
if self.start_ms < 0 or self.duration_ms <= 0:
|
||||
raise ValueError("timestamps/duration must be non-negative/positive")
|
||||
for x in (self.start_ms, self.duration_ms):
|
||||
if not math.isclose(x / dt_ms, round(x / dt_ms), abs_tol=1e-8):
|
||||
raise ValueError(f"{x} ms is not aligned to dt={dt_ms} ms")
|
||||
value = np.asarray(self.value)
|
||||
if value.ndim > 1 or (value.ndim == 1 and len(value) not in (1, len(target))):
|
||||
raise ValueError("value must be scalar or match target count")
|
||||
if self.value_kind == "poisson_rate_hz" and np.any(value < 0):
|
||||
raise ValueError("Poisson rate cannot be negative")
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InputBatch:
|
||||
events: tuple[InputEvent, ...]
|
||||
batch_id: str
|
||||
version: str = INPUT_BATCH_VERSION
|
||||
metadata: dict[str, Any] = field(default_factory=dict)
|
||||
|
||||
def validate(self, neuron_count: int, dt_ms: float) -> None:
|
||||
if self.version != INPUT_BATCH_VERSION:
|
||||
raise ValueError(f"InputBatch version must be {INPUT_BATCH_VERSION}")
|
||||
for event in self.events:
|
||||
event.validate(neuron_count, dt_ms)
|
||||
|
||||
|
||||
class VisualAdapterV1:
|
||||
"""Maps abstract channels to audited visual hierarchy populations."""
|
||||
|
||||
def __init__(self, config: dict[str, Any], phase4c_root: Path):
|
||||
self.config = config
|
||||
self.phase4c_root = phase4c_root
|
||||
self.modes: dict[str, np.ndarray] = {}
|
||||
for mode, spec in config["visual_adapter"]["modes"].items():
|
||||
population = spec["hierarchy_population"]
|
||||
path = phase4c_root / "visual-audit" / f"hierarchy-{population}.parquet"
|
||||
if not path.is_file():
|
||||
raise FileNotFoundError(path)
|
||||
self.modes[mode] = pq.read_table(path, columns=["neuron_idx"])["neuron_idx"].to_numpy().astype(np.int64)
|
||||
|
||||
def population(self, mode: str) -> np.ndarray:
|
||||
if mode not in self.modes:
|
||||
raise KeyError(mode)
|
||||
return self.modes[mode].copy()
|
||||
|
||||
def encode(
|
||||
self, mode: str, channel_values: np.ndarray, start_ms: float, duration_ms: float,
|
||||
source_id: str, channel_id: str, value_kind: str = "poisson_rate_hz",
|
||||
) -> InputBatch:
|
||||
if mode not in self.modes:
|
||||
raise KeyError(mode)
|
||||
spec = self.config["visual_adapter"]["modes"][mode]
|
||||
targets = self.modes[mode]
|
||||
values = np.asarray(channel_values, dtype=np.float32)
|
||||
if values.shape != (len(targets),):
|
||||
raise ValueError(f"visual channel must have shape {(len(targets),)}, got {values.shape}")
|
||||
selected = np.flatnonzero(values != 0).astype(np.int64)
|
||||
target = targets[selected]
|
||||
selected_values = values[selected]
|
||||
event = InputEvent(
|
||||
target_neuron_idx=target,
|
||||
stimulation_mode=spec["input_mode"],
|
||||
value_kind=value_kind,
|
||||
value=selected_values,
|
||||
amplitude_mv=float(self.config["visual_adapter"]["external_drive_mv"]),
|
||||
start_ms=start_ms,
|
||||
duration_ms=duration_ms,
|
||||
source_id=source_id,
|
||||
channel_id=channel_id,
|
||||
metadata={
|
||||
"adapter": self.config["names"]["visual_adapter"],
|
||||
"adapter_mode": mode,
|
||||
"biological_fidelity": spec["biological_fidelity"],
|
||||
"connectome_weights_modified": False,
|
||||
},
|
||||
)
|
||||
return InputBatch(events=(event,), batch_id=f"{source_id}:{channel_id}:{start_ms}", metadata={"adapter_mode": mode})
|
||||
|
||||
|
||||
class BrainRuntime:
|
||||
"""Persistent exact-state BrainRuntime API. It has no game-specific dependency."""
|
||||
|
||||
def __init__(self, config_path: Path, extension_build_dir: Path):
|
||||
self.config_path = config_path.resolve()
|
||||
self.cfg = yaml.safe_load(self.config_path.read_text(encoding="utf-8"))
|
||||
self.dt = float(self.cfg["model"]["dt_ms"])
|
||||
self.n = 166700
|
||||
self.model_root = Path(self.cfg["paths"]["model_root"])
|
||||
self.runtime_root = Path(self.cfg["paths"]["runtime_root"])
|
||||
self.input_root = Path(self.cfg["paths"]["connectome_root"])
|
||||
runtime_module_dir = str(self.runtime_root)
|
||||
if runtime_module_dir not in sys.path:
|
||||
sys.path.insert(0, runtime_module_dir)
|
||||
import phase4c as runtime_module
|
||||
self.runtime_module = runtime_module
|
||||
p4cfg = yaml.safe_load((self.runtime_root / "phase4c.yaml").read_text(encoding="utf-8"))
|
||||
self.paths = runtime_module.exact_paths(p4cfg)
|
||||
self.indptr_np, self.indices_np, self.raw_np = runtime_module.p4b.validate_graph(p4cfg, self.paths)
|
||||
self.table = runtime_module.full_metadata(self.paths["metadata"])
|
||||
self.population_masks = runtime_module.populations(self.table)
|
||||
if not torch.cuda.is_available() or torch.cuda.device_count() != 1:
|
||||
raise RuntimeError("BrainRuntime requires exactly one visible CUDA GPU")
|
||||
self.device = torch.device("cuda:0")
|
||||
props = torch.cuda.get_device_properties(self.device)
|
||||
if self.cfg["device"]["required_name_substring"] not in props.name:
|
||||
raise RuntimeError(f"required RTX 4090, got {props.name}")
|
||||
extension_build_dir.mkdir(parents=True, exist_ok=True)
|
||||
build_experiment_root = extension_build_dir.parent.parent
|
||||
(build_experiment_root / "compiler-reports/extension-build").mkdir(parents=True, exist_ok=True)
|
||||
self.extension, report = runtime_module.build_extension(self.runtime_root, build_experiment_root)
|
||||
if self.extension is None:
|
||||
raise RuntimeError(f"runtime-v1 extension unavailable: {report}")
|
||||
base = runtime_module.Runtime(p4cfg, self.table, self.indptr_np, self.indices_np, self.raw_np, self.device, self.extension)
|
||||
self.indptr, self.indices, self.weights = base.indptr, base.indices, base.weights
|
||||
self.signs, self.superclass = base.signs, base.superclass
|
||||
self.neuron_type = np.asarray(self.table["annotation_type"].to_pylist(), dtype=object)
|
||||
self.neuron_class = np.asarray(self.table["annotation_class"].to_pylist(), dtype=object)
|
||||
self.body_id = self.table["body_id"].to_numpy().astype(np.uint64)
|
||||
m = self.cfg["model"]
|
||||
self.delay_steps = int(round(float(m["synaptic_delay_ms"]) / self.dt))
|
||||
self.queue_len = self.delay_steps + 1
|
||||
self.refractory_steps = int(round(float(m["refractory_ms"]) / self.dt))
|
||||
self.alpha = self.dt / float(m["membrane_tau_ms"])
|
||||
self.g_decay = 1.0 - self.dt / float(m["synaptic_tau_ms"])
|
||||
self.rest = float(m["resting_potential_mv"])
|
||||
self.reset_value = float(m["reset_potential_mv"])
|
||||
self.threshold = float(m["threshold_mv"])
|
||||
self.output_populations = self._output_populations()
|
||||
self.reset(0)
|
||||
|
||||
def _output_populations(self) -> dict[str, np.ndarray]:
|
||||
s = self.superclass
|
||||
return {
|
||||
"descending": np.flatnonzero(np.isin(s, ["descending_neuron", "descending_neuron_tbc"])).astype(np.int32),
|
||||
"motor_related": np.flatnonzero(np.isin(s, ["vnc_motor", "cb_motor"])).astype(np.int32),
|
||||
"ascending": np.flatnonzero(np.isin(s, ["ascending_neuron", "sensory_ascending", "sensory_ascending_tbc", "efferent_ascending"])).astype(np.int32),
|
||||
}
|
||||
|
||||
def reset(self, seed: int, initial_state: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
self.seed = int(seed)
|
||||
self.generator = torch.Generator(device=self.device); self.generator.manual_seed(self.seed)
|
||||
self.v = torch.full((self.n,), self.rest, dtype=torch.float32, device=self.device)
|
||||
self.g = torch.zeros(self.n, dtype=torch.float32, device=self.device)
|
||||
self.refractory = torch.zeros(self.n, dtype=torch.int32, device=self.device)
|
||||
self.delay_queue = torch.zeros((self.queue_len, self.n), dtype=torch.float32, device=self.device)
|
||||
self.spike_counts = torch.zeros(self.n, dtype=torch.int32, device=self.device)
|
||||
self.simulation_step = 0; self.sequence_id = 0
|
||||
self.spike_steps: list[np.ndarray] = []; self.spike_nodes: list[np.ndarray] = []
|
||||
self.spike_origins: list[np.ndarray] = []
|
||||
self.drive_steps: list[np.ndarray] = []; self.drive_nodes: list[np.ndarray] = []
|
||||
self.drive_values: list[np.ndarray] = []; self.input_audit: list[dict[str, Any]] = []
|
||||
if initial_state is not None:
|
||||
self.restore_state(initial_state)
|
||||
return self.get_state_summary()
|
||||
|
||||
@property
|
||||
def simulation_time_ms(self) -> float:
|
||||
return self.simulation_step * self.dt
|
||||
|
||||
def _event_values(self, event: InputEvent, targets: torch.Tensor) -> torch.Tensor:
|
||||
value = np.asarray(event.value, dtype=np.float32)
|
||||
if value.ndim == 0:
|
||||
return torch.full((len(targets),), float(value), dtype=torch.float32, device=self.device)
|
||||
return torch.as_tensor(value, dtype=torch.float32, device=self.device)
|
||||
|
||||
def _active(self, event: InputEvent, time_ms: float) -> bool:
|
||||
return event.start_ms - 1e-9 <= time_ms < event.start_ms + event.duration_ms - 1e-9
|
||||
|
||||
def _hybrid_edges(self, sources: torch.Tensor) -> torch.Tensor:
|
||||
starts = self.indptr[sources]
|
||||
degrees = self.indptr[sources + 1] - starts
|
||||
cumulative = torch.cumsum(degrees, 0)
|
||||
total = int(cumulative[-1].item()) if cumulative.numel() else 0
|
||||
output = torch.empty(total, dtype=torch.int64, device=self.device)
|
||||
if total:
|
||||
self.extension.fill_edges(self.indptr, sources, cumulative - degrees, output)
|
||||
return output
|
||||
|
||||
def step(self, input_batch: InputBatch | None, biological_ms: float) -> dict[str, Any]:
|
||||
steps = int(round(float(biological_ms) / self.dt))
|
||||
if not math.isclose(steps * self.dt, float(biological_ms), abs_tol=1e-8) or steps <= 0:
|
||||
raise ValueError("biological_ms must be a positive multiple of dt")
|
||||
batch = input_batch or InputBatch(events=(), batch_id="zero-input")
|
||||
batch.validate(self.n, self.dt)
|
||||
batch_start = self.simulation_time_ms
|
||||
prepared = [(e, torch.as_tensor(e.target_neuron_idx, dtype=torch.int64, device=self.device)) for e in batch.events]
|
||||
for event in batch.events:
|
||||
self.input_audit.append({
|
||||
"batch_id": batch.batch_id, "source_id": event.source_id, "channel_id": event.channel_id,
|
||||
"mode": event.stimulation_mode, "value_kind": event.value_kind,
|
||||
"start_ms": event.start_ms, "duration_ms": event.duration_ms,
|
||||
"target_count": int(len(event.target_neuron_idx)), "metadata": event.metadata,
|
||||
"external_drive_is_connectome_transmission": False,
|
||||
})
|
||||
start_event_count = sum(len(x) for x in self.spike_nodes)
|
||||
start_drive_count = sum(len(x) for x in self.drive_nodes)
|
||||
for _ in range(steps):
|
||||
global_step = self.simulation_step; time_ms = self.simulation_time_ms
|
||||
slot = global_step % self.queue_len
|
||||
self.g.mul_(self.g_decay).add_(self.delay_queue[slot]); self.delay_queue[slot].zero_()
|
||||
spike_requests: list[tuple[InputEvent, torch.Tensor, torch.Tensor]] = []
|
||||
for event, targets in prepared:
|
||||
if not self._active(event, time_ms) or targets.numel() == 0:
|
||||
continue
|
||||
values = self._event_values(event, targets)
|
||||
if event.value_kind == "poisson_rate_hz":
|
||||
selected = torch.rand(len(targets), generator=self.generator, device=self.device) < (values * self.dt / 1000.0)
|
||||
elif event.value_kind == "binary":
|
||||
selected = values != 0
|
||||
elif event.value_kind == "scalar_intensity":
|
||||
selected = values != 0
|
||||
else: # signed_modulation
|
||||
selected = torch.rand(len(targets), generator=self.generator, device=self.device) < (torch.abs(values) * self.dt / 1000.0)
|
||||
if event.stimulation_mode == "external_synaptic_drive":
|
||||
chosen = targets[selected]
|
||||
if chosen.numel():
|
||||
signed_value = values[selected] * float(event.amplitude_mv)
|
||||
if event.value_kind == "poisson_rate_hz":
|
||||
signed_value = torch.full_like(signed_value, float(event.amplitude_mv))
|
||||
self.g.index_add_(0, chosen, signed_value)
|
||||
node_np = chosen.cpu().numpy().astype(np.int32, copy=True)
|
||||
val_np = signed_value.cpu().numpy().astype(np.float32, copy=True)
|
||||
self.drive_steps.append(np.full(len(node_np), global_step, dtype=np.int32))
|
||||
self.drive_nodes.append(node_np); self.drive_values.append(val_np)
|
||||
else:
|
||||
spike_requests.append((event, targets, selected))
|
||||
self.refractory.clamp_min_(1).sub_(1)
|
||||
available = self.refractory == 0
|
||||
self.v[available] += self.alpha * (self.g[available] - (self.v[available] - self.rest))
|
||||
self.v[~available] = self.reset_value
|
||||
natural = available & (self.v >= self.threshold)
|
||||
spikes = natural.clone()
|
||||
injected = torch.zeros(self.n, dtype=torch.bool, device=self.device)
|
||||
for _event, targets, selected in spike_requests:
|
||||
accepted = selected & available[targets]
|
||||
chosen = targets[accepted]
|
||||
if chosen.numel():
|
||||
spikes[chosen] = True; injected[chosen] = ~natural[chosen]
|
||||
sources = torch.nonzero(spikes, as_tuple=False).flatten()
|
||||
self.spike_counts.add_(spikes.to(torch.int32))
|
||||
if sources.numel():
|
||||
self.v[sources] = self.reset_value; self.refractory[sources] = self.refractory_steps
|
||||
edge_ids = self._hybrid_edges(sources)
|
||||
if edge_ids.numel():
|
||||
self.delay_queue[(global_step + self.delay_steps) % self.queue_len].scatter_add_(0, self.indices[edge_ids].long(), self.weights[edge_ids])
|
||||
src_np = sources.cpu().numpy().astype(np.int32, copy=True)
|
||||
origin_np = np.where(injected[sources].cpu().numpy(), "external_spike_injection", "network_spike")
|
||||
self.spike_steps.append(np.full(len(src_np), global_step, dtype=np.int32))
|
||||
self.spike_nodes.append(src_np); self.spike_origins.append(origin_np)
|
||||
self.simulation_step += 1
|
||||
torch.cuda.synchronize()
|
||||
self.sequence_id += 1
|
||||
return {
|
||||
"api_version": API_VERSION, "sequence_id": self.sequence_id,
|
||||
"batch_id": batch.batch_id, "start_time_ms": batch_start,
|
||||
"end_time_ms": self.simulation_time_ms, "biological_ms": biological_ms,
|
||||
"new_spikes": sum(len(x) for x in self.spike_nodes) - start_event_count,
|
||||
"new_external_drive_events": sum(len(x) for x in self.drive_nodes) - start_drive_count,
|
||||
}
|
||||
|
||||
def _events(self) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
|
||||
steps = np.concatenate(self.spike_steps) if self.spike_steps else np.empty(0, np.int32)
|
||||
nodes = np.concatenate(self.spike_nodes) if self.spike_nodes else np.empty(0, np.int32)
|
||||
origins = np.concatenate(self.spike_origins) if self.spike_origins else np.empty(0, object)
|
||||
return steps, nodes, origins
|
||||
|
||||
def read_spikes(self, window_ms: float | None = None) -> dict[str, Any]:
|
||||
steps, nodes, origins = self._events()
|
||||
if window_ms is not None:
|
||||
threshold = self.simulation_step - int(round(window_ms / self.dt))
|
||||
mask = steps >= threshold; steps, nodes, origins = steps[mask], nodes[mask], origins[mask]
|
||||
return {
|
||||
"simulation_time_ms": self.simulation_time_ms,
|
||||
"step": steps, "timestamp_ms": steps.astype(np.float64) * self.dt,
|
||||
"neuron_idx": nodes, "origin": origins,
|
||||
}
|
||||
|
||||
def read_population(self, population_id: str, window_ms: float) -> dict[str, Any]:
|
||||
if population_id not in self.output_populations:
|
||||
raise KeyError(population_id)
|
||||
members = self.output_populations[population_id]
|
||||
events = self.read_spikes(window_ms)
|
||||
mask = np.isin(events["neuron_idx"], members)
|
||||
nodes = events["neuron_idx"][mask]
|
||||
counts = np.zeros(len(members), dtype=np.int32)
|
||||
if len(nodes):
|
||||
lookup = np.full(self.n, -1, dtype=np.int32); lookup[members] = np.arange(len(members), dtype=np.int32)
|
||||
np.add.at(counts, lookup[nodes], 1)
|
||||
seconds = window_ms / 1000.0
|
||||
return {
|
||||
"simulation_time_ms": self.simulation_time_ms, "population_id": population_id,
|
||||
"neuron_idx": members.copy(), "spike_counts": counts,
|
||||
"firing_rate_hz": counts.astype(np.float32) / seconds,
|
||||
"active_mask": counts > 0, "aggregate_rate_hz": float(counts.sum() / (len(members) * seconds)),
|
||||
"total_spikes": int(counts.sum()), "active_neurons": int(np.count_nonzero(counts)),
|
||||
}
|
||||
|
||||
def read_output(self, output_spec: dict[str, Any], window_ms: float) -> dict[str, Any]:
|
||||
population = output_spec.get("population", "descending")
|
||||
aggregation = output_spec.get("aggregation", "per-neuron")
|
||||
base = self.read_population(population, window_ms)
|
||||
members = base["neuron_idx"]; counts = base["spike_counts"]
|
||||
if aggregation == "per-neuron":
|
||||
return base
|
||||
if aggregation not in {"per-type", "per-class"}:
|
||||
raise ValueError("aggregation must be per-neuron, per-type, or per-class")
|
||||
labels = self.neuron_type if aggregation == "per-type" else self.neuron_class
|
||||
result: dict[str, dict[str, float | int]] = {}
|
||||
for label in sorted({"<null>" if labels[i] is None else str(labels[i]) for i in members}):
|
||||
mask = np.asarray([("<null>" if labels[i] is None else str(labels[i])) == label for i in members])
|
||||
result[label] = {"neuron_count": int(mask.sum()), "spike_count": int(counts[mask].sum()), "mean_rate_hz": float(base["firing_rate_hz"][mask].mean())}
|
||||
return {"simulation_time_ms": self.simulation_time_ms, "population_id": population, "aggregation": aggregation, "features": result}
|
||||
|
||||
def fixed_temporal_bins(self, population_id: str, bin_ms: float, window_ms: float) -> dict[str, Any]:
|
||||
if bin_ms not in self.cfg["readout"]["bins_ms"]:
|
||||
raise ValueError("unsupported fixed bin")
|
||||
ev = self.read_spikes(window_ms); members = self.output_populations[population_id]
|
||||
mask = np.isin(ev["neuron_idx"], members); times = ev["timestamp_ms"][mask]
|
||||
start = self.simulation_time_ms - window_ms
|
||||
edges = np.arange(start, self.simulation_time_ms + bin_ms + 1e-9, bin_ms)
|
||||
counts, _ = np.histogram(times, bins=edges)
|
||||
return {"simulation_time_ms": self.simulation_time_ms, "population_id": population_id, "bin_ms": bin_ms, "bin_start_ms": edges[:-1], "spike_counts": counts.astype(np.int32)}
|
||||
|
||||
def get_state_summary(self) -> dict[str, Any]:
|
||||
return {
|
||||
"api_version": API_VERSION, "runtime": self.cfg["names"]["runtime"],
|
||||
"simulation_step": self.simulation_step, "simulation_time_ms": self.simulation_time_ms,
|
||||
"sequence_id": self.sequence_id, "seed": self.seed,
|
||||
"neurons": self.n, "edges": int(self.indices.numel()),
|
||||
"active_neurons_total": int(torch.count_nonzero(self.spike_counts).item()),
|
||||
"spikes_total": int(self.spike_counts.sum().item()),
|
||||
"external_drive_events_total": sum(len(x) for x in self.drive_nodes),
|
||||
}
|
||||
|
||||
def checkpoint_state(self) -> dict[str, Any]:
|
||||
return {
|
||||
"api_version": API_VERSION, "seed": self.seed, "simulation_step": self.simulation_step,
|
||||
"sequence_id": self.sequence_id, "generator_state": self.generator.get_state().cpu(),
|
||||
"v": self.v.clone(), "g": self.g.clone(), "refractory": self.refractory.clone(),
|
||||
"delay_queue": self.delay_queue.clone(), "spike_counts": self.spike_counts.clone(),
|
||||
"spike_steps": copy.deepcopy(self.spike_steps), "spike_nodes": copy.deepcopy(self.spike_nodes),
|
||||
"spike_origins": copy.deepcopy(self.spike_origins), "drive_steps": copy.deepcopy(self.drive_steps),
|
||||
"drive_nodes": copy.deepcopy(self.drive_nodes), "drive_values": copy.deepcopy(self.drive_values),
|
||||
"input_audit": copy.deepcopy(self.input_audit),
|
||||
}
|
||||
|
||||
def restore_state(self, state: dict[str, Any]) -> dict[str, Any]:
|
||||
if state.get("api_version") != API_VERSION:
|
||||
raise ValueError("checkpoint API version mismatch")
|
||||
self.seed = int(state["seed"]); self.simulation_step = int(state["simulation_step"]); self.sequence_id = int(state["sequence_id"])
|
||||
self.generator.set_state(state["generator_state"].cpu())
|
||||
for name in ("v", "g", "refractory", "delay_queue", "spike_counts"):
|
||||
setattr(self, name, state[name].clone().to(self.device))
|
||||
for name in ("spike_steps", "spike_nodes", "spike_origins", "drive_steps", "drive_nodes", "drive_values", "input_audit"):
|
||||
setattr(self, name, copy.deepcopy(state[name]))
|
||||
return self.get_state_summary()
|
||||
|
||||
def activity_packet(self, window_ms: float, input_metadata: dict[str, Any] | None = None) -> dict[str, Any]:
|
||||
ev = self.read_spikes(window_ms)
|
||||
aggregates = {name: {k: v for k, v in self.read_population(name, window_ms).items() if k in ("active_neurons", "total_spikes", "aggregate_rate_hz")} for name in self.output_populations}
|
||||
return {
|
||||
"contract_version": ACTIVITY_STREAM_VERSION, "simulation_time_ms": self.simulation_time_ms,
|
||||
"sequence_id": self.sequence_id, "window_ms": window_ms,
|
||||
"spike_neuron_idx": ev["neuron_idx"].tolist(), "spike_timestamp_ms": ev["timestamp_ms"].tolist(),
|
||||
"population_aggregates": aggregates, "active_neuron_count": int(len(np.unique(ev["neuron_idx"]))),
|
||||
"input_channel_metadata": input_metadata or {}, "representation": "event-only",
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
2ad17a93bf22339db56ae3869af9a2a7c26ca7359d8923c693d71dde63927812 VERSION.json
|
||||
1c4e3e6f06708bfec30b24e92297f320b2b8b87ebbbe6da501f5f2db4f74762a brain_runtime.py
|
||||
9ed8e4246252bdfc590c88e2993b7e105f278c6c4943e1c688f387c1cf54aeb7 phase4d.py
|
||||
e94301362e630e59fa248f883ae09d369dce720bce970d0d2f47ef6a6ed36a2a phase4d.yaml
|
||||
4eb6390bbfcea3d03c70215cc2492ae233d90ea0d08f5fb3376d2cdcf92a7f52 MANIFEST.json
|
||||
@@ -0,0 +1,535 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4D orchestration: stable Brain I/O API, timing, readout and regression."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import shutil
|
||||
import time
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pyarrow as pa
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
from brain_runtime import (
|
||||
ACTIVITY_STREAM_VERSION, API_VERSION, INPUT_BATCH_VERSION,
|
||||
BrainRuntime, InputBatch, InputEvent, VisualAdapterV1,
|
||||
)
|
||||
|
||||
|
||||
def sha256(path: Path, chunk: int = 8 << 20) -> str:
|
||||
h = hashlib.sha256()
|
||||
with path.open("rb") as f:
|
||||
while data := f.read(chunk): h.update(data)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def write_json(path: Path, value: Any) -> None:
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
path.write_text(json.dumps(value, indent=2, sort_keys=True) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def np_json(value: Any) -> Any:
|
||||
if isinstance(value, np.ndarray): return value.tolist()
|
||||
if isinstance(value, np.generic): return value.item()
|
||||
raise TypeError(type(value).__name__)
|
||||
|
||||
|
||||
def metadata_populations(runtime: BrainRuntime) -> dict[str, np.ndarray]:
|
||||
return {k: v.copy() for k, v in runtime.output_populations.items()}
|
||||
|
||||
|
||||
def freeze_populations(exp: Path, runtime: BrainRuntime) -> dict[str, Any]:
|
||||
expressions = {
|
||||
"descending": "annotation_superclass in {'descending_neuron','descending_neuron_tbc'}",
|
||||
"motor_related": "annotation_superclass in {'vnc_motor','cb_motor'}",
|
||||
"ascending": "annotation_superclass in {'ascending_neuron','sensory_ascending','sensory_ascending_tbc','efferent_ascending'}",
|
||||
}
|
||||
nt = runtime.table["neurotransmitter"].to_pylist()
|
||||
result: dict[str, Any] = {}
|
||||
for name, nodes in metadata_populations(runtime).items():
|
||||
table = pa.table({
|
||||
"neuron_idx": pa.array(nodes, type=pa.int32()),
|
||||
"body_id": pa.array(runtime.body_id[nodes], type=pa.uint64()),
|
||||
"superclass": pa.array([runtime.superclass[i] for i in nodes], type=pa.string()),
|
||||
"type": pa.array([runtime.neuron_type[i] for i in nodes], type=pa.string()),
|
||||
"class": pa.array([runtime.neuron_class[i] for i in nodes], type=pa.string()),
|
||||
"neurotransmitter": pa.array([nt[i] for i in nodes], type=pa.string()),
|
||||
})
|
||||
pq.write_table(table, exp / "populations" / f"{name}.parquet", compression="zstd")
|
||||
nts = Counter("<null>" if nt[i] is None else str(nt[i]) for i in nodes)
|
||||
result[name] = {
|
||||
"selection_expression": expressions[name], "count": int(len(nodes)),
|
||||
"neurotransmitter_statistics": dict(sorted(nts.items())),
|
||||
"membership_file": str(exp / "populations" / f"{name}.parquet"),
|
||||
}
|
||||
write_json(exp / "populations/summary.json", result)
|
||||
return result
|
||||
|
||||
|
||||
def audit_visual_sides(exp: Path, runtime: BrainRuntime, adapter: VisualAdapterV1) -> dict[str, Any]:
|
||||
values = runtime.table["annotation_rootSide"].to_pylist() if "annotation_rootSide" in runtime.table.column_names else [None] * runtime.n
|
||||
result: dict[str, Any] = {}
|
||||
for mode in ("engineering-first-order", "engineering-second-order"):
|
||||
nodes = adapter.population(mode); c = Counter("<null>" if values[i] is None else str(values[i]) for i in nodes)
|
||||
coverage = 1.0 - c.get("<null>", 0) / len(nodes)
|
||||
reliable = coverage >= 0.90 and c.get("L", 0) > 0 and c.get("R", 0) > 0
|
||||
result[mode] = {"field": "annotation_rootSide", "counts": dict(c), "non_null_coverage": coverage, "left_right_patterns_enabled": reliable}
|
||||
write_json(exp / "visual-adapter/left-right-audit.json", result)
|
||||
return result
|
||||
|
||||
|
||||
def freeze_contracts(exp: Path, cfg: dict[str, Any], runtime: BrainRuntime, adapter: VisualAdapterV1) -> dict[str, Any]:
|
||||
input_schema = {
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"title": "MaleCNS InputBatch",
|
||||
"version": INPUT_BATCH_VERSION,
|
||||
"type": "object",
|
||||
"required": ["version", "batch_id", "events"],
|
||||
"properties": {
|
||||
"version": {"const": INPUT_BATCH_VERSION},
|
||||
"batch_id": {"type": "string"},
|
||||
"events": {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"required": ["target_neuron_idx", "stimulation_mode", "value_kind", "value", "start_ms", "duration_ms", "source_id", "channel_id"],
|
||||
"properties": {
|
||||
"target_neuron_idx": {"type": "array", "items": {"type": "integer", "minimum": 0, "maximum": 166699}},
|
||||
"stimulation_mode": {"enum": ["spike_injection", "external_synaptic_drive"]},
|
||||
"value_kind": {"enum": ["binary", "scalar_intensity", "poisson_rate_hz", "signed_modulation"]},
|
||||
"value": {"oneOf": [{"type": "number"}, {"type": "array", "items": {"type": "number"}}]},
|
||||
"start_ms": {"type": "number", "minimum": 0},
|
||||
"duration_ms": {"type": "number", "exclusiveMinimum": 0},
|
||||
"source_id": {"type": "string"},
|
||||
"channel_id": {"type": "string"},
|
||||
"amplitude_mv": {"type": "number"},
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
"timestamp_unit": "biological ms",
|
||||
"external_drive_is_connectome_transmission": False,
|
||||
}
|
||||
write_json(exp / "configs/input-batch.schema.json", input_schema)
|
||||
mode_records: dict[str, Any] = {}
|
||||
for mode, nodes in adapter.modes.items():
|
||||
path = exp / "visual-adapter" / f"{mode}.parquet"
|
||||
pq.write_table(pa.table({"neuron_idx": pa.array(nodes.astype(np.int32), type=pa.int32()), "body_id": pa.array(runtime.body_id[nodes], type=pa.uint64())}), path, compression="zstd")
|
||||
spec = cfg["visual_adapter"]["modes"][mode]
|
||||
mode_records[mode] = {"count": int(len(nodes)), "membership_file": str(path), **spec}
|
||||
write_json(exp / "visual-adapter/membership-summary.json", mode_records)
|
||||
readout = {
|
||||
"api_version": API_VERSION, "methods": ["read_spikes", "read_population", "read_output", "fixed_temporal_bins"],
|
||||
"features": ["spike count vector", "firing-rate vector", "active mask", "population aggregate rate", "excitatory/inhibitory activity", "descending feature vector", "motor-related feature vector"],
|
||||
"aggregations": ["per-neuron", "per-type", "per-class"], "fixed_bins_ms": cfg["readout"]["bins_ms"],
|
||||
"action_decoder": False,
|
||||
}
|
||||
write_json(exp / "readout/readout-contract.json", readout)
|
||||
return {"input_batch_schema": str(exp / "configs/input-batch.schema.json"), "visual_modes": mode_records, "readout_contract": readout}
|
||||
|
||||
|
||||
def patterns(nodes: np.ndarray, cfg: dict[str, Any]) -> dict[str, np.ndarray]:
|
||||
rng = np.random.default_rng(int(cfg["synthetic_patterns"]["deterministic_seed"]))
|
||||
order = rng.permutation(len(nodes))
|
||||
sparse_n = max(1, int(round(len(nodes) * float(cfg["synthetic_patterns"]["sparse_fraction"]))))
|
||||
dense_n = max(1, int(round(len(nodes) * float(cfg["synthetic_patterns"]["dense_fraction"]))))
|
||||
moving_n = max(1, int(round(len(nodes) * float(cfg["synthetic_patterns"]["moving_fraction"]))))
|
||||
def vector(selection: np.ndarray, rate: float) -> np.ndarray:
|
||||
out = np.zeros(len(nodes), dtype=np.float32); out[selection] = rate; return out
|
||||
rate = float(cfg["visual_adapter"]["synthetic_rate_hz"])
|
||||
return {
|
||||
"sparse-A": vector(order[:sparse_n], rate),
|
||||
"sparse-B": vector(order[sparse_n:2*sparse_n], rate),
|
||||
"dense": vector(order[:dense_n], rate),
|
||||
"moving": vector(order[2*sparse_n:2*sparse_n+moving_n], rate),
|
||||
"zero": np.zeros(len(nodes), dtype=np.float32),
|
||||
}
|
||||
|
||||
|
||||
def direct_batch(nodes: np.ndarray, mode: str, rate_hz: float, start_ms: float, duration_ms: float, source: str) -> InputBatch:
|
||||
return InputBatch(events=(InputEvent(
|
||||
target_neuron_idx=nodes.astype(np.int64), stimulation_mode="spike_injection",
|
||||
value_kind="poisson_rate_hz", value=float(rate_hz), start_ms=start_ms,
|
||||
duration_ms=duration_ms, source_id=source, channel_id=mode,
|
||||
metadata={"biological_fidelity": "reference_population", "external_drive_is_connectome_transmission": False},
|
||||
),), batch_id=f"{source}:{mode}:{start_ms}")
|
||||
|
||||
|
||||
def feature(runtime: BrainRuntime, window: float) -> np.ndarray:
|
||||
d = runtime.read_population("descending", window)["spike_counts"]
|
||||
m = runtime.read_population("motor_related", window)["spike_counts"]
|
||||
return np.concatenate([d, m]).astype(np.float32)
|
||||
|
||||
|
||||
def output_stats(runtime: BrainRuntime, window: float) -> dict[str, Any]:
|
||||
d = runtime.read_population("descending", window); m = runtime.read_population("motor_related", window)
|
||||
return {
|
||||
"descending_active": d["active_neurons"], "descending_spikes": d["total_spikes"],
|
||||
"motor_active": m["active_neurons"], "motor_spikes": m["total_spikes"],
|
||||
"descending_rate_hz": d["aggregate_rate_hz"], "motor_rate_hz": m["aggregate_rate_hz"],
|
||||
}
|
||||
|
||||
|
||||
def population_distances(runtime: BrainRuntime, sources: np.ndarray) -> np.ndarray:
|
||||
return runtime.runtime_module.p4b.directed_distances(runtime.indptr_np, runtime.indices_np, sources.astype(np.int64), runtime.n)
|
||||
|
||||
|
||||
def propagation_depth(runtime: BrainRuntime, distance: np.ndarray) -> int:
|
||||
active = runtime.spike_counts.cpu().numpy() > 0
|
||||
values = distance[active & (distance >= 0)]
|
||||
return int(values.max(initial=0))
|
||||
|
||||
|
||||
def make_source_batch(cfg: dict[str, Any], runtime: BrainRuntime, adapter: VisualAdapterV1,
|
||||
source: str, start: float, duration: float, pattern_cache: dict[str, dict[str, np.ndarray]]) -> tuple[InputBatch, np.ndarray]:
|
||||
if source.startswith("engineering-"):
|
||||
mode = source
|
||||
p = pattern_cache[mode]["sparse-A"]
|
||||
batch = adapter.encode(mode, p, start, duration, "phase4d-window", "sparse-A")
|
||||
targets = adapter.population(mode)[p != 0]
|
||||
return batch, targets
|
||||
mask = runtime.population_masks[source]
|
||||
nodes = np.flatnonzero(mask).astype(np.int64)
|
||||
rate = 10.0 if source == "auditory" else 50.0
|
||||
return direct_batch(nodes, source, rate, start, duration, "phase4d-window"), nodes
|
||||
|
||||
|
||||
def window_experiments(exp: Path, cfg: dict[str, Any], runtime: BrainRuntime,
|
||||
adapter: VisualAdapterV1, pattern_cache: dict[str, dict[str, np.ndarray]]) -> dict[str, Any]:
|
||||
records: list[dict[str, Any]] = []; distances: dict[str, np.ndarray] = {}
|
||||
for source in cfg["window_experiment"]["sources"]:
|
||||
_, targets = make_source_batch(cfg, runtime, adapter, source, 0.0, 5.0, pattern_cache)
|
||||
distances[source] = population_distances(runtime, targets)
|
||||
for window in cfg["window_experiment"]["windows_ms"]:
|
||||
for seed in cfg["window_experiment"]["seeds"]:
|
||||
outcomes = []
|
||||
for repeat in range(2):
|
||||
runtime.reset(int(seed)); batch, _ = make_source_batch(cfg, runtime, adapter, source, 0.0, float(window), pattern_cache)
|
||||
torch.cuda.synchronize(); started = time.perf_counter(); step_result = runtime.step(batch, float(window)); torch.cuda.synchronize(); wall = time.perf_counter() - started
|
||||
stats = output_stats(runtime, float(window)); f = feature(runtime, float(window))
|
||||
outcomes.append((runtime.read_spikes(), runtime.v.cpu().numpy().copy(), runtime.g.cpu().numpy().copy(), f, stats, step_result, wall))
|
||||
a, b = outcomes
|
||||
deterministic = np.array_equal(a[0]["step"], b[0]["step"]) and np.array_equal(a[0]["neuron_idx"], b[0]["neuron_idx"]) and np.array_equal(a[3], b[3])
|
||||
input_count = int(a[5]["new_spikes"] if source in ("auditory", "olfactory") else a[5]["new_external_drive_events"])
|
||||
rec = {
|
||||
"source": source, "window_ms": float(window), "seed": int(seed),
|
||||
"wall_latency_seconds": a[6], "max_theoretical_control_hz": 1.0 / a[6],
|
||||
"realtime_factor_wall_over_biological": a[6] / (float(window) / 1000.0),
|
||||
"input_spikes_or_drive_events": input_count,
|
||||
"total_spikes": int(len(a[0]["neuron_idx"])),
|
||||
"propagation_depth": propagation_depth(runtime, distances[source]),
|
||||
**a[4], "feature_l2_norm": float(np.linalg.norm(a[3])),
|
||||
"repeat_spike_events_exact": deterministic,
|
||||
"repeat_feature_exact": bool(np.array_equal(a[3], b[3])),
|
||||
"repeat_final_voltage_max_abs_error": float(np.max(np.abs(a[1] - b[1]))),
|
||||
"repeat_final_g_max_abs_error": float(np.max(np.abs(a[2] - b[2]))),
|
||||
}
|
||||
records.append(rec)
|
||||
write_json(exp / "latency/window-trials.json", records)
|
||||
summary: dict[str, Any] = {"trial_count": len(records), "all_repeat_deterministic": all(x["repeat_spike_events_exact"] for x in records)}
|
||||
for source in cfg["window_experiment"]["sources"]:
|
||||
rows = [x for x in records if x["source"] == source]
|
||||
summary[source] = {}
|
||||
for window in cfg["window_experiment"]["windows_ms"]:
|
||||
r = [x for x in rows if x["window_ms"] == float(window)]
|
||||
summary[source][str(window)] = {
|
||||
"wall_latency_seconds_mean": float(np.mean([x["wall_latency_seconds"] for x in r])),
|
||||
"max_theoretical_control_hz_mean": float(np.mean([x["max_theoretical_control_hz"] for x in r])),
|
||||
"descending_active_mean": float(np.mean([x["descending_active"] for x in r])),
|
||||
"motor_active_mean": float(np.mean([x["motor_active"] for x in r])),
|
||||
"feature_l2_norm_mean": float(np.mean([x["feature_l2_norm"] for x in r])),
|
||||
}
|
||||
def shortest(key: str) -> float | None:
|
||||
candidates = sorted({x["window_ms"] for x in records if x["source"].startswith("engineering-") and x[key] > 0})
|
||||
return candidates[0] if candidates else None
|
||||
summary["shortest_engineering_descending_signal_ms"] = shortest("descending_active")
|
||||
summary["shortest_engineering_motor_signal_ms"] = shortest("motor_active")
|
||||
write_json(exp / "latency/summary.json", summary)
|
||||
return {"records": records, "summary": summary}
|
||||
|
||||
|
||||
def continuous_control(exp: Path, cfg: dict[str, Any], runtime: BrainRuntime,
|
||||
adapter: VisualAdapterV1, pattern_cache: dict[str, dict[str, np.ndarray]]) -> dict[str, Any]:
|
||||
output: dict[str, Any] = {}
|
||||
duration = float(cfg["continuous_control"]["biological_duration_ms"])
|
||||
sequence = cfg["continuous_control"]["sequence"]
|
||||
mode = "engineering-first-order"
|
||||
for period in cfg["continuous_control"]["periods_ms"]:
|
||||
period = float(period); runtime.reset(int(cfg["continuous_control"]["seed"]))
|
||||
intervals: list[dict[str, Any]] = []; previous = None
|
||||
total_wall = 0.0
|
||||
for k, start in enumerate(np.arange(0.0, duration, period)):
|
||||
pattern_name = sequence[k % len(sequence)]
|
||||
values = pattern_cache[mode][pattern_name]
|
||||
batch = adapter.encode(mode, values, float(start), period, "continuous-control", pattern_name)
|
||||
torch.cuda.synchronize(); t0 = time.perf_counter(); step_result = runtime.step(batch, period); torch.cuda.synchronize(); wall = time.perf_counter() - t0; total_wall += wall
|
||||
f = feature(runtime, period); stats = output_stats(runtime, period)
|
||||
recent = runtime.read_spikes(period); adapter_nodes = adapter.population(mode)
|
||||
target_spikes = int(np.count_nonzero(np.isin(recent["neuron_idx"], adapter_nodes)))
|
||||
latency = None
|
||||
ev = runtime.read_spikes(period)
|
||||
output_members = np.concatenate([runtime.output_populations["descending"], runtime.output_populations["motor_related"]])
|
||||
omask = np.isin(ev["neuron_idx"], output_members)
|
||||
if np.any(omask): latency = float(ev["timestamp_ms"][omask].min() - start)
|
||||
intervals.append({
|
||||
"interval": k, "start_ms": float(start), "end_ms": float(start + period), "pattern": pattern_name,
|
||||
"wall_seconds": wall, "response_latency_ms": latency, **stats,
|
||||
"feature_norm": float(np.linalg.norm(f)),
|
||||
"feature_change_l2": None if previous is None else float(np.linalg.norm(f - previous)),
|
||||
"input_spikes": step_result["new_spikes"], "drive_events": step_result["new_external_drive_events"],
|
||||
"adapter_target_spikes": target_spikes,
|
||||
})
|
||||
previous = f
|
||||
zero = [x for x in intervals if x["pattern"] == "zero"]
|
||||
change_latencies = [x["response_latency_ms"] for x in intervals if x["response_latency_ms"] is not None]
|
||||
output[str(int(period))] = {
|
||||
"period_ms": period, "continuous_state_without_reset": True, "interval_count": len(intervals),
|
||||
"wall_seconds": total_wall, "biological_ms": duration,
|
||||
"median_response_latency_ms": float(np.median(change_latencies)) if change_latencies else None,
|
||||
"max_response_latency_ms": float(np.max(change_latencies)) if change_latencies else None,
|
||||
"zero_input_descending_spikes": int(sum(x["descending_spikes"] for x in zero)),
|
||||
"zero_input_motor_spikes": int(sum(x["motor_spikes"] for x in zero)),
|
||||
"zero_input_adapter_target_spikes": int(sum(x["adapter_target_spikes"] for x in zero)),
|
||||
"zero_input_intervals_with_activity": int(sum((x["descending_spikes"] + x["motor_spikes"]) > 0 for x in zero)),
|
||||
"intervals": intervals,
|
||||
}
|
||||
write_json(exp / "continuous-control" / f"period-{int(period)}ms.json", output[str(int(period))])
|
||||
write_json(exp / "continuous-control/summary.json", {k: {x: v[x] for x in v if x != "intervals"} for k, v in output.items()})
|
||||
return output
|
||||
|
||||
|
||||
def metric_pair(a: np.ndarray, b: np.ndarray) -> dict[str, float]:
|
||||
an, bn = float(np.linalg.norm(a)), float(np.linalg.norm(b))
|
||||
cosine = float(np.dot(a, b) / (an * bn)) if an and bn else (1.0 if an == bn else 0.0)
|
||||
if np.std(a) and np.std(b): corr = float(np.corrcoef(a, b)[0, 1])
|
||||
else: corr = 1.0 if np.array_equal(a, b) else 0.0
|
||||
aa, bb = a > 0, b > 0; union = np.count_nonzero(aa | bb)
|
||||
return {"cosine_distance": 1.0 - cosine, "euclidean_distance": float(np.linalg.norm(a - b)), "correlation": corr, "active_population_overlap_jaccard": float(np.count_nonzero(aa & bb) / union) if union else 1.0}
|
||||
|
||||
|
||||
def separability(exp: Path, cfg: dict[str, Any], runtime: BrainRuntime,
|
||||
adapter: VisualAdapterV1, pattern_cache: dict[str, dict[str, np.ndarray]], side_audit: dict[str, Any]) -> dict[str, Any]:
|
||||
mode = "engineering-first-order"; pats = dict(pattern_cache[mode])
|
||||
pats.pop("zero")
|
||||
if side_audit[mode]["left_right_patterns_enabled"]:
|
||||
root_side = runtime.table["annotation_rootSide"].to_pylist(); nodes = adapter.population(mode); rate = float(cfg["visual_adapter"]["synthetic_rate_hz"])
|
||||
pats["left"] = np.asarray([rate if root_side[i] == "L" else 0.0 for i in nodes], dtype=np.float32)
|
||||
pats["right"] = np.asarray([rate if root_side[i] == "R" else 0.0 for i in nodes], dtype=np.float32)
|
||||
features: dict[str, list[np.ndarray]] = {}
|
||||
trial_rows: list[dict[str, Any]] = []
|
||||
for name, values in pats.items():
|
||||
features[name] = []
|
||||
for seed in (44301, 44302, 44303):
|
||||
runtime.reset(seed); batch = adapter.encode(mode, values, 0.0, 100.0, "separability", name)
|
||||
runtime.step(batch, 100.0); f = feature(runtime, 100.0); features[name].append(f)
|
||||
trial_rows.append({"pattern": name, "seed": seed, "active_channels": int(np.count_nonzero(values)), "feature_norm": float(np.linalg.norm(f)), **output_stats(runtime, 100.0)})
|
||||
means = {k: np.mean(v, axis=0) for k, v in features.items()}
|
||||
pairwise: dict[str, Any] = {}
|
||||
names = sorted(means)
|
||||
for i, a in enumerate(names):
|
||||
for b in names[i+1:]: pairwise[f"{a}__{b}"] = metric_pair(means[a], means[b])
|
||||
within = [float(np.linalg.norm(x - means[k])) for k, xs in features.items() for x in xs]
|
||||
between = [v["euclidean_distance"] for v in pairwise.values()]
|
||||
matrix = np.stack([x for k in names for x in features[k]])
|
||||
centered = matrix - matrix.mean(axis=0, keepdims=True)
|
||||
u, s, _ = np.linalg.svd(centered, full_matrices=False)
|
||||
coords = u[:, :2] * s[:2]
|
||||
labels = [k for k in names for _ in features[k]]
|
||||
pca_table = pa.table({"pattern": labels, "pc1": coords[:, 0], "pc2": coords[:, 1]})
|
||||
pq.write_table(pca_table, exp / "separability/pca-coordinates.parquet", compression="zstd")
|
||||
result = {
|
||||
"classifier_trained": False, "action_decoder_created": False,
|
||||
"patterns": names, "trial_statistics": trial_rows, "pairwise": pairwise,
|
||||
"within_condition_distance_mean": float(np.mean(within)),
|
||||
"between_condition_distance_mean": float(np.mean(between)),
|
||||
"between_to_within_ratio": float(np.mean(between) / np.mean(within)) if np.mean(within) else None,
|
||||
"all_pairwise_mean_vectors_distinct": all(v["euclidean_distance"] > 0 for v in pairwise.values()),
|
||||
"pca_usage": "offline visualization analysis only; not an action decoder",
|
||||
}
|
||||
write_json(exp / "separability/summary.json", result)
|
||||
return result
|
||||
|
||||
|
||||
def api_tests(exp: Path, runtime: BrainRuntime) -> dict[str, Any]:
|
||||
nodes = np.flatnonzero(runtime.population_masks["auditory"]).astype(np.int64)
|
||||
batch20 = direct_batch(nodes, "auditory", 10, 0.0, 20.0, "api-test")
|
||||
runtime.reset(44400); runtime.step(batch20, 20.0); one = runtime.read_spikes(); one_v = runtime.v.cpu().numpy().copy()
|
||||
runtime.reset(44400); runtime.step(batch20, 10.0); checkpoint = runtime.checkpoint_state(); runtime.step(batch20, 10.0); two = runtime.read_spikes(); two_v = runtime.v.cpu().numpy().copy()
|
||||
runtime.restore_state(checkpoint); runtime.step(batch20, 10.0); replay = runtime.read_spikes(); replay_v = runtime.v.cpu().numpy().copy()
|
||||
per_neuron = runtime.read_output({"population": "descending", "aggregation": "per-neuron"}, 20.0)
|
||||
per_type = runtime.read_output({"population": "descending", "aggregation": "per-type"}, 20.0)
|
||||
per_class = runtime.read_output({"population": "motor_related", "aggregation": "per-class"}, 20.0)
|
||||
bins = {str(x): runtime.fixed_temporal_bins("descending", float(x), 20.0)["spike_counts"].tolist() for x in runtime.cfg["readout"]["bins_ms"] if float(x) <= 20.0}
|
||||
result = {
|
||||
"reset_same_seed_continuous_chunk_spikes_exact": bool(np.array_equal(one["step"], two["step"]) and np.array_equal(one["neuron_idx"], two["neuron_idx"])),
|
||||
"continuous_step_does_not_reset_clock": runtime.simulation_time_ms == 20.0,
|
||||
"checkpoint_restore_spikes_exact": bool(np.array_equal(two["step"], replay["step"]) and np.array_equal(two["neuron_idx"], replay["neuron_idx"])),
|
||||
"checkpoint_restore_voltage_exact": bool(np.array_equal(two_v, replay_v)),
|
||||
"single_vs_chunk_voltage_max_abs_error": float(np.max(np.abs(one_v - two_v))),
|
||||
"timestamps_present": "timestamp_ms" in replay,
|
||||
"per_neuron_readout_shape": int(len(per_neuron["neuron_idx"])),
|
||||
"per_type_aggregation_labels": int(len(per_type["features"])),
|
||||
"per_class_aggregation_labels": int(len(per_class["features"])),
|
||||
"fixed_temporal_bins_tested": bins,
|
||||
"readout_modes_passed": len(per_neuron["neuron_idx"]) == len(runtime.output_populations["descending"]) and len(per_type["features"]) > 0 and len(per_class["features"]) > 0,
|
||||
"all_passed": False,
|
||||
}
|
||||
result["all_passed"] = all(result[k] for k in ("reset_same_seed_continuous_chunk_spikes_exact", "continuous_step_does_not_reset_clock", "checkpoint_restore_spikes_exact", "checkpoint_restore_voltage_exact", "timestamps_present", "readout_modes_passed"))
|
||||
write_json(exp / "audit/api-tests.json", result)
|
||||
return result
|
||||
|
||||
|
||||
def regression(exp: Path, cfg: dict[str, Any], runtime: BrainRuntime) -> dict[str, Any]:
|
||||
root = Path(cfg["paths"]["phase4c_root"]) / "regression/reference"
|
||||
results: dict[str, Any] = {}; tolerance = float(cfg["regression"]["float_absolute_tolerance"])
|
||||
for spec in cfg["regression"]["reference_trials"]:
|
||||
name = spec["name"]; ref_dir = root / name
|
||||
events = pq.read_table(ref_dir / "spike-events.parquet", columns=["step", "neuron_idx"])
|
||||
ref_steps = events["step"].to_numpy(); ref_nodes = events["neuron_idx"].to_numpy()
|
||||
ref_counts = np.load(ref_dir / "per-neuron-spike-counts.int32.npy")
|
||||
ref_v = np.load(ref_dir / "final-voltage.float32.npy"); ref_g = np.load(ref_dir / "final-g.float32.npy")
|
||||
nodes = np.flatnonzero(runtime.population_masks[spec["population"]]).astype(np.int64)
|
||||
batch = direct_batch(nodes, spec["population"], float(spec["rate_hz"]), 0.0, float(spec["duration_ms"]), "phase4c-regression")
|
||||
runtime.reset(int(spec["seed"])); t0 = time.perf_counter(); runtime.step(batch, float(spec["duration_ms"])); wall = time.perf_counter() - t0
|
||||
got = runtime.read_spikes(); counts = runtime.spike_counts.cpu().numpy()
|
||||
vd = np.abs(runtime.v.cpu().numpy() - ref_v); gd = np.abs(runtime.g.cpu().numpy() - ref_g)
|
||||
record = {
|
||||
"spike_sequence_exact": bool(np.array_equal(ref_steps, got["step"]) and np.array_equal(ref_nodes, got["neuron_idx"])),
|
||||
"spike_counts_exact": bool(np.array_equal(ref_counts, counts)),
|
||||
"final_voltage_max_abs_error": float(vd.max(initial=0)), "final_voltage_mean_abs_error": float(vd.mean()),
|
||||
"final_g_max_abs_error": float(gd.max(initial=0)), "final_g_mean_abs_error": float(gd.mean()),
|
||||
"state_within_phase4c_tolerance": bool(vd.max(initial=0) <= tolerance and gd.max(initial=0) <= tolerance),
|
||||
"wall_seconds": wall, "event_count": int(len(got["neuron_idx"])),
|
||||
}
|
||||
record["passed"] = record["spike_sequence_exact"] and record["spike_counts_exact"] and record["state_within_phase4c_tolerance"]
|
||||
results[name] = record
|
||||
write_json(exp / "regression" / f"{name}.json", record)
|
||||
write_json(exp / "regression/summary.json", {"all_passed": all(x["passed"] for x in results.values()), "trials": results})
|
||||
return results
|
||||
|
||||
|
||||
def stream_contract(exp: Path, runtime: BrainRuntime) -> dict[str, Any]:
|
||||
schema = {
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema", "title": "MaleCNS Activity Packet",
|
||||
"contract_version": ACTIVITY_STREAM_VERSION, "type": "object",
|
||||
"required": ["contract_version", "simulation_time_ms", "sequence_id", "spike_neuron_idx", "population_aggregates", "active_neuron_count", "input_channel_metadata"],
|
||||
"properties": {
|
||||
"simulation_time_ms": {"type": "number"}, "sequence_id": {"type": "integer"},
|
||||
"spike_neuron_idx": {"type": "array", "items": {"type": "integer"}},
|
||||
"spike_timestamp_ms": {"type": "array", "items": {"type": "number"}},
|
||||
"population_aggregates": {"type": "object"}, "active_neuron_count": {"type": "integer"},
|
||||
"input_channel_metadata": {"type": "object"}, "representation": {"const": "event-only"},
|
||||
},
|
||||
"dense_time_by_neuron_matrix_forbidden": True,
|
||||
"supported_modes": ["event-only stream", "temporal aggregation", "saved-event replay"],
|
||||
}
|
||||
write_json(exp / "activity-stream/activity-packet.schema.json", schema)
|
||||
packet = runtime.activity_packet(20.0, {"source": "contract-test"})
|
||||
path = exp / "activity-stream/replay.jsonl"
|
||||
path.write_text(json.dumps(packet, default=np_json) + "\n", encoding="utf-8")
|
||||
replay = json.loads(path.read_text().splitlines()[0])
|
||||
result = {"contract_frozen": True, "version": ACTIVITY_STREAM_VERSION, "event_only": True, "replay_exact": replay == json.loads(json.dumps(packet, default=np_json)), "schema": str(exp / "activity-stream/activity-packet.schema.json")}
|
||||
write_json(exp / "activity-stream/replay-test.json", result)
|
||||
return result
|
||||
|
||||
|
||||
def freeze_io(exp: Path, cfg: dict[str, Any], code_dir: Path, api_tests_result: dict[str, Any]) -> dict[str, Any]:
|
||||
root = Path(cfg["paths"]["model_root"]) / "io-v1"; root.mkdir(parents=True, exist_ok=True)
|
||||
for name in ("VERSION.json", "MANIFEST.json", "checksums.sha256"):
|
||||
path = root / name
|
||||
if path.exists(): path.chmod(0o644)
|
||||
sources = [code_dir / "brain_runtime.py", code_dir / "phase4d.py", code_dir / "phase4d.yaml"]
|
||||
for src in sources:
|
||||
dst = root / src.name
|
||||
if dst.exists(): dst.chmod(0o644)
|
||||
shutil.copy2(src, dst); dst.chmod(0o444)
|
||||
version = {
|
||||
"name": cfg["names"]["io_api"], "api_version": API_VERSION,
|
||||
"input_batch_version": INPUT_BATCH_VERSION, "activity_stream_version": ACTIVITY_STREAM_VERSION,
|
||||
"runtime_dependency": cfg["names"]["runtime"], "game_agnostic": True,
|
||||
"api_methods": ["reset", "step", "read_spikes", "read_population", "read_output", "get_state_summary", "checkpoint_state", "restore_state"],
|
||||
"continuous_gpu_state": True, "api_tests": api_tests_result,
|
||||
"prohibitions": cfg["prohibitions"],
|
||||
}
|
||||
write_json(root / "VERSION.json", version); (root / "VERSION.json").chmod(0o444)
|
||||
files = sorted(p for p in root.iterdir() if p.is_file() and p.name not in ("MANIFEST.json", "checksums.sha256"))
|
||||
manifest = {"artifact": "io-v1", "files": [{"path": p.name, "bytes": p.stat().st_size, "sha256": sha256(p)} for p in files]}
|
||||
write_json(root / "MANIFEST.json", manifest); (root / "MANIFEST.json").chmod(0o444)
|
||||
targets = files + [root / "MANIFEST.json"]
|
||||
(root / "checksums.sha256").write_text("".join(f"{sha256(p)} {p.name}\n" for p in targets), encoding="utf-8"); (root / "checksums.sha256").chmod(0o444)
|
||||
return version
|
||||
|
||||
|
||||
def seal(exp: Path, cfg: dict[str, Any], code: list[Path], input_files: list[Path], summary: dict[str, Any]) -> None:
|
||||
controls = {exp / "manifests/manifest.json", exp / "manifests/checksums.sha256"}
|
||||
outputs = sorted(p for p in exp.rglob("*") if p.is_file() and p not in controls)
|
||||
manifest = {
|
||||
"phase": "4D", "status": summary["status"], "created_unix_time": time.time(),
|
||||
"software": {"python": platform.python_version(), "torch": torch.__version__, "numpy": np.__version__, "pyarrow": pa.__version__},
|
||||
"code": [{"path": str(p), "sha256": sha256(p)} for p in code],
|
||||
"immutable_inputs": [{"path": str(p), "sha256": sha256(p)} for p in input_files],
|
||||
"outputs": [{"path": str(p), "bytes": p.stat().st_size, "sha256": sha256(p)} for p in outputs],
|
||||
"prohibitions": cfg["prohibitions"],
|
||||
}
|
||||
write_json(exp / "manifests/manifest.json", manifest); targets = outputs + [exp / "manifests/manifest.json"]
|
||||
(exp / "manifests/checksums.sha256").write_text("".join(f"{sha256(p)} {p.relative_to(exp)}\n" for p in targets), encoding="utf-8")
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", required=True, type=Path); args = ap.parse_args()
|
||||
cfg = yaml.safe_load(args.config.read_text(encoding="utf-8")); exp = Path(cfg["paths"]["experiment_root"])
|
||||
for sub in ("configs", "populations", "visual-adapter", "readout", "latency", "continuous-control", "separability", "activity-stream", "regression", "audit", "compiler-reports/extension-build", "manifests"):
|
||||
(exp / sub).mkdir(parents=True, exist_ok=True)
|
||||
shutil.copy2(args.config, exp / "configs/phase4d.yaml")
|
||||
runtime_root = Path(cfg["paths"]["runtime_root"]); phase4c = Path(cfg["paths"]["phase4c_root"])
|
||||
immutable = [runtime_root / "VERSION.json", runtime_root / "checksums.sha256", phase4c / "manifests/checksums.sha256", phase4c / "visual-audit/summary.json"]
|
||||
before = {str(p): sha256(p) for p in immutable}
|
||||
runtime = BrainRuntime(args.config, exp / "audit/extension-build")
|
||||
adapter = VisualAdapterV1(cfg, phase4c)
|
||||
population_summary = freeze_populations(exp, runtime)
|
||||
contracts = freeze_contracts(exp, cfg, runtime, adapter)
|
||||
side = audit_visual_sides(exp, runtime, adapter)
|
||||
pattern_cache = {mode: patterns(adapter.population(mode), cfg) for mode in ("engineering-first-order", "engineering-second-order")}
|
||||
pattern_summary = {mode: {name: int(np.count_nonzero(v)) for name, v in pats.items()} for mode, pats in pattern_cache.items()}
|
||||
write_json(exp / "visual-adapter/synthetic-patterns.json", pattern_summary)
|
||||
write_json(exp / "visual-adapter/modes.json", cfg["visual_adapter"])
|
||||
api = api_tests(exp, runtime)
|
||||
windows = window_experiments(exp, cfg, runtime, adapter, pattern_cache)
|
||||
continuous = continuous_control(exp, cfg, runtime, adapter, pattern_cache)
|
||||
sep = separability(exp, cfg, runtime, adapter, pattern_cache, side)
|
||||
reg = regression(exp, cfg, runtime)
|
||||
stream = stream_contract(exp, runtime)
|
||||
after = {str(p): sha256(p) for p in immutable}
|
||||
if before != after: raise RuntimeError("Phase 4C/runtime-v1 immutable input changed")
|
||||
reg_pass = all(x["passed"] for x in reg.values())
|
||||
status = "PASS" if api["all_passed"] and reg_pass and stream["contract_frozen"] else "FAILED"
|
||||
summary = {
|
||||
"status": status, "brain_runtime_api_frozen": api["all_passed"], "api_tests": api,
|
||||
"visual_adapter_modes_complete": True, "output_populations": population_summary,
|
||||
"contracts": contracts,
|
||||
"window_experiments": windows["summary"],
|
||||
"continuous_control": {k: {x: v[x] for x in v if x != "intervals"} for k, v in continuous.items()},
|
||||
"separability": sep, "activity_stream": stream,
|
||||
"phase4c_regression_all_passed": reg_pass, "phase4c_regression": reg,
|
||||
"immutable_inputs_unchanged": True, "left_right_audit": side,
|
||||
"no_stable_retro_rom_buttons_training_or_web": True,
|
||||
}
|
||||
write_json(exp / "audit/summary.json", summary)
|
||||
code_dir = Path(__file__).resolve().parent
|
||||
io_version = freeze_io(exp, cfg, code_dir, api); summary["io_version"] = io_version
|
||||
write_json(exp / "audit/summary.json", summary)
|
||||
seal(exp, cfg, [code_dir / "brain_runtime.py", Path(__file__).resolve(), args.config.resolve()], immutable, summary)
|
||||
if status != "PASS": raise RuntimeError("Phase 4D validation failed; see audit/summary.json")
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,102 @@
|
||||
phase: 4D
|
||||
names:
|
||||
runtime: MaleCNS-LIF-runtime-v1
|
||||
io_api: MaleCNS-BrainIO-v1
|
||||
visual_adapter: MaleCNS-VisualAdapter-v1
|
||||
activity_stream: MaleCNS-ActivityStream-v1
|
||||
|
||||
paths:
|
||||
connectome_root: /data/digital-drosophila/datasets/male-cns-v1.0/processed/neuron-only
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif
|
||||
runtime_root: /data/digital-drosophila/models/male-cns-lif/runtime-v1
|
||||
phase4c_root: /data/digital-drosophila/experiments/phase4c-runtime
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4d-io
|
||||
|
||||
device:
|
||||
required_name_substring: RTX 4090
|
||||
launcher_visible_gpu_uuid: GPU-79f3a85d-adce-7833-7003-6488140ca596
|
||||
|
||||
model:
|
||||
dt_ms: 0.1
|
||||
membrane_tau_ms: 20.0
|
||||
synaptic_tau_ms: 5.0
|
||||
refractory_ms: 2.2
|
||||
resting_potential_mv: -52.0
|
||||
reset_potential_mv: -52.0
|
||||
threshold_mv: -45.0
|
||||
synaptic_delay_ms: 1.8
|
||||
w_syn_mv_per_anatomical_synapse: 0.275
|
||||
|
||||
sign_policy:
|
||||
neurotransmitter_sign:
|
||||
acetylcholine: 1
|
||||
gaba: -1
|
||||
glutamate: -1
|
||||
histamine: -1
|
||||
dopamine: 1
|
||||
octopamine: 1
|
||||
serotonin: 1
|
||||
unknown_sign: 0
|
||||
unclear_sign: 0
|
||||
|
||||
input_batch:
|
||||
version: 1.0.0
|
||||
allowed_modes: [spike_injection, external_synaptic_drive]
|
||||
refractory_respecting_spike_injection: true
|
||||
external_drive_application: add to g after decay and delayed-connectome arrival, before membrane integration
|
||||
|
||||
visual_adapter:
|
||||
version: 1.0.0
|
||||
modes:
|
||||
reference-photoreceptor:
|
||||
hierarchy_population: visual_sensory
|
||||
input_mode: spike_injection
|
||||
biological_fidelity: reference_comparison_only
|
||||
engineering-first-order:
|
||||
hierarchy_population: first_order
|
||||
input_mode: external_synaptic_drive
|
||||
biological_fidelity: engineering_adapter
|
||||
engineering-second-order:
|
||||
hierarchy_population: second_order
|
||||
input_mode: external_synaptic_drive
|
||||
biological_fidelity: engineering_adapter
|
||||
synthetic_rate_hz: 50.0
|
||||
external_drive_mv: 8.0
|
||||
|
||||
readout:
|
||||
bins_ms: [1, 5, 10, 20, 50]
|
||||
output_populations: [descending, motor_related, ascending]
|
||||
|
||||
window_experiment:
|
||||
windows_ms: [5, 10, 20, 30, 50, 100]
|
||||
sources: [engineering-first-order, engineering-second-order, auditory, olfactory]
|
||||
seeds: [44001, 44002]
|
||||
|
||||
continuous_control:
|
||||
biological_duration_ms: 1000.0
|
||||
periods_ms: [10, 20, 50, 100]
|
||||
sequence: [sparse-A, sparse-B, dense, zero, sparse-A]
|
||||
seed: 44100
|
||||
|
||||
synthetic_patterns:
|
||||
sparse_fraction: 0.05
|
||||
dense_fraction: 0.25
|
||||
moving_fraction: 0.05
|
||||
deterministic_seed: 44200
|
||||
|
||||
regression:
|
||||
float_absolute_tolerance: 0.00390625
|
||||
reference_trials:
|
||||
- {name: auditory-10hz, population: auditory, rate_hz: 10, seed: 42001, duration_ms: 1000.0}
|
||||
- {name: auditory-100hz, population: auditory, rate_hz: 100, seed: 42001, duration_ms: 1000.0}
|
||||
- {name: olfactory-50hz, population: olfactory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
- {name: mechanosensory-50hz, population: mechanosensory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
- {name: gustatory-50hz, population: gustatory, rate_hz: 50, seed: 42003, duration_ms: 1000.0}
|
||||
|
||||
prohibitions:
|
||||
stable_retro: false
|
||||
rom_access: false
|
||||
controller_buttons: false
|
||||
training: false
|
||||
web_server: false
|
||||
connectome_or_sign_changes: false
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"file_count": 3,
|
||||
"files": [
|
||||
{
|
||||
"bytes": 489,
|
||||
"path": "VERSION.json",
|
||||
"sha256": "7e413f5112dab08dd869b69a1c58537f6507b075275d7a6bb4255495f4c105ef"
|
||||
},
|
||||
{
|
||||
"bytes": 49126,
|
||||
"path": "phase4k.py",
|
||||
"sha256": "6ca7dd7b3fec7f970f8a43ac8286dd0f05b2674ac3996b9b20583b636719abe7"
|
||||
},
|
||||
{
|
||||
"bytes": 3690,
|
||||
"path": "phase4k.yaml",
|
||||
"sha256": "3721440cba03036552c358c1a9d3320637e5914da901f74e86189aa297192fd1"
|
||||
}
|
||||
],
|
||||
"generated_utc": "2026-09-20T20:07:05.654584+00:00",
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/kc-recruitment-audit-v1",
|
||||
"total_bytes": 53305
|
||||
}
|
||||
@@ -0,0 +1,13 @@
|
||||
{
|
||||
"diagnostic_only": true,
|
||||
"eligibility_v2_modified": false,
|
||||
"labels": {
|
||||
"KC_recruitment": "KC-RECRUITMENT-SOURCE-IDENTIFIED",
|
||||
"MBON_readout_insensitive": false,
|
||||
"odor_input": "ODOR-INPUT-NOT-GLOMERULUS-GROUNDED",
|
||||
"plasticity_transfer": "FAIL-PLASTICITY-TRANSFER",
|
||||
"whole_brain_recruitment_contamination": false
|
||||
},
|
||||
"name": "MaleCNS-KC-recruitment-MBON-causal-audit-v1",
|
||||
"summary_sha256": "0aab0f51ef73e5276d215d35833868e02008336e72166baf9f67496d6a90a825"
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
5fbae0b970ac26c506f27d06d01b2e716ce45b1cb023439372151ccc3016ac02 MANIFEST.json
|
||||
7e413f5112dab08dd869b69a1c58537f6507b075275d7a6bb4255495f4c105ef VERSION.json
|
||||
6ca7dd7b3fec7f970f8a43ac8286dd0f05b2674ac3996b9b20583b636719abe7 phase4k.py
|
||||
3721440cba03036552c358c1a9d3320637e5914da901f74e86189aa297192fd1 phase4k.yaml
|
||||
@@ -0,0 +1,609 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4K: read-only KC recruitment and MBON causal-input audit."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gc
|
||||
import json
|
||||
import math
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
from phase4h_audit import sha256, tree_manifest, verify_checksum, write_json
|
||||
from phase4i import Validator, phase4c_regression
|
||||
from phase4j import APLSimulator, verify_frozen
|
||||
|
||||
|
||||
KC_CATEGORIES = ("PN_olfactory", "KC", "APL", "MBON", "DAN", "other")
|
||||
MBON_CATEGORIES = ("plastic_KC", "nonplastic_KC", "DAN", "MBON", "feedback_neuron", "other")
|
||||
|
||||
|
||||
def seal(root: Path, manifest: Path, checksums: Path) -> None:
|
||||
write_json(manifest, tree_manifest(root, {manifest, checksums}))
|
||||
targets = sorted(x for x in root.rglob("*") if x.is_file() and x != checksums)
|
||||
checksums.write_text("".join(f"{sha256(x)} {x.relative_to(root)}\n" for x in targets), encoding="utf-8")
|
||||
|
||||
|
||||
def first_latency(first_steps: np.ndarray, members: np.ndarray, dt: float) -> float | None:
|
||||
values = first_steps[members]
|
||||
return None if np.all(values < 0) else float(values[values >= 0].min() * dt)
|
||||
|
||||
|
||||
def response(run: dict[str, Any], v: Validator) -> dict[str, Any]:
|
||||
return {
|
||||
"MBON_spikes": int(run["counts"][v.mbon].sum()),
|
||||
"MBON_first_latency_ms": first_latency(run["first_steps"], v.mbon, run["dt_ms"]),
|
||||
"downstream_spikes": int(run["counts"][v.downstream].sum()),
|
||||
"downstream_first_latency_ms": first_latency(run["first_steps"], v.downstream, run["dt_ms"]),
|
||||
"MBON_final_g": run["final_g_mbon"],
|
||||
"MBON_final_v": run["final_v_mbon"],
|
||||
}
|
||||
|
||||
|
||||
def compare(pre: dict[str, Any], post: dict[str, Any]) -> dict[str, Any]:
|
||||
def delta_latency(key: str) -> float | None:
|
||||
a, b = pre[key], post[key]
|
||||
return None if a is None or b is None else float(b - a)
|
||||
return {
|
||||
"MBON_delta_g_max_abs": float(np.max(np.abs(post["MBON_final_g"] - pre["MBON_final_g"]))),
|
||||
"MBON_delta_v_max_abs": float(np.max(np.abs(post["MBON_final_v"] - pre["MBON_final_v"]))),
|
||||
"MBON_spike_count_delta": int(post["MBON_spikes"] - pre["MBON_spikes"]),
|
||||
"MBON_spike_l1": int(abs(post["MBON_spikes"] - pre["MBON_spikes"])),
|
||||
"MBON_first_latency_delta_ms": delta_latency("MBON_first_latency_ms"),
|
||||
"downstream_spike_count_delta": int(post["downstream_spikes"] - pre["downstream_spikes"]),
|
||||
"downstream_first_latency_delta_ms": delta_latency("downstream_first_latency_ms"),
|
||||
}
|
||||
|
||||
|
||||
class AuditSimulator(APLSimulator):
|
||||
"""Frozen LIF runtime with side-channel drive and event-causality accounting."""
|
||||
|
||||
def __init__(self, v: Validator, cfg: dict[str, Any], exp: Path):
|
||||
super().__init__(v, cfg, exp)
|
||||
self.base_np = v.base.detach().cpu().numpy().copy()
|
||||
self.indices_np = v.rt.indices_np
|
||||
self.raw_np = v.rt.raw_np
|
||||
self.edge_sources = self.sources
|
||||
self.dan = np.unique(v.confirmed["source_neuron_idx"].to_numpy(np.int32))
|
||||
|
||||
pn_olf = (v.cls == "ALPN") | (v.cls == "olfactory")
|
||||
node_code = np.full(v.n, KC_CATEGORIES.index("other"), np.int8)
|
||||
node_code[pn_olf] = KC_CATEGORIES.index("PN_olfactory")
|
||||
node_code[v.kc] = KC_CATEGORIES.index("KC")
|
||||
node_code[self.apl] = KC_CATEGORIES.index("APL")
|
||||
node_code[v.mbon] = KC_CATEGORIES.index("MBON")
|
||||
node_code[self.dan] = KC_CATEGORIES.index("DAN")
|
||||
self.node_kc_code = node_code
|
||||
self.edge_kc_code_gpu = torch.as_tensor(node_code[self.edge_sources], dtype=torch.int8, device=self.device)
|
||||
|
||||
target_is_kc = np.zeros(v.n, bool); target_is_kc[v.kc] = True
|
||||
feedback = np.unique(self.edge_sources[target_is_kc[self.indices_np]])
|
||||
is_candidate = np.zeros(len(self.indices_np), bool); is_candidate[v.edge_ids] = True
|
||||
edge_mbon_code = np.full(len(self.indices_np), MBON_CATEGORIES.index("other"), np.int8)
|
||||
src_kc = v.cls[self.edge_sources] == "Kenyon_Cell"
|
||||
edge_mbon_code[src_kc & is_candidate] = MBON_CATEGORIES.index("plastic_KC")
|
||||
edge_mbon_code[src_kc & ~is_candidate] = MBON_CATEGORIES.index("nonplastic_KC")
|
||||
edge_mbon_code[np.isin(self.edge_sources, self.dan)] = MBON_CATEGORIES.index("DAN")
|
||||
edge_mbon_code[np.isin(self.edge_sources, v.mbon)] = MBON_CATEGORIES.index("MBON")
|
||||
structurally_feedback = np.isin(self.edge_sources, feedback) & ~src_kc & ~np.isin(self.edge_sources, np.concatenate([self.dan, v.mbon]))
|
||||
edge_mbon_code[structurally_feedback] = MBON_CATEGORIES.index("feedback_neuron")
|
||||
self.edge_mbon_code_gpu = torch.as_tensor(edge_mbon_code, dtype=torch.int8, device=self.device)
|
||||
|
||||
kc_lookup = np.full(v.n, -1, np.int32); kc_lookup[v.kc] = np.arange(len(v.kc), dtype=np.int32)
|
||||
mbon_lookup = np.full(v.n, -1, np.int32); mbon_lookup[v.mbon] = np.arange(len(v.mbon), dtype=np.int32)
|
||||
self.kc_lookup_gpu = torch.as_tensor(kc_lookup, dtype=torch.int32, device=self.device)
|
||||
self.mbon_lookup_gpu = torch.as_tensor(mbon_lookup, dtype=torch.int32, device=self.device)
|
||||
self.mbon_gpu = torch.as_tensor(v.mbon, dtype=torch.int64, device=self.device)
|
||||
|
||||
def population_label(self, neuron: int) -> str:
|
||||
return KC_CATEGORIES[int(self.node_kc_code[int(neuron)])]
|
||||
|
||||
@staticmethod
|
||||
def _dominant_arrivals(
|
||||
item: tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray] | None,
|
||||
score: np.ndarray, parent: np.ndarray, edge: np.ndarray, arrival: np.ndarray,
|
||||
) -> None:
|
||||
if item is None or not len(item[0]):
|
||||
return
|
||||
targets, values, parents, edges, arrivals = item
|
||||
order = np.lexsort((-values, targets))
|
||||
st = targets[order]
|
||||
take = np.r_[True, st[1:] != st[:-1]]
|
||||
chosen = order[take]; nodes = targets[chosen]; better = values[chosen] > score[nodes]
|
||||
nodes = nodes[better]; chosen = chosen[better]
|
||||
score[nodes] = values[chosen]; parent[nodes] = parents[chosen]
|
||||
edge[nodes] = edges[chosen]; arrival[nodes] = arrivals[chosen]
|
||||
|
||||
def run_audit(
|
||||
self, targets: np.ndarray, seed: int, mode: str, duration_ms: float,
|
||||
delta: np.ndarray | None = None, rate_hz: float | None = None, pulses: list[float] | None = None,
|
||||
dan_targets: np.ndarray | None = None, dan_pulses: list[float] | None = None,
|
||||
causal: bool = False, collect_kc_spikes: bool = False,
|
||||
) -> dict[str, Any]:
|
||||
if delta is None:
|
||||
delta = np.zeros(len(self.v.edge_ids), np.float32)
|
||||
self.v.install(delta)
|
||||
if mode in {"graded", "disabled"}:
|
||||
self.rt.weights[self.apl_edge_gpu] = 0.0
|
||||
self.rt.reset(int(seed)); steps = int(round(float(duration_ms) / self.dt))
|
||||
target_gpu = torch.as_tensor(targets, dtype=torch.int64, device=self.device)
|
||||
forced = self._forced(steps, targets, seed, rate_hz, pulses)
|
||||
dan = np.empty(0, np.int32) if dan_targets is None else np.asarray(dan_targets, np.int32)
|
||||
dan_gpu = torch.as_tensor(dan, dtype=torch.int64, device=self.device)
|
||||
dan_forced = self._forced(steps, dan, seed + 991, None, dan_pulses)
|
||||
|
||||
qlen = self.rt.queue_len; nk, nm = len(self.v.kc), len(self.v.mbon)
|
||||
kq = torch.zeros((qlen, len(KC_CATEGORIES), nk), dtype=torch.float32, device=self.device)
|
||||
mq = torch.zeros((qlen, len(MBON_CATEGORIES), nm), dtype=torch.float32, device=self.device)
|
||||
mpq = torch.zeros_like(mq); mdq = torch.zeros((qlen, nm), dtype=torch.float32, device=self.device)
|
||||
kg = torch.zeros((len(KC_CATEGORIES), nk), dtype=torch.float32, device=self.device)
|
||||
mg = torch.zeros((len(MBON_CATEGORIES), nm), dtype=torch.float32, device=self.device)
|
||||
mdg = torch.zeros(nm, dtype=torch.float32, device=self.device)
|
||||
kr = torch.zeros_like(kg); mr = torch.zeros_like(mg); mrp = torch.zeros_like(mg); mdr = torch.zeros(nm, dtype=torch.float32, device=self.device)
|
||||
kt = torch.zeros((steps, len(KC_CATEGORIES)), dtype=torch.float32, device=self.device)
|
||||
mt = torch.zeros((steps, len(MBON_CATEGORIES)), dtype=torch.float32, device=self.device)
|
||||
counts = torch.zeros(self.v.n, dtype=torch.int32, device=self.device)
|
||||
first = torch.full((self.v.n,), -1, dtype=torch.int32, device=self.device)
|
||||
kc_apl_first = torch.zeros(nk, dtype=torch.float32, device=self.device)
|
||||
|
||||
params = self.cfg["apl_overlay"]
|
||||
apl_state = torch.zeros(len(self.apl), dtype=torch.float32, device=self.device)
|
||||
apl_decay = math.exp(-self.dt / float(params["tau_ms"]))
|
||||
alpha, gd = self.rt.alpha, self.rt.g_decay
|
||||
rest, reset, threshold, refsteps = self.rt.rest, self.rt.reset_value, self.rt.threshold, self.rt.refractory_steps
|
||||
delta_by_edge = torch.zeros(len(self.indices_np), dtype=torch.float32, device=self.device)
|
||||
delta_by_edge[torch.as_tensor(self.v.edge_ids, dtype=torch.int64, device=self.device)] = torch.as_tensor(delta, dtype=torch.float32, device=self.device)
|
||||
|
||||
event_rows: list[dict[str, Any]] = []; event_parent: list[int] = []; event_depth: list[int] = []
|
||||
causal_score = np.zeros(self.v.n, np.float32); causal_parent = np.full(self.v.n, -1, np.int64)
|
||||
causal_edge = np.full(self.v.n, -1, np.int64); causal_arrival = np.full(self.v.n, -1, np.int32)
|
||||
causal_ring: list[tuple[np.ndarray, np.ndarray, np.ndarray, np.ndarray, np.ndarray] | None] = [None] * qlen
|
||||
current_event = np.full(self.v.n, -1, np.int64)
|
||||
kc_spike_steps: list[np.ndarray] = []; kc_spike_nodes: list[np.ndarray] = []
|
||||
kc_local_lookup = np.full(self.v.n, -1, np.int32); kc_local_lookup[self.v.kc] = np.arange(nk, dtype=np.int32)
|
||||
forced_set = set(map(int, targets)); dan_set = set(map(int, dan))
|
||||
|
||||
torch.cuda.synchronize(); t0 = time.perf_counter()
|
||||
for step in range(steps):
|
||||
slot = step % qlen
|
||||
self.rt.g.mul_(gd).add_(self.rt.delay_queue[slot]); self.rt.delay_queue[slot].zero_()
|
||||
ka = kq[slot]; ma = mq[slot]; mpa = mpq[slot]; mda = mdq[slot]
|
||||
kg.mul_(gd).add_(ka); mg.mul_(gd).add_(ma); mdg.mul_(gd).add_(mda)
|
||||
kr.add_(ka); mr.add_(ma); mrp.add_(mpa); mdr.add_(mda)
|
||||
kt[step] = ka.sum(dim=1); mt[step] = ma.sum(dim=1)
|
||||
ka.zero_(); ma.zero_(); mpa.zero_(); mda.zero_()
|
||||
if causal:
|
||||
causal_score *= float(gd)
|
||||
self._dominant_arrivals(causal_ring[slot], causal_score, causal_parent, causal_edge, causal_arrival)
|
||||
causal_ring[slot] = None
|
||||
|
||||
self.rt.refractory.clamp_min_(1).sub_(1); available = self.rt.refractory == 0
|
||||
self.rt.v[available] += alpha * (self.rt.g[available] - (self.rt.v[available] - rest)); self.rt.v[~available] = reset
|
||||
spikes = available & (self.rt.v >= threshold)
|
||||
accepted = forced[step] & available[target_gpu]
|
||||
if accepted.any():
|
||||
spikes[target_gpu[accepted]] = True
|
||||
dacc = torch.zeros(len(dan), dtype=torch.bool, device=self.device)
|
||||
if len(dan):
|
||||
dacc = dan_forced[step] & available[dan_gpu]
|
||||
if dacc.any():
|
||||
spikes[dan_gpu[dacc]] = True
|
||||
kc_new = spikes[self.kc_gpu] & (first[self.kc_gpu] < 0)
|
||||
if kc_new.any():
|
||||
kc_apl_first[kc_new] = kr[KC_CATEGORIES.index("APL"), kc_new]
|
||||
sources = torch.nonzero(spikes, as_tuple=False).flatten()
|
||||
counts.add_(spikes.to(torch.int32)); new = spikes & (first < 0); first[new] = step
|
||||
if collect_kc_spikes:
|
||||
kn = torch.nonzero(spikes[self.kc_gpu], as_tuple=False).flatten().cpu().numpy()
|
||||
if len(kn):
|
||||
kc_spike_steps.append(np.full(len(kn), step, np.int32)); kc_spike_nodes.append(self.v.kc[kn])
|
||||
|
||||
accepted_nodes = set(target_gpu[accepted].cpu().numpy().astype(int).tolist()) if bool(accepted.any()) else set()
|
||||
dan_nodes = set(dan_gpu[dacc].cpu().numpy().astype(int).tolist()) if len(dan) and bool(dacc.any()) else set()
|
||||
source_np = sources.cpu().numpy().astype(np.int32) if causal and sources.numel() else np.empty(0, np.int32)
|
||||
if causal and len(source_np):
|
||||
ids = np.arange(len(event_rows), len(event_rows) + len(source_np), dtype=np.int64)
|
||||
current_event[source_np] = ids
|
||||
for eid, node in zip(ids, source_np):
|
||||
root = int(node) in accepted_nodes and int(node) in forced_set
|
||||
par = -1 if root else int(causal_parent[node]); is_causal = root or par >= 0
|
||||
depth = 0 if root else (event_depth[par] + 1 if is_causal else -1)
|
||||
event_rows.append({"event_id": int(eid), "step": int(step), "timestamp_ms": float(step * self.dt), "neuron_idx": int(node),
|
||||
"forced_kind": "injected_A" if root else ("matched_DAN" if int(node) in dan_nodes and int(node) in dan_set else "none"),
|
||||
"causally_reachable_from_A": bool(is_causal), "parent_event_id": par,
|
||||
"parent_edge_id": -1 if root else int(causal_edge[node]), "parent_arrival_step": -1 if root else int(causal_arrival[node]),
|
||||
"dominant_decayed_contribution": None if root else float(causal_score[node]), "path_depth": int(depth)})
|
||||
event_parent.append(par); event_depth.append(depth)
|
||||
|
||||
if sources.numel():
|
||||
self.rt.v[sources] = reset; self.rt.refractory[sources] = refsteps
|
||||
edge_ids = self.rt._hybrid_edges(sources)
|
||||
if edge_ids.numel():
|
||||
next_slot = (step + self.rt.delay_steps) % qlen
|
||||
targets_all = self.rt.indices[edge_ids].long(); weights = self.rt.weights[edge_ids]
|
||||
self.rt.delay_queue[next_slot].scatter_add_(0, targets_all, weights)
|
||||
kloc = self.kc_lookup_gpu[targets_all]; kcode = self.edge_kc_code_gpu[edge_ids]
|
||||
for ci in range(len(KC_CATEGORIES)):
|
||||
mask = (kloc >= 0) & (kcode == ci)
|
||||
if mask.any():
|
||||
kq[next_slot, ci].scatter_add_(0, kloc[mask].long(), weights[mask])
|
||||
mloc = self.mbon_lookup_gpu[targets_all]; mcode = self.edge_mbon_code_gpu[edge_ids]
|
||||
for ci in range(len(MBON_CATEGORIES)):
|
||||
mask = (mloc >= 0) & (mcode == ci)
|
||||
if mask.any():
|
||||
mq[next_slot, ci].scatter_add_(0, mloc[mask].long(), weights[mask])
|
||||
mpq[next_slot, ci].scatter_add_(0, mloc[mask].long(), weights[mask].clamp_min(0))
|
||||
mm = mloc >= 0
|
||||
if mm.any():
|
||||
mdq[next_slot].scatter_add_(0, mloc[mm].long(), delta_by_edge[edge_ids[mm]])
|
||||
|
||||
if causal:
|
||||
eids = edge_ids.cpu().numpy().astype(np.int64); pre = self.edge_sources[eids]
|
||||
pe = current_event[pre]; w = self.base_np[eids]
|
||||
cm = (pe >= 0) & (w > 0)
|
||||
if np.any(cm):
|
||||
causal_ring[next_slot] = (self.indices_np[eids[cm]].astype(np.int32), w[cm].astype(np.float32), pe[cm].astype(np.int64),
|
||||
eids[cm], np.full(np.count_nonzero(cm), step + self.rt.delay_steps, np.int32))
|
||||
if causal and len(source_np):
|
||||
current_event[source_np] = -1
|
||||
|
||||
if mode == "graded":
|
||||
apl_state.mul_(apl_decay).add_(torch.mv(self.input_gpu, spikes.to(torch.float32)) * float(params["normalization"]))
|
||||
inhibition = torch.matmul(apl_state, self.output_gpu) * float(params["gain"])
|
||||
self.rt.g.index_add_(0, self.kc_gpu, -inhibition)
|
||||
kg[KC_CATEGORIES.index("APL")].sub_(inhibition); kr[KC_CATEGORIES.index("APL")].sub_(inhibition)
|
||||
kt[step, KC_CATEGORIES.index("APL")].sub_(inhibition.sum())
|
||||
|
||||
torch.cuda.synchronize(); wall = time.perf_counter() - t0
|
||||
counts_np = counts.cpu().numpy(); first_np = first.cpu().numpy()
|
||||
kshadow = kg.cpu().numpy(); mshadow = mg.cpu().numpy(); mdshadow = mdg.cpu().numpy()
|
||||
final_k = self.rt.g[self.kc_gpu].cpu().numpy(); final_m = self.rt.g[self.mbon_gpu].cpu().numpy()
|
||||
kc_spikes = pd.DataFrame({"step": np.concatenate(kc_spike_steps) if kc_spike_steps else np.empty(0, np.int32),
|
||||
"neuron_idx": np.concatenate(kc_spike_nodes) if kc_spike_nodes else np.empty(0, np.int32)})
|
||||
return {
|
||||
"counts": counts_np, "first_steps": first_np, "dt_ms": self.dt, "wall_seconds": wall,
|
||||
"final_g_kc": final_k, "final_g_mbon": final_m, "final_v_mbon": self.rt.v[self.mbon_gpu].cpu().numpy().copy(),
|
||||
"kc_drive_received": kr.cpu().numpy(), "kc_g_shadow": kshadow, "kc_apl_at_first": kc_apl_first.cpu().numpy(),
|
||||
"mbon_drive_received": mr.cpu().numpy(), "mbon_positive_received": mrp.cpu().numpy(),
|
||||
"mbon_delta_received": mdr.cpu().numpy(), "mbon_g_shadow": mshadow, "mbon_delta_g_shadow": mdshadow,
|
||||
"kc_timeseries": kt.cpu().numpy(), "mbon_timeseries": mt.cpu().numpy(), "events": pd.DataFrame(event_rows), "kc_spikes": kc_spikes,
|
||||
"shadow_validation": {"KC_final_g_max_abs_error": float(np.max(np.abs(kshadow.sum(axis=0) - final_k))),
|
||||
"MBON_final_g_max_abs_error": float(np.max(np.abs(mshadow.sum(axis=0) - final_m)))},
|
||||
}
|
||||
|
||||
|
||||
def save_drive(condition: str, run: dict[str, Any], v: Validator, kc_rows: list[dict[str, Any]], mbon_rows: list[dict[str, Any]], time_rows: list[dict[str, Any]]) -> dict[str, Any]:
|
||||
for ci, category in enumerate(KC_CATEGORIES):
|
||||
for local, neuron in enumerate(v.kc):
|
||||
kc_rows.append({"condition": condition, "category": category, "neuron_idx": int(neuron), "body_id": int(v.meta.loc[neuron, "body_id"]),
|
||||
"cumulative_signed_drive": float(run["kc_drive_received"][ci, local]), "final_g_component": float(run["kc_g_shadow"][ci, local])})
|
||||
for ci, category in enumerate(MBON_CATEGORIES):
|
||||
for local, neuron in enumerate(v.mbon):
|
||||
mbon_rows.append({"condition": condition, "category": category, "neuron_idx": int(neuron), "body_id": int(v.meta.loc[neuron, "body_id"]),
|
||||
"cumulative_signed_drive": float(run["mbon_drive_received"][ci, local]), "cumulative_positive_drive": float(run["mbon_positive_received"][ci, local]),
|
||||
"final_g_component": float(run["mbon_g_shadow"][ci, local]), "actual_delta_cumulative_drive": float(run["mbon_delta_received"][local]),
|
||||
"actual_delta_final_g": float(run["mbon_delta_g_shadow"][local]), "final_total_g": float(run["final_g_mbon"][local]),
|
||||
"final_v": float(run["final_v_mbon"][local]), "threshold_margin_mv": float(v.rt.threshold - run["final_v_mbon"][local])})
|
||||
for step in range(len(run["kc_timeseries"])):
|
||||
row = {"condition": condition, "step": step, "timestamp_ms": step * run["dt_ms"]}
|
||||
row.update({f"KC_{x}": float(run["kc_timeseries"][step, i]) for i, x in enumerate(KC_CATEGORIES)})
|
||||
row.update({f"MBON_{x}": float(run["mbon_timeseries"][step, i]) for i, x in enumerate(MBON_CATEGORIES)})
|
||||
time_rows.append(row)
|
||||
ksum = run["kc_drive_received"].sum(axis=1); msum = run["mbon_drive_received"].sum(axis=1); mpos = run["mbon_positive_received"].sum(axis=1)
|
||||
return {"KC_cumulative_signed_drive": {x: float(ksum[i]) for i, x in enumerate(KC_CATEGORIES)},
|
||||
"MBON_cumulative_signed_drive": {x: float(msum[i]) for i, x in enumerate(MBON_CATEGORIES)},
|
||||
"MBON_cumulative_positive_drive": {x: float(mpos[i]) for i, x in enumerate(MBON_CATEGORIES)},
|
||||
"wall_seconds": run["wall_seconds"], "shadow_validation": run["shadow_validation"]}
|
||||
|
||||
|
||||
def classify_path(labels: list[str], apl_modulation: bool, path_length: int) -> str:
|
||||
if path_length == 1:
|
||||
return "direct recurrent"
|
||||
middle = labels[1:-1]
|
||||
if any(x == "PN_olfactory" for x in middle):
|
||||
return "PN/olfactory feedback"
|
||||
if any(x in {"MBON", "DAN"} for x in middle):
|
||||
return "MBON/DAN feedback"
|
||||
if any(x == "APL" for x in middle) or apl_modulation:
|
||||
return "APL-related"
|
||||
return "other CNS loop" if path_length > 0 else "unexplained"
|
||||
|
||||
|
||||
def recruitment_paths(sim: AuditSimulator, run: dict[str, Any], set_a: np.ndarray, set_b: np.ndarray, actual_delta: np.ndarray, exp: Path) -> dict[str, Any]:
|
||||
events = run["events"].copy(); events.to_parquet(exp / "recruitment/direct-A-all-spike-events.parquet", index=False)
|
||||
first_event = events.sort_values(["step", "event_id"]).drop_duplicates("neuron_idx").set_index("neuron_idx")
|
||||
changed = actual_delta != 0; cand_src = sim.v.cand["source_neuron_idx"].to_numpy(np.int32)
|
||||
b_changed_edges = changed & np.isin(cand_src, set_b); b_changed_sources = np.unique(cand_src[b_changed_edges])
|
||||
event_map = {int(row["event_id"]): row for row in events.to_dict(orient="records")}
|
||||
recruited = np.setdiff1d(sim.v.kc[run["counts"][sim.v.kc] > 0], set_a)
|
||||
node_rows: list[dict[str, Any]] = []; edge_rows: list[dict[str, Any]] = []; summary_rows: list[dict[str, Any]] = []
|
||||
kc_local = {int(x): i for i, x in enumerate(sim.v.kc)}
|
||||
for target in recruited:
|
||||
if int(target) not in first_event.index:
|
||||
continue
|
||||
terminal = first_event.loc[int(target)]; eid = int(terminal["event_id"]); chain: list[dict[str, Any]] = []
|
||||
seen = set()
|
||||
while eid >= 0 and eid not in seen:
|
||||
seen.add(eid); ev = event_map[eid]; chain.append(ev); eid = int(ev["parent_event_id"])
|
||||
chain.reverse(); rooted = bool(chain and chain[0]["forced_kind"] == "injected_A")
|
||||
labels = [sim.population_label(int(x["neuron_idx"])) for x in chain]
|
||||
apl_mod = abs(float(run["kc_apl_at_first"][kc_local[int(target)]])) > 0
|
||||
path_length = len(chain) - 1 if rooted else 0
|
||||
category = classify_path(labels, apl_mod, path_length) if rooted else "unexplained"
|
||||
for order, ev in enumerate(chain):
|
||||
neuron = int(ev["neuron_idx"])
|
||||
node_rows.append({"target_recruited_kc": int(target), "path_order": order, "event_id": int(ev["event_id"]), "neuron_idx": neuron,
|
||||
"body_id": int(sim.v.meta.loc[neuron, "body_id"]), "population": sim.population_label(neuron),
|
||||
"annotation_type": sim.v.types[neuron], "annotation_class": sim.v.cls[neuron], "timestamp_ms": float(ev["timestamp_ms"]),
|
||||
"forced_kind": ev["forced_kind"]})
|
||||
for order, ev in enumerate(chain[1:], start=1):
|
||||
edge_id = int(ev["parent_edge_id"]); parent = chain[order - 1]
|
||||
edge_rows.append({"target_recruited_kc": int(target), "path_order": order, "edge_id": edge_id,
|
||||
"source_neuron_idx": int(parent["neuron_idx"]), "target_neuron_idx": int(ev["neuron_idx"]),
|
||||
"source_spike_time_ms": float(parent["timestamp_ms"]), "arrival_time_ms": float(ev["parent_arrival_step"] * run["dt_ms"]),
|
||||
"target_spike_time_ms": float(ev["timestamp_ms"]), "effective_weight": float(sim.base_np[edge_id]),
|
||||
"raw_synapse_count": int(sim.raw_np[edge_id]), "synaptic_sign": int(np.sign(sim.base_np[edge_id])),
|
||||
"source_population": sim.population_label(int(parent["neuron_idx"])), "target_population": sim.population_label(int(ev["neuron_idx"]))})
|
||||
summary_rows.append({"neuron_idx": int(target), "body_id": int(sim.v.meta.loc[target, "body_id"]), "annotation_type": sim.v.types[target],
|
||||
"first_activation_ms": float(terminal["timestamp_ms"]), "causal_path_found": rooted, "path_length": path_length,
|
||||
"recruitment_class": category, "APL_involved_or_modulatory": apl_mod or ("APL" in labels),
|
||||
"is_injected_B": bool(target in set_b), "is_B_source_with_changed_edge": bool(target in b_changed_sources),
|
||||
"changed_B_edge_count": int(np.count_nonzero(b_changed_edges & (cand_src == target)))})
|
||||
summary_df = pd.DataFrame(summary_rows); summary_df.to_parquet(exp / "recruitment/recruited-kc-summary.parquet", index=False)
|
||||
pd.DataFrame(node_rows).to_parquet(exp / "recruitment/causal-path-nodes.parquet", index=False)
|
||||
pd.DataFrame(edge_rows).to_parquet(exp / "recruitment/causal-path-edges.parquet", index=False)
|
||||
counts = summary_df["recruitment_class"].value_counts().to_dict() if len(summary_df) else {}
|
||||
b_rows = summary_df[summary_df["is_B_source_with_changed_edge"]] if len(summary_df) else summary_df
|
||||
report = {"label": "KC-RECRUITMENT-SOURCE-IDENTIFIED" if len(summary_df) and bool(summary_df["causal_path_found"].all()) else "FAIL-KC-RECRUITMENT-SOURCE-IDENTIFIED",
|
||||
"injected_A_KC": int(len(set_a)), "active_non_A_KC": int(len(recruited)), "traced_recruited_KC": int(len(summary_df)),
|
||||
"causal_paths_found": int(summary_df["causal_path_found"].sum()) if len(summary_df) else 0,
|
||||
"classification": {str(k): int(v) for k, v in counts.items()},
|
||||
"B_set_KC_active": int(np.count_nonzero(run["counts"][set_b])), "B_source_changed_edges": int(np.count_nonzero(b_changed_edges)),
|
||||
"unique_B_source_KC_with_changed_edges": int(len(b_changed_sources)),
|
||||
"B_source_path_classes": {str(k): int(v) for k, v in b_rows["recruitment_class"].value_counts().to_dict().items()} if len(b_rows) else {},
|
||||
"causal_rule": sim.cfg["causal_attribution"]["rule"], "static_shortest_path_used": False,
|
||||
"interpretation_limit": "dominant positive arrival under exact runtime timing is an attribution path, not an intervention-based proof"}
|
||||
write_json(exp / "recruitment/summary.json", report); return report
|
||||
|
||||
|
||||
def drive_audit(sim: AuditSimulator, set_a: np.ndarray, set_b: np.ndarray, olf_a: np.ndarray, cfg: dict[str, Any], exp: Path) -> tuple[dict[str, Any], dict[str, dict[str, Any]]]:
|
||||
dc, oc = cfg["direct_kc"], cfg["olfactory"]; params = cfg["apl_overlay"]
|
||||
specs = {
|
||||
"original_olfactory_CS": dict(targets=olf_a, seed=oc["seed"], mode="original", duration_ms=oc["duration_ms"], rate_hz=oc["rate_hz"]),
|
||||
"APL_overlay_olfactory_CS": dict(targets=olf_a, seed=oc["seed"], mode="graded", duration_ms=oc["duration_ms"], rate_hz=oc["rate_hz"]),
|
||||
"direct_KC_SET_A": dict(targets=set_a, seed=dc["evaluation_seed"], mode="original", duration_ms=dc["duration_ms"], pulses=dc["pulse_times_ms"], collect_kc_spikes=True),
|
||||
"direct_KC_SET_B": dict(targets=set_b, seed=dc["evaluation_seed"], mode="original", duration_ms=dc["duration_ms"], pulses=dc["pulse_times_ms"], collect_kc_spikes=True),
|
||||
}
|
||||
kr: list[dict[str, Any]] = []; mr: list[dict[str, Any]] = []; tr: list[dict[str, Any]] = []; runs = {}; summary = {}
|
||||
for name, spec in specs.items():
|
||||
run = sim.run_audit(delta=np.zeros(len(sim.v.edge_ids), np.float32), **spec); runs[name] = run
|
||||
summary[name] = save_drive(name, run, sim.v, kr, mr, tr)
|
||||
pd.DataFrame(kr).to_parquet(exp / "drive-decomposition/kc-incoming-drive.parquet", index=False)
|
||||
pd.DataFrame(mr).to_parquet(exp / "drive-decomposition/mbon-incoming-drive-baseline.parquet", index=False)
|
||||
pd.DataFrame(tr).to_parquet(exp / "drive-decomposition/drive-timeseries.parquet", index=False)
|
||||
write_json(exp / "drive-decomposition/summary.json", summary); return summary, runs
|
||||
|
||||
|
||||
def mbon_audit(sim: AuditSimulator, base_runs: dict[str, dict[str, Any]], set_a: np.ndarray, set_b: np.ndarray, actual_delta: np.ndarray, cfg: dict[str, Any], exp: Path) -> tuple[dict[str, Any], dict[str, dict[str, Any]]]:
|
||||
dc = cfg["direct_kc"]; posts = {}
|
||||
for label, targets in (("A", set_a), ("B", set_b)):
|
||||
posts[label] = sim.run_audit(targets, dc["evaluation_seed"], "original", dc["duration_ms"], delta=actual_delta, pulses=dc["pulse_times_ms"], collect_kc_spikes=True)
|
||||
rows = []; report = {}
|
||||
for label, pre in (("A", base_runs["direct_KC_SET_A"]), ("B", base_runs["direct_KC_SET_B"])):
|
||||
post = posts[label]; p = response(pre, sim.v); q = response(post, sim.v); eff = compare(p, q)
|
||||
plastic_i = MBON_CATEGORIES.index("plastic_KC")
|
||||
plastic_positive = float(post["mbon_positive_received"][plastic_i].sum()); total_positive = float(post["mbon_positive_received"].sum())
|
||||
total_signed = float(post["mbon_drive_received"].sum()); delta_signed = float(post["mbon_delta_received"].sum())
|
||||
report[label] = {"effect": eff, "total_cumulative_signed_drive": total_signed, "total_excitatory_drive": total_positive,
|
||||
"plastic_KC_excitatory_drive": plastic_positive, "plastic_KC_fraction_of_total_excitatory_drive": plastic_positive / total_positive if total_positive else None,
|
||||
"actual_delta_cumulative_drive": delta_signed, "actual_delta_over_total_excitatory_drive": delta_signed / total_positive if total_positive else None,
|
||||
"actual_delta_over_total_signed_drive": delta_signed / total_signed if total_signed else None,
|
||||
"final_total_g_sum": float(post["final_g_mbon"].sum()), "actual_delta_final_g_sum": float(post["mbon_delta_g_shadow"].sum())}
|
||||
for state, run in (("pre", pre), ("post", post)):
|
||||
for ci, category in enumerate(MBON_CATEGORIES):
|
||||
for local, neuron in enumerate(sim.v.mbon):
|
||||
rows.append({"stimulus": label, "state": state, "category": category, "neuron_idx": int(neuron), "body_id": int(sim.v.meta.loc[neuron, "body_id"]),
|
||||
"cumulative_signed_drive": float(run["mbon_drive_received"][ci, local]), "cumulative_positive_drive": float(run["mbon_positive_received"][ci, local]),
|
||||
"final_g_component": float(run["mbon_g_shadow"][ci, local]), "actual_delta_final_g": float(run["mbon_delta_g_shadow"][local]),
|
||||
"final_total_g": float(run["final_g_mbon"][local]), "final_v": float(run["final_v_mbon"][local]),
|
||||
"threshold_margin_mv": float(sim.v.rt.threshold - run["final_v_mbon"][local])})
|
||||
pd.DataFrame(rows).to_parquet(exp / "mbon-decomposition/pre-post-mbon-drive.parquet", index=False)
|
||||
write_json(exp / "mbon-decomposition/summary.json", report); return report, posts
|
||||
|
||||
|
||||
def sensitivity(sim: AuditSimulator, set_a: np.ndarray, actual_delta: np.ndarray, cfg: dict[str, Any], exp: Path) -> dict[str, Any]:
|
||||
dc, sc = cfg["direct_kc"], cfg["sensitivity"]; changed = actual_delta != 0; rows = []
|
||||
for seed in sc["evaluation_seeds"]:
|
||||
base_delta = np.zeros(len(sim.v.edge_ids), np.float32)
|
||||
base = sim.run_audit(set_a, int(seed), "original", dc["duration_ms"], delta=base_delta, pulses=dc["pulse_times_ms"])
|
||||
bp = response(base, sim.v)
|
||||
for fraction in sc["depression_fractions"]:
|
||||
delta = np.zeros(len(sim.v.edge_ids), np.float32); delta[changed] = -float(fraction) * sim.v.base_candidate[changed]
|
||||
run = base if float(fraction) == 0 else sim.run_audit(set_a, int(seed), "original", dc["duration_ms"], delta=delta, pulses=dc["pulse_times_ms"])
|
||||
q = response(run, sim.v); eff = compare(bp, q)
|
||||
functional = bool(eff["MBON_spike_l1"] > 0 or (eff["MBON_first_latency_delta_ms"] not in (None, 0.0)))
|
||||
rows.append({"seed": int(seed), "depression_fraction": float(fraction), "changed_edges": int(np.count_nonzero(changed)),
|
||||
**eff, "functional_MBON_output_change": functional})
|
||||
frame = pd.DataFrame(rows); frame.to_parquet(exp / "sensitivity/controlled-depression-curve.parquet", index=False)
|
||||
aggregate = []
|
||||
for frac, group in frame.groupby("depression_fraction"):
|
||||
aggregate.append({"depression_fraction": float(frac), "seeds": int(len(group)), "functional_seed_count": int(group.functional_MBON_output_change.sum()),
|
||||
"MBON_delta_g_max_abs_mean": float(group.MBON_delta_g_max_abs.mean()), "MBON_delta_v_max_abs_mean": float(group.MBON_delta_v_max_abs.mean()),
|
||||
"MBON_spike_l1_mean": float(group.MBON_spike_l1.mean()), "downstream_spike_abs_delta_mean": float(group.downstream_spike_count_delta.abs().mean())})
|
||||
functional_levels = [x["depression_fraction"] for x in aggregate if x["functional_seed_count"] > 0]
|
||||
at50 = next(x for x in aggregate if x["depression_fraction"] == 0.5)
|
||||
insensitive = at50["functional_seed_count"] == 0
|
||||
report = {"changed_real_edges": int(np.count_nonzero(changed)), "curve": aggregate,
|
||||
"first_fraction_with_any_functional_MBON_change": min(functional_levels) if functional_levels else None,
|
||||
"MBON_READOUT_INSENSITIVE": bool(insensitive), "label": "MBON-READOUT-INSENSITIVE" if insensitive else "MBON-READOUT-SENSITIVE-WITHIN-TESTED-RANGE",
|
||||
"diagnostic_only_not_saved_as_learned_model": True, "criterion": sc["functional_change_definition"]}
|
||||
write_json(exp / "sensitivity/summary.json", report); return report
|
||||
|
||||
|
||||
def isolated_replay_one(sim: AuditSimulator, spikes: pd.DataFrame, targets: np.ndarray, delta: np.ndarray, duration_ms: float) -> dict[str, Any]:
|
||||
target_lookup = {int(x): i for i, x in enumerate(targets)}; nt = len(targets); steps = int(round(duration_ms / sim.dt)); qlen = sim.rt.queue_len
|
||||
relevant = np.isin(sim.edge_sources, sim.v.kc) & np.isin(sim.indices_np, targets)
|
||||
edge_ids = np.flatnonzero(relevant); by_source: dict[int, tuple[np.ndarray, np.ndarray]] = {}
|
||||
full_delta = np.zeros(len(sim.indices_np), np.float32); full_delta[sim.v.edge_ids] = delta
|
||||
weights = sim.base_np[edge_ids] + full_delta[edge_ids]
|
||||
for source in np.unique(sim.edge_sources[edge_ids]):
|
||||
mask = sim.edge_sources[edge_ids] == source; ids = edge_ids[mask]
|
||||
by_source[int(source)] = (np.asarray([target_lookup[int(x)] for x in sim.indices_np[ids]], np.int32), weights[mask].astype(np.float32))
|
||||
grouped = {int(k): g.neuron_idx.to_numpy(np.int32) for k, g in spikes.groupby("step")}
|
||||
q = np.zeros((qlen, nt), np.float32); g = np.zeros(nt, np.float32); v = np.full(nt, sim.rt.rest, np.float32); ref = np.zeros(nt, np.int32)
|
||||
counts = np.zeros(nt, np.int32); first = np.full(nt, -1, np.int32)
|
||||
for step in range(steps):
|
||||
slot = step % qlen; g = g * np.float32(sim.rt.g_decay) + q[slot]; q[slot].fill(0)
|
||||
ref = np.maximum(ref, 1) - 1; available = ref == 0
|
||||
v[available] += np.float32(sim.rt.alpha) * (g[available] - (v[available] - np.float32(sim.rt.rest))); v[~available] = np.float32(sim.rt.reset_value)
|
||||
fired = available & (v >= np.float32(sim.rt.threshold)); counts += fired; first[(first < 0) & fired] = step; v[fired] = np.float32(sim.rt.reset_value); ref[fired] = sim.rt.refractory_steps
|
||||
for source in grouped.get(step, np.empty(0, np.int32)):
|
||||
if int(source) in by_source:
|
||||
loc, w = by_source[int(source)]; np.add.at(q[(step + sim.rt.delay_steps) % qlen], loc, w)
|
||||
return {"counts": counts, "first": first, "g": g, "v": v, "targets": targets}
|
||||
|
||||
|
||||
def isolated_replay(sim: AuditSimulator, base_runs: dict[str, dict[str, Any]], actual_delta: np.ndarray, cfg: dict[str, Any], whole_direct_status: str, exp: Path) -> dict[str, Any]:
|
||||
changed = actual_delta != 0; targets = np.unique(sim.v.cand.loc[changed, "target_neuron_idx"].to_numpy(np.int32)); rows = []; effects = {}
|
||||
for label, key in (("A", "direct_KC_SET_A"), ("B", "direct_KC_SET_B")):
|
||||
spikes = base_runs[key]["kc_spikes"]
|
||||
pre = isolated_replay_one(sim, spikes, targets, np.zeros(len(actual_delta), np.float32), cfg["direct_kc"]["duration_ms"])
|
||||
post = isolated_replay_one(sim, spikes, targets, actual_delta, cfg["direct_kc"]["duration_ms"])
|
||||
def lat(x: dict[str, Any]) -> float | None:
|
||||
return None if np.all(x["first"] < 0) else float(x["first"][x["first"] >= 0].min() * sim.dt)
|
||||
effect = {"delta_g_max_abs": float(np.max(np.abs(post["g"] - pre["g"]))), "delta_v_max_abs": float(np.max(np.abs(post["v"] - pre["v"]))),
|
||||
"spike_l1": int(np.abs(post["counts"] - pre["counts"]).sum()), "pre_first_latency_ms": lat(pre), "post_first_latency_ms": lat(post),
|
||||
"latency_delta_ms": None if lat(pre) is None or lat(post) is None else float(lat(post) - lat(pre))}
|
||||
effects[label] = effect
|
||||
for state, x in (("pre", pre), ("post", post)):
|
||||
for local, neuron in enumerate(targets):
|
||||
rows.append({"stimulus": label, "state": state, "neuron_idx": int(neuron), "body_id": int(sim.v.meta.loc[neuron, "body_id"]),
|
||||
"spikes": int(x["counts"][local]), "first_latency_ms": None if x["first"][local] < 0 else float(x["first"][local] * sim.dt),
|
||||
"final_g": float(x["g"][local]), "final_v": float(x["v"][local])})
|
||||
pd.DataFrame(rows).to_parquet(exp / "isolated-replay/real-edge-replay.parquet", index=False)
|
||||
af = effects["A"]["spike_l1"] > 0 or effects["A"]["latency_delta_ms"] not in (None, 0.0)
|
||||
bf = effects["B"]["spike_l1"] > 0 or effects["B"]["latency_delta_ms"] not in (None, 0.0)
|
||||
transfer = bool(af and not bf); contamination = bool(transfer and whole_direct_status != "PASS-DIRECT-KC-MEMORY-v2")
|
||||
report = {"target_MBON": int(len(targets)), "actual_changed_edges": int(np.count_nonzero(changed)), "effects": effects,
|
||||
"PLASTICITY_TRANSFER_PASS": transfer, "plasticity_label": "PLASTICITY-TRANSFER-PASS" if transfer else "FAIL-PLASTICITY-TRANSFER",
|
||||
"WHOLE_BRAIN_RECRUITMENT_CONTAMINATION": contamination,
|
||||
"weak_effect_label": None if af else "KC-MBON-PLASTICITY-EFFECT-TOO-WEAK",
|
||||
"spike_train_source": "deterministically regenerated from frozen Phase 4J protocol because exact Phase 4J event trains were not persisted",
|
||||
"new_edges_added": False}
|
||||
write_json(exp / "isolated-replay/summary.json", report); return report
|
||||
|
||||
|
||||
def olfactory_provenance(sim: AuditSimulator, cfg: dict[str, Any], exp: Path) -> dict[str, Any]:
|
||||
meta = sim.v.meta; olf = np.flatnonzero(sim.v.rt.population_masks["olfactory"]).astype(np.int32); olf.sort(); take = int(cfg["olfactory"]["targets_per_cs"])
|
||||
current = np.concatenate([olf[:take], olf[take:2 * take]])
|
||||
explicit = [x for x in meta.columns if any(term in x.lower() for term in ("glomer", "receptor", "sensillum", "antenn"))]
|
||||
fields = [x for x in ("neuron_idx", "body_id", "annotation_superclass", "annotation_class", "annotation_subclass", "annotation_type", "annotation_instance", "side", "neurotransmitter", *explicit) if x in meta.columns]
|
||||
fields = list(dict.fromkeys(fields))
|
||||
table = meta.loc[current, fields].copy(); table.insert(0, "CS", np.repeat(["A", "B"], take)); table.to_parquet(exp / "olfactory-provenance/current-CS-neurons.parquet", index=False)
|
||||
pn = np.flatnonzero(sim.v.cls == "ALPN").astype(np.int32); is_pn_kc = np.isin(sim.edge_sources, pn) & np.isin(sim.indices_np, sim.v.kc); eids = np.flatnonzero(is_pn_kc)
|
||||
edges = pd.DataFrame({"edge_id": eids, "source_neuron_idx": sim.edge_sources[eids], "target_neuron_idx": sim.indices_np[eids], "raw_synapse_count": sim.raw_np[eids]})
|
||||
edges["source_body_id"] = meta.loc[edges.source_neuron_idx, "body_id"].to_numpy(); edges["target_body_id"] = meta.loc[edges.target_neuron_idx, "body_id"].to_numpy()
|
||||
edges.to_parquet(exp / "olfactory-provenance/ALPN-to-KC-real-edges.parquet", index=False)
|
||||
identity_coverage = {x: int((table[x].notna() & ~table[x].fillna("").astype(str).eq("")).sum()) for x in explicit}
|
||||
# Phase 4J selected two contiguous neuron_idx slices from the broad olfactory
|
||||
# mask; it did not select or match by any receptor/glomerulus/PN identity.
|
||||
grounded = False
|
||||
report = {"olfactory_population_definition": "annotation_class == olfactory", "olfactory_population_count": int(len(olf)),
|
||||
"current_CS_definition": "first 64 and next 64 neuron_idx after sorting the broad olfactory population", "current_CS_neurons": int(len(current)),
|
||||
"explicit_glomerulus_or_receptor_columns": explicit, "explicit_identity_nonempty_counts_in_current_CS": identity_coverage,
|
||||
"current_CS_selection_used_explicit_identity": False, "current_CS_glomerulus_grounded": grounded,
|
||||
"label": "GLOMERULUS-GROUNDED" if grounded else "ODOR-INPUT-NOT-GLOMERULUS-GROUNDED",
|
||||
"ALPN_count": int(len(pn)), "ALPN_to_KC_edges": int(len(eids)), "ALPN_to_KC_synapse_weight": int(sim.raw_np[eids].sum()),
|
||||
"DoOR_imported": False, "reason_DoOR_not_imported": "Phase 4K mapping audit found no explicit MaleCNS glomerulus/receptor key for a non-guessing join",
|
||||
"prepared_next_input_scheme": "odor response vector -> explicit receptor/glomerulus mapping -> matched MaleCNS PN identities -> real ALPN-to-KC edges",
|
||||
"unmatched_channels": "all until an explicit official receptor/glomerulus-to-MaleCNS mapping is supplied"}
|
||||
write_json(exp / "olfactory-provenance/summary.json", report); return report
|
||||
|
||||
|
||||
def dataset_checks(cfg: dict[str, Any]) -> dict[str, Any]:
|
||||
root = Path(cfg["paths"]["dataset_root"])
|
||||
commands = {
|
||||
"raw_core_sha256": f"cd {root}/raw && sha256sum -c _manifests/checksums.sha256 >/dev/null",
|
||||
"raw_skeleton_md5": f"cd {root}/raw/skeletons && md5sum -c ../_manifests/skeletons/checksums.md5 >/dev/null",
|
||||
"phase3_sha256": f"cd {root}/processed && sha256sum -c audit/checksums.sha256 >/dev/null",
|
||||
"phase3b_audit_sha256": f"cd {root}/processed/neuron-only/audit && sha256sum -c checksums.sha256 >/dev/null",
|
||||
"phase3b_166700_sha256": f"cd {root}/processed/neuron-only && sha256sum -c audit/phase3b-166700/checksums.sha256 >/dev/null",
|
||||
"phase1_phase2_backup_sha256": f"sha256sum -c {root}/backups/male-cns-v1.0-phase1-phase2-20260919-1518.tar.zst.sha256 >/dev/null",
|
||||
}
|
||||
results = {}
|
||||
for name, command in commands.items():
|
||||
proc = subprocess.run(["bash", "-lc", command], capture_output=True, text=True)
|
||||
results[name] = {"passed": proc.returncode == 0, "returncode": proc.returncode, "stderr": proc.stderr[-1000:]}
|
||||
results["all_passed"] = all(x["passed"] for x in results.values())
|
||||
return results
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", type=Path, required=True); args = ap.parse_args()
|
||||
cfg = yaml.safe_load(args.config.read_text()); exp = Path(cfg["paths"]["experiment_root"]); model = Path(cfg["paths"]["phase4k_model_root"])
|
||||
for sub in ("recruitment", "drive-decomposition", "mbon-decomposition", "sensitivity", "isolated-replay", "olfactory-provenance", "regression", "audit", "manifests", "compiler-reports/extension-build"):
|
||||
(exp / sub).mkdir(parents=True, exist_ok=True)
|
||||
model.mkdir(parents=True, exist_ok=True)
|
||||
before = verify_frozen(cfg); p4j_checks = [verify_checksum(Path(cfg["paths"]["phase4j_root"]), Path(cfg["paths"]["phase4j_root"]) / "manifests/checksums.sha256"),
|
||||
verify_checksum(Path(cfg["paths"]["phase4j_model_root"]), Path(cfg["paths"]["phase4j_model_root"]) / "checksums.sha256")]
|
||||
if not all(x["passed"] for x in p4j_checks): raise RuntimeError("Phase 4J frozen checksum failure")
|
||||
shutil.copy2(args.config, exp / "audit/phase4k.yaml"); shutil.copy2(Path(__file__).resolve(), exp / "audit/phase4k.py")
|
||||
root = Path(cfg["paths"]["connectome_root"]); base_files = [root / "gpu/csr-indptr.int64.npy", root / "gpu/csr-indices.int32.npy", root / "gpu/csr-weights.int64.npy", root / "neuron-index/unified-metadata.parquet"]
|
||||
base_before = {str(x): sha256(x) for x in base_files}
|
||||
v = Validator(cfg, args.config.resolve(), exp); sim = AuditSimulator(v, cfg, exp)
|
||||
sets = pd.read_parquet(Path(cfg["paths"]["phase4j_root"]) / "direct-kc-memory/direct-kc-ensembles.parquet")
|
||||
set_a = np.sort(sets.loc[sets["set"] == "KC_SET_A", "neuron_idx"].to_numpy(np.int32)); set_b = np.sort(sets.loc[sets["set"] == "KC_SET_B", "neuron_idx"].to_numpy(np.int32))
|
||||
overlay = np.load(Path(cfg["paths"]["phase4j_root"]) / f"direct-kc-memory/seed-{cfg['direct_kc']['trace_seed']}/overlay.npz")
|
||||
if not np.array_equal(overlay["edge_id"], v.edge_ids): raise RuntimeError("Phase 4J overlay edge identity mismatch")
|
||||
actual_delta = overlay["plastic_delta_mv"].astype(np.float32)
|
||||
olf = np.flatnonzero(v.rt.population_masks["olfactory"]).astype(np.int32); olf.sort(); olf_a = olf[:cfg["olfactory"]["targets_per_cs"]]
|
||||
|
||||
drive_summary, base_runs = drive_audit(sim, set_a, set_b, olf_a, cfg, exp)
|
||||
dan = v.dan_targets(cfg["direct_kc"]["matched_dan_type"], cfg["direct_kc"]["target_compartment_verbatim"])
|
||||
causal_run = sim.run_audit(set_a, cfg["direct_kc"]["trace_seed"], "original", cfg["direct_kc"]["duration_ms"],
|
||||
delta=np.zeros(len(v.edge_ids), np.float32), pulses=cfg["direct_kc"]["pulse_times_ms"], dan_targets=dan,
|
||||
dan_pulses=[cfg["direct_kc"]["dan_time_ms"]], causal=True, collect_kc_spikes=True)
|
||||
recruitment = recruitment_paths(sim, causal_run, set_a, set_b, actual_delta, exp)
|
||||
mbon, post_runs = mbon_audit(sim, base_runs, set_a, set_b, actual_delta, cfg, exp)
|
||||
curve = sensitivity(sim, set_a, actual_delta, cfg, exp)
|
||||
p4j_summary = json.loads((Path(cfg["paths"]["phase4j_root"]) / "direct-kc-memory/summary.json").read_text())
|
||||
replay = isolated_replay(sim, base_runs, actual_delta, cfg, p4j_summary["status"], exp)
|
||||
odor = olfactory_provenance(sim, cfg, exp)
|
||||
|
||||
labels = {"KC_recruitment": recruitment["label"], "plasticity_transfer": replay["plasticity_label"],
|
||||
"whole_brain_recruitment_contamination": replay["WHOLE_BRAIN_RECRUITMENT_CONTAMINATION"],
|
||||
"MBON_readout_insensitive": curve["MBON_READOUT_INSENSITIVE"], "odor_input": odor["label"]}
|
||||
del sim, v; gc.collect(); torch.cuda.empty_cache()
|
||||
reg = phase4c_regression(cfg, args.config.resolve(), exp)
|
||||
after = verify_frozen(cfg); p4j_after = [verify_checksum(Path(cfg["paths"]["phase4j_root"]), Path(cfg["paths"]["phase4j_root"]) / "manifests/checksums.sha256"),
|
||||
verify_checksum(Path(cfg["paths"]["phase4j_model_root"]), Path(cfg["paths"]["phase4j_model_root"]) / "checksums.sha256")]
|
||||
dataset = dataset_checks(cfg); base_after = {str(x): sha256(x) for x in base_files}
|
||||
integrity = before["all_passed"] and after["all_passed"] and all(x["passed"] for x in p4j_checks + p4j_after) and dataset["all_passed"] and base_before == base_after and reg["all_passed"]
|
||||
integrity_report = {"all_passed": integrity, "phase4c_to_4i_before": before["all_passed"], "phase4c_to_4i_after": after["all_passed"],
|
||||
"phase4j_before": all(x["passed"] for x in p4j_checks), "phase4j_after": all(x["passed"] for x in p4j_after),
|
||||
"dataset_phase1_to_3b": dataset, "base_CSR_unchanged": base_before == base_after, "phase4c_reference_regression": reg["all_passed"]}
|
||||
write_json(exp / "audit/integrity.json", integrity_report); write_json(exp / "audit/base-sha256.json", {"before": base_before, "after": base_after})
|
||||
summary = {"phase": "4K", "labels": labels, "recruitment": recruitment, "drive_decomposition": drive_summary, "MBON_decomposition": mbon,
|
||||
"sensitivity": curve, "isolated_replay": replay, "olfactory_provenance": odor, "integrity": integrity_report,
|
||||
"recommendation": "repair upstream odor/glomerulus-grounded encoding and KC recruitment before changing MBON plasticity/readout" if odor["label"] == "ODOR-INPUT-NOT-GLOMERULUS-GROUNDED" else "use causal audit to choose upstream recruitment versus readout work",
|
||||
"eligibility_v2_modified": False, "base_modified": False, "plastic_mask_expanded": False, "stopped_before_phase4l": True}
|
||||
write_json(exp / "audit/final-summary.json", summary)
|
||||
shutil.copy2(args.config, model / "phase4k.yaml"); shutil.copy2(Path(__file__).resolve(), model / "phase4k.py")
|
||||
write_json(model / "VERSION.json", {"name": cfg["name"], "diagnostic_only": True, "labels": labels, "eligibility_v2_modified": False,
|
||||
"summary_sha256": sha256(exp / "audit/final-summary.json")})
|
||||
print(json.dumps({"labels": labels, "integrity": integrity}, indent=2)); sys.stdout.flush()
|
||||
seal(model, model / "MANIFEST.json", model / "checksums.sha256"); seal(exp, exp / "manifests/manifest.json", exp / "manifests/checksums.sha256")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,101 @@
|
||||
phase: 4K
|
||||
name: MaleCNS-KC-recruitment-MBON-causal-audit-v1
|
||||
|
||||
paths:
|
||||
connectome_root: /data/digital-drosophila/datasets/male-cns-v1.0/processed/neuron-only
|
||||
dataset_root: /data/digital-drosophila/datasets/male-cns-v1.0
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif
|
||||
phase4k_model_root: /data/digital-drosophila/models/male-cns-lif/kc-recruitment-audit-v1
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4k-kc-recruitment-audit
|
||||
reference_root: /data/digital-drosophila/models/male-cns-lif/reference-v1
|
||||
runtime_root: /data/digital-drosophila/models/male-cns-lif/runtime-v1
|
||||
io_root: /data/digital-drosophila/models/male-cns-lif/io-v1
|
||||
phase4c_root: /data/digital-drosophila/experiments/phase4c-runtime
|
||||
phase4d_root: /data/digital-drosophila/experiments/phase4d-io
|
||||
phase4e_root: /data/digital-drosophila/experiments/phase4e-visual-calibration
|
||||
phase4f_root: /data/digital-drosophila/experiments/phase4f-retinotopic-vision
|
||||
phase4f_model_root: /data/digital-drosophila/models/male-cns-lif/visual-encoder-v1
|
||||
phase4g_root: /data/digital-drosophila/experiments/phase4g-frozen-interface
|
||||
phase4g_model_root: /data/digital-drosophila/models/male-cns-lif/frozen-interface-v1
|
||||
phase4g_learned_root: /data/digital-drosophila/models/learned/phase4g
|
||||
phase4h_root: /data/digital-drosophila/experiments/phase4h-dopamine-plasticity
|
||||
phase4h_model_root: /data/digital-drosophila/models/male-cns-lif/dopamine-plasticity-v1
|
||||
phase4h_learned_root: /data/digital-drosophila/models/learned/phase4h
|
||||
phase4i_root: /data/digital-drosophila/experiments/phase4i-mb-mechanistic-validation
|
||||
phase4i_model_root: /data/digital-drosophila/models/male-cns-lif/mb-mechanistic-v1
|
||||
phase4j_root: /data/digital-drosophila/experiments/phase4j-mb-physiology-repair
|
||||
phase4j_model_root: /data/digital-drosophila/models/male-cns-lif/mb-physiology-v1
|
||||
|
||||
names:
|
||||
reference: MaleCNS-LIF-reference-v1
|
||||
runtime: MaleCNS-LIF-runtime-v1
|
||||
|
||||
model:
|
||||
dt_ms: 0.1
|
||||
membrane_tau_ms: 20.0
|
||||
synaptic_tau_ms: 5.0
|
||||
refractory_ms: 2.2
|
||||
resting_potential_mv: -52.0
|
||||
reset_potential_mv: -52.0
|
||||
threshold_mv: -45.0
|
||||
synaptic_delay_ms: 1.8
|
||||
w_syn_mv_per_anatomical_synapse: 0.275
|
||||
|
||||
sign_policy:
|
||||
neurotransmitter_sign: {acetylcholine: 1, gaba: -1, glutamate: -1, dopamine: 1, octopamine: 1, serotonin: 1, histamine: -1}
|
||||
unknown_sign: 0
|
||||
unclear_sign: 0
|
||||
|
||||
readout:
|
||||
bins_ms: [1, 5, 10, 20, 50, 100]
|
||||
|
||||
device:
|
||||
required_name_substring: RTX 4090
|
||||
launcher_visible_gpu_uuid: GPU-79f3a85d-adce-7833-7003-6488140ca596
|
||||
|
||||
expected_graph:
|
||||
neurons: 166700
|
||||
directed_edges: 25582938
|
||||
synaptic_contact_weight_sum: 124177617
|
||||
phase4h_plastic_candidate_edges: 14232
|
||||
plastic_mask_edges: 14232
|
||||
|
||||
apl_overlay:
|
||||
name: APL-Graded-Inhibition-v1
|
||||
tau_ms: 20.0
|
||||
gain: 0.01
|
||||
normalization: 1.0
|
||||
immutable_phase4j_parameters: true
|
||||
|
||||
direct_kc:
|
||||
target_compartment_verbatim: "y5B'2a"
|
||||
matched_dan_type: PAM15
|
||||
pulse_times_ms: [10.0, 30.0, 50.0, 70.0, 90.0]
|
||||
dan_time_ms: 90.0
|
||||
duration_ms: 120.0
|
||||
trace_seed: 50201
|
||||
evaluation_seed: 50301
|
||||
|
||||
olfactory:
|
||||
rate_hz: 50.0
|
||||
duration_ms: 100.0
|
||||
seed: 50101
|
||||
targets_per_cs: 64
|
||||
|
||||
sensitivity:
|
||||
depression_fractions: [0.0, 0.01, 0.05, 0.10, 0.25, 0.50]
|
||||
evaluation_seeds: [50301, 50302, 50303, 50304, 50305]
|
||||
functional_change_definition: any_MBON_spike_count_or_first_latency_change
|
||||
|
||||
causal_attribution:
|
||||
rule: dominant_positive_synaptic_arrival_with_runtime_g_decay
|
||||
exact_delay_steps: 18
|
||||
shortest_static_path_forbidden: true
|
||||
|
||||
prohibitions:
|
||||
training: false
|
||||
modify_eta_or_eligibility_v2: false
|
||||
expand_plastic_mask: false
|
||||
modify_threshold_or_gain: false
|
||||
save_sensitivity_as_learned_model: false
|
||||
decoder_classifier_rl_game_visual: false
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"file_count": 6,
|
||||
"files": [
|
||||
{
|
||||
"bytes": 363,
|
||||
"path": "VERSION.json",
|
||||
"sha256": "cc79527b6f36804f8ba4fc383988c08b550cb0eb43567f8b4ee3948f6231063b"
|
||||
},
|
||||
{
|
||||
"bytes": 6464,
|
||||
"path": "literature-audit.json",
|
||||
"sha256": "820ef25d0dff122a3cad1a763074ff5e6f74a5a7e08fb00620454ab4164f3fc4"
|
||||
},
|
||||
{
|
||||
"bytes": 35313,
|
||||
"path": "phase4m.py",
|
||||
"sha256": "3d1b64861559c9c04e08153b4056d968ab556637916d388170b36fc6348fa822"
|
||||
},
|
||||
{
|
||||
"bytes": 2339,
|
||||
"path": "phase4m.yaml",
|
||||
"sha256": "52c364d2ada03ba0c3b06220e91641c74ee79486c35d8c283237f02a98a2e471"
|
||||
},
|
||||
{
|
||||
"bytes": 5545,
|
||||
"path": "phase4m_finalize.py",
|
||||
"sha256": "58a2fd9bfc89725c741201f43ef06cf34840f78734190faad193ae8cee89c6ab"
|
||||
},
|
||||
{
|
||||
"bytes": 5487,
|
||||
"path": "phase4m_performance.py",
|
||||
"sha256": "7fab670032732d36823499c135a725f5ab7c8aa8305be780ee51f7eb0b8dfd57"
|
||||
}
|
||||
],
|
||||
"generated_utc": "2026-09-21T06:00:10.958765+00:00",
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/kcmbon-efficacy-v1",
|
||||
"total_bytes": 55511
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"base_CSR_read_only": true,
|
||||
"candidates": {
|
||||
"KCMBON-FunctionalTransfer-v2": true,
|
||||
"KCMBON-SynapticEfficacy-v1": true,
|
||||
"MaleCNS-OlfactoryAssociativeMemory-v1-candidate": false,
|
||||
"PresynapticEfficacy-v1": true
|
||||
},
|
||||
"name": "KCMBON-SynapticEfficacy-v1",
|
||||
"summary_sha256": "f0ba8af69ea749c7329f950d320e0330a0b9ad339730eb36e0a107b049eae38e"
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
6d7dd50ee8554af53223c9a3bcef60389a25f9eb75fa572f42cec034a171fc05 MANIFEST.json
|
||||
cc79527b6f36804f8ba4fc383988c08b550cb0eb43567f8b4ee3948f6231063b VERSION.json
|
||||
820ef25d0dff122a3cad1a763074ff5e6f74a5a7e08fb00620454ab4164f3fc4 literature-audit.json
|
||||
3d1b64861559c9c04e08153b4056d968ab556637916d388170b36fc6348fa822 phase4m.py
|
||||
52c364d2ada03ba0c3b06220e91641c74ee79486c35d8c283237f02a98a2e471 phase4m.yaml
|
||||
58a2fd9bfc89725c741201f43ef06cf34840f78734190faad193ae8cee89c6ab phase4m_finalize.py
|
||||
7fab670032732d36823499c135a725f5ab7c8aa8305be780ee51f7eb0b8dfd57 phase4m_performance.py
|
||||
@@ -0,0 +1,87 @@
|
||||
{
|
||||
"phase": "4M",
|
||||
"status": "LITERATURE-AUDIT-COMPLETE-BEFORE-IMPLEMENTATION",
|
||||
"scope": "Adult Drosophila KC-to-MBON plasticity; evidence is not a quantitative calibration of MaleCNS v1.0 synapses",
|
||||
"sources": [
|
||||
{
|
||||
"source": "Stahl et al., eLife 2022",
|
||||
"doi": "10.7554/eLife.76712",
|
||||
"pmid": "35285796",
|
||||
"url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC8956283/",
|
||||
"experiment_type": "in vivo learning protocols with acetylcholine sensor imaging along KC axonal compartments",
|
||||
"species_stage": "adult Drosophila melanogaster",
|
||||
"compartment": "multiple MB axonal compartments; longitudinally graded",
|
||||
"measured_quantity": "learning-related KC acetylcholine release change",
|
||||
"finding": "compartmentalized bidirectional ACh-release plasticity; valence and location matter",
|
||||
"model_interpretation": "supports an independently represented presynaptic transmitter-efficacy layer, but supplies no MaleCNS edge-specific efficacy or update coefficient"
|
||||
},
|
||||
{
|
||||
"source": "Davidson, Kaushik and Hige, eNeuro 2023",
|
||||
"doi": "10.1523/ENEURO.0275-23.2023",
|
||||
"pmid": "37848287",
|
||||
"url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC10616905/",
|
||||
"experiment_type": "odor-DAN pairing with in vivo MBON calcium, KC acetylcholine and individual bouton calcium imaging",
|
||||
"species_stage": "adult female Drosophila melanogaster, 3-6 days after eclosion",
|
||||
"compartment": "gamma5/beta-prime-2a and gamma1-pedc; gamma5/beta-prime-2a is related to but not an exact measured MaleCNS edge mapping for y5B'2a",
|
||||
"measured_quantity": "compartment-specific KC ACh release and heterogeneous single-bouton calcium signals",
|
||||
"finding": "paired DAN activity depressed odor-evoked ACh release, while bulk KC axonal calcium was unchanged; strongly responsive boutons could depress and weakly responsive boutons could potentiate",
|
||||
"model_interpretation": "supports local presynaptic efficacy and cautions against assigning identical biological release responses to all terminals; bouton-level parameters are not inferable from MaleCNS neuron-level CSR"
|
||||
},
|
||||
{
|
||||
"source": "Cohn et al., Cell 2015",
|
||||
"doi": "10.1016/j.cell.2015.11.019",
|
||||
"pmid": "26687359",
|
||||
"url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC4732734/",
|
||||
"experiment_type": "ex vivo KC stimulation and MBON electrophysiology with DAN activation",
|
||||
"species_stage": "adult Drosophila melanogaster",
|
||||
"compartment": "gamma4 and other tested MB domains",
|
||||
"measured_quantity": "KC-to-MBON synaptic current and presynaptic modulation",
|
||||
"finding": "dopaminergic protocols can potentiate or depress synaptic transmission; temporal order can reverse direction",
|
||||
"model_interpretation": "reinforces compartment/timing dependence; does not justify changing frozen Phase 4J timing polarity or eta"
|
||||
},
|
||||
{
|
||||
"source": "Pribbenow et al., eLife 2022",
|
||||
"doi": "10.7554/eLife.80445",
|
||||
"pmid": "36250621",
|
||||
"url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC9733945/",
|
||||
"experiment_type": "MBON receptor-subunit genetics, in vivo receptor dynamics and appetitive/familiarity behavior",
|
||||
"species_stage": "adult Drosophila melanogaster",
|
||||
"compartment": "M4/M6 and alpha-prime-3-related circuits, not a validated edge-level mapping to current MaleCNS y5B'2a target",
|
||||
"measured_quantity": "postsynaptic nAChR alpha5 requirement and alpha2 receptor dynamics",
|
||||
"finding": "postsynaptic cholinergic plasticity can contribute to memory induction/expression, with behavior/compartment specificity",
|
||||
"model_interpretation": "supports a separate postsynaptic research branch, but no defensible quantitative update for the current y5B'2a MaleCNS compartment; do not run a mixed branch here"
|
||||
},
|
||||
{
|
||||
"source": "Zhang et al., Journal of Neuroscience 2019",
|
||||
"doi": "10.1523/JNEUROSCI.1420-19.2019",
|
||||
"pmid": null,
|
||||
"url": "https://pmc.ncbi.nlm.nih.gov/articles/PMC6855682/",
|
||||
"experiment_type": "aversive conditioning with presynaptic and postsynaptic calcium imaging",
|
||||
"species_stage": "adult Drosophila melanogaster",
|
||||
"compartment": "alpha-prime-3",
|
||||
"measured_quantity": "pre- and postsynaptic odor-evoked calcium suppression",
|
||||
"finding": "both sides showed suppression across a similar post-training period; causality and quantitative partition are unresolved",
|
||||
"model_interpretation": "supports mixed mechanisms in some compartments, not automatic transfer of a postsynaptic coefficient to y5B'2a"
|
||||
}
|
||||
],
|
||||
"mechanism_conclusion": {
|
||||
"presynaptic": "direct support for compartment-specific learning-related KC ACh release modulation; exact release probability per MaleCNS edge remains unknown",
|
||||
"postsynaptic": "direct support for nAChR-related MBON changes in specific other circuits; no validated quantitative rule for the current target compartment",
|
||||
"mixed": "biologically plausible and compartment-dependent, but not justified as one blended MaleCNS y5B'2a rule",
|
||||
"timing_pairings": "LTD and opposite-direction changes depend on KC/DAN timing and training protocol; frozen eligibility-v2 is retained, not recalibrated",
|
||||
"saturation_recovery": "the cited studies do not identify a universal edge-level saturation/recovery constant for MaleCNS; physical nonnegative E and the pre-existing Phase 4J 25-percent diagnostic bound are explicit engineering assumptions, not measured values"
|
||||
},
|
||||
"decisions_before_simulation": {
|
||||
"main_branch": "PresynapticEfficacy-v1",
|
||||
"postsynaptic_branch": "AUDIT-ONLY-NOT-RUN-COMPARTMENT-SPECIFIC-QUANTITATIVE-RULE-UNKNOWN",
|
||||
"efficacy_interpretation": "dimensionless per-edge effective KC terminal drive multiplier, not measured release probability",
|
||||
"E_initial": 1.0,
|
||||
"E_unit_conversion": "one frozen eta_effective_mv increment is converted to a dimensionless per-anatomical-contact fraction by division by frozen W_syn_mv_per_anatomical_synapse; this is an explicit engineering unit convention, not fitted to Phase 4K sensitivity",
|
||||
"unbounded_multiplicative_lower_limit": 0.0,
|
||||
"bounded_multiplicative_lower_limit": 0.75,
|
||||
"bounded_limit_provenance": "Phase 4J frozen max_fractional_depression=0.25, not newly tuned",
|
||||
"five_percent_sensitivity_used_for_tuning": false,
|
||||
"numeric_efficacy_claimed_as_measured": false,
|
||||
"random_heterogeneity_added": false
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,360 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Phase 4M: isolated KC->MBON efficacy mechanisms on frozen MaleCNS inputs."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import gc
|
||||
import json
|
||||
import resource
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from collections import defaultdict
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import torch
|
||||
import yaml
|
||||
|
||||
from phase4h_audit import sha256, verify_checksum, write_json
|
||||
from phase4i import Validator, phase4c_regression
|
||||
from phase4j import apply_v2, effect, verify_frozen
|
||||
from phase4k import dataset_checks, seal
|
||||
from phase4l import OdorSimulator, pn_drive
|
||||
|
||||
|
||||
MECHANISMS = ("additive", "multiplicative", "bounded_multiplicative")
|
||||
|
||||
|
||||
def frozen_checks(cfg: dict[str, Any]) -> dict[str, Any]:
|
||||
earlier = verify_frozen(cfg); later = {}
|
||||
for key, control in (("phase4j_root", "manifests/checksums.sha256"), ("phase4j_model_root", "checksums.sha256"),
|
||||
("phase4k_root", "manifests/checksums.sha256"), ("phase4k_model_root", "checksums.sha256"),
|
||||
("phase4l_root", "manifests/checksums.sha256"), ("phase4l_model_root", "checksums.sha256")):
|
||||
root = Path(cfg["paths"][key]); later[key] = verify_checksum(root, root / control)
|
||||
report = {"phase4c_to_4i": earlier["all_passed"], "phase4j_to_4l": {k: v["passed"] for k, v in later.items()}}
|
||||
report["all_passed"] = bool(earlier["all_passed"] and all(v["passed"] for v in later.values()))
|
||||
if not report["all_passed"]: raise RuntimeError("frozen Phase 4C-4L checksum failure")
|
||||
return report
|
||||
|
||||
|
||||
def frozen_hashes(cfg: dict[str, Any]) -> dict[str, str]:
|
||||
p = cfg["paths"]; root = Path(p["connectome_root"]); files = [root / "gpu/csr-indptr.int64.npy", root / "gpu/csr-indices.int32.npy", root / "gpu/csr-weights.int64.npy", root / "neuron-index/unified-metadata.parquet",
|
||||
Path(p["phase4h_root"]) / "plastic-mask/plastic-edge-candidates.parquet",
|
||||
Path(p["phase4j_root"]) / "audit/phase4j.py", Path(p["phase4j_model_root"]) / "phase4j.yaml",
|
||||
Path(p["phase4l_root"]) / "odor-encoder/frozen-odor-panel.parquet", Path(p["phase4l_root"]) / "odor-encoder/mapped-glomerular-vectors.npz",
|
||||
Path(p["phase4l_root"]) / "receptor-mapping/olfactory-mapping.parquet", Path(p["phase4l_root"]) / "manifests/checksums.sha256",
|
||||
Path(p["phase4l_model_root"]) / "checksums.sha256"]
|
||||
for f in files:
|
||||
if not f.is_file(): raise FileNotFoundError(f)
|
||||
return {str(f): sha256(f) for f in files}
|
||||
|
||||
|
||||
def load_odor_inputs(cfg: dict[str, Any]) -> tuple[pd.DataFrame, dict[str, np.ndarray], dict[str, Any], pd.DataFrame]:
|
||||
exp = Path(cfg["paths"]["phase4l_root"]); panel = pd.read_parquet(exp / "odor-encoder/frozen-odor-panel.parquet"); vecfile = np.load(exp / "odor-encoder/mapped-glomerular-vectors.npz"); vectors = {str(oid): vecfile[str(oid)].copy() for oid in panel.odor_id}
|
||||
selection = json.loads((exp / "odor-encoder/selection.json").read_text()); mapping = pd.read_parquet(exp / "receptor-mapping/olfactory-mapping.parquet")
|
||||
if selection["ODOR_A"]["odor_id"] not in vectors or selection["ODOR_B"]["odor_id"] not in vectors: raise RuntimeError("frozen odor identity mismatch")
|
||||
return panel, vectors, selection, mapping
|
||||
|
||||
|
||||
def anatomical_audit(v: Validator, cfg: dict[str, Any], exp: Path) -> tuple[np.ndarray, np.ndarray]:
|
||||
cand = v.cand; raw = np.asarray(v.rt.raw_np[v.edge_ids], dtype=np.int64); sign = v.rt.signs[cand.source_neuron_idx.to_numpy(np.int32)]
|
||||
w = (raw.astype(np.float32) * sign.astype(np.float32) * np.float32(cfg["model"]["w_syn_mv_per_anatomical_synapse"])).astype(np.float32)
|
||||
base = v.base_candidate.astype(np.float32)
|
||||
if not np.array_equal(raw, cand.raw_synapse_count.to_numpy(np.int64)): raise RuntimeError("plastic mask/raw CSR count mismatch")
|
||||
if not np.array_equal(w, base): raise RuntimeError("anatomical effective weight mismatch with frozen runtime")
|
||||
if not np.all(sign == 1): raise RuntimeError("presynaptic efficacy branch requires cholinergic positive KC candidates")
|
||||
if len(cand) != 14232 or len(np.unique(v.edge_ids)) != 14232: raise RuntimeError("plastic mask changed")
|
||||
report = {"plastic_mask_edges": len(cand), "raw_count_min": int(raw.min()), "raw_count_median": float(np.median(raw)), "raw_count_max": int(raw.max()),
|
||||
"raw_count_sum": int(raw.sum()), "anatomical_weight_min_mv": float(w.min()), "anatomical_weight_max_mv": float(w.max()),
|
||||
"all_cholinergic_positive": True, "anatomical_formula_verified_exactly": True, "base_CSR_read_only": True,
|
||||
"E_birth_state": 1.0, "other_brain_edges_E": 1.0}
|
||||
write_json(exp / "anatomical-weight/summary.json", report); return raw, w
|
||||
|
||||
|
||||
class ReplayEngine:
|
||||
"""Real KC spikes + existing KC->MBON edges + frozen MBON LIF, no invented edge."""
|
||||
|
||||
def __init__(self, sim: OdorSimulator, targets: np.ndarray):
|
||||
self.sim = sim; self.targets = np.asarray(targets, np.int32); self.dt = sim.dt; self.rt = sim.rt; v = sim.v
|
||||
indices = v.rt.indices_np; sources = sim.sources; mask = np.isin(sources, v.kc) & np.isin(indices, self.targets); self.ids = np.flatnonzero(mask)
|
||||
self.src = sources[self.ids].astype(np.int32); self.target = indices[self.ids].astype(np.int32)
|
||||
pos = {int(x): i for i, x in enumerate(self.targets)}; self.target_local = np.fromiter((pos[int(x)] for x in self.target), np.int32, len(self.target))
|
||||
all_base = v.base.detach().cpu().numpy(); self.base = all_base[self.ids].astype(np.float32)
|
||||
can_index = {int(eid): i for i, eid in enumerate(v.edge_ids)}; self.candidate_local = np.fromiter((can_index.get(int(x), -1) for x in self.ids), np.int32, len(self.ids))
|
||||
self.by_source = {int(s): np.flatnonzero(self.src == s) for s in np.unique(self.src)}
|
||||
if not np.all(self.base > 0): raise RuntimeError("isolated KC replay contains nonpositive source weights")
|
||||
|
||||
def run(self, spikes: pd.DataFrame, delta: np.ndarray, duration_ms: float) -> dict[str, Any]:
|
||||
weights = self.base.copy(); valid = self.candidate_local >= 0; weights[valid] += delta[self.candidate_local[valid]]
|
||||
grouped = {int(k): g.neuron_idx.to_numpy(np.int32) for k, g in spikes.groupby("step")}; nsteps = int(round(duration_ms / self.dt)); n = len(self.targets); q = np.zeros((self.rt.queue_len, n), np.float32)
|
||||
conductance = np.zeros(n, np.float32); voltage = np.full(n, self.rt.rest, np.float32); refractory = np.zeros(n, np.int32); counts = np.zeros(n, np.int32); first = np.full(n, -1, np.int32)
|
||||
integrated_g = np.zeros(n, np.float64); transmitted = 0.0; delivered = 0.0; wall0 = time.perf_counter()
|
||||
for step in range(nsteps):
|
||||
slot = step % self.rt.queue_len; conductance = conductance * np.float32(self.rt.g_decay) + q[slot]; delivered += float(q[slot].sum(dtype=np.float64)); q[slot].fill(0)
|
||||
integrated_g += conductance; refractory = np.maximum(refractory, 1) - 1; avail = refractory == 0
|
||||
voltage[avail] += np.float32(self.rt.alpha) * (conductance[avail] - (voltage[avail] - np.float32(self.rt.rest))); voltage[~avail] = np.float32(self.rt.reset_value)
|
||||
fired = avail & (voltage >= np.float32(self.rt.threshold)); counts += fired; first[(first < 0) & fired] = step; voltage[fired] = np.float32(self.rt.reset_value); refractory[fired] = self.rt.refractory_steps
|
||||
for source in grouped.get(step, np.empty(0, np.int32)):
|
||||
rel = self.by_source.get(int(source))
|
||||
if rel is not None:
|
||||
np.add.at(q[(step + self.rt.delay_steps) % self.rt.queue_len], self.target_local[rel], weights[rel]); transmitted += float(weights[rel].sum(dtype=np.float64))
|
||||
return {"counts": counts, "first": first, "final_g": conductance, "final_v": voltage, "integrated_g": integrated_g,
|
||||
"transmitted_weighted_events_mv": transmitted, "delivered_input_mv": delivered, "wall_seconds": time.perf_counter() - wall0}
|
||||
|
||||
|
||||
def replay_effect(pre: dict[str, Any], post: dict[str, Any], dt: float) -> dict[str, Any]:
|
||||
def first_ms(x: dict[str, Any]) -> float | None:
|
||||
a = x["first"]; return None if np.all(a < 0) else float(a[a >= 0].min() * dt)
|
||||
a, b = first_ms(pre), first_ms(post)
|
||||
return {"delta_g_max_abs_mv": float(np.abs(post["final_g"] - pre["final_g"]).max()), "delta_g_mean_abs_mv": float(np.abs(post["final_g"] - pre["final_g"]).mean()),
|
||||
"delta_v_max_abs_mv": float(np.abs(post["final_v"] - pre["final_v"]).max()), "delta_v_mean_abs_mv": float(np.abs(post["final_v"] - pre["final_v"]).mean()),
|
||||
"integrated_g_delta_sum_mv_steps": float((post["integrated_g"] - pre["integrated_g"]).sum()),
|
||||
"MBON_spikes_pre": int(pre["counts"].sum()), "MBON_spikes_post": int(post["counts"].sum()), "MBON_spike_count_delta": int(post["counts"].sum() - pre["counts"].sum()),
|
||||
"MBON_spike_L1": int(np.abs(post["counts"] - pre["counts"]).sum()), "first_latency_pre_ms": a, "first_latency_post_ms": b,
|
||||
"first_latency_delta_ms": None if a is None or b is None else float(b - a), "transmitted_delta_mv": float(post["transmitted_weighted_events_mv"] - pre["transmitted_weighted_events_mv"]),
|
||||
"transmitted_reduction_percent": float(100 * (pre["transmitted_weighted_events_mv"] - post["transmitted_weighted_events_mv"]) / pre["transmitted_weighted_events_mv"]) if pre["transmitted_weighted_events_mv"] else None,
|
||||
"MBON_total_KC_input_reduction_percent": float(100 * (pre["delivered_input_mv"] - post["delivered_input_mv"]) / pre["delivered_input_mv"]) if pre["delivered_input_mv"] else None,
|
||||
"isolated_input_scope": "all real KC->target-MBON edges, not all whole-brain MBON inputs"}
|
||||
|
||||
|
||||
def latency_direction(row: dict[str, Any]) -> bool:
|
||||
value = row.get("first_latency_delta_ms"); return bool(row["MBON_spike_count_delta"] < 0 or (value is not None and value > 0))
|
||||
|
||||
|
||||
def compare_A_B(a: dict[str, Any], b: dict[str, Any]) -> bool:
|
||||
if not latency_direction(a): return False
|
||||
if a["MBON_spike_L1"] > b["MBON_spike_L1"]: return True
|
||||
al = 0.0 if a["first_latency_delta_ms"] is None else max(float(a["first_latency_delta_ms"]), 0.0)
|
||||
bl = 0.0 if b["first_latency_delta_ms"] is None else max(float(b["first_latency_delta_ms"]), 0.0)
|
||||
return bool(al > bl and a["MBON_spike_L1"] >= b["MBON_spike_L1"])
|
||||
|
||||
|
||||
def main_experiment(v: Validator, sim: OdorSimulator, cfg: dict[str, Any], mcfg: dict[str, Any], exp: Path,
|
||||
panel: pd.DataFrame, vectors: dict[str, np.ndarray], selection: dict[str, Any], mapping: pd.DataFrame,
|
||||
anatomical: np.ndarray) -> tuple[dict[str, Any], dict[int, dict[int, dict[str, np.ndarray]]], ReplayEngine]:
|
||||
protocol = mcfg["protocol"]; points = list(map(int, protocol["pairing_counts"])); maxpair = max(points); seeds = list(map(int, protocol["train_seeds"])); eval_seeds = list(map(int, protocol["evaluation_seeds"]));
|
||||
if points != list(map(int, cfg["plasticity_accumulation"]["pairing_counts"])) or seeds != list(map(int, cfg["plasticity_accumulation"]["train_seeds"])) or eval_seeds != list(map(int, cfg["plasticity_accumulation"]["evaluation_seeds"])): raise RuntimeError("Phase 4L frozen pairing protocol mismatch")
|
||||
compartment = str(protocol["target_compartment_verbatim"]); dan = v.dan_targets(str(protocol["matched_dan_type"]), compartment); duration = float(protocol["duration_ms"])
|
||||
gloms = selection["glomeruli_order"]; matched_sum = float(np.median(panel.mapped_response_sum)); pn = {}
|
||||
for label in ("ODOR_A", "ODOR_B"):
|
||||
oid = selection[label]["odor_id"]; target, rate, _ = pn_drive(oid, vectors[oid], mapping, gloms, str(protocol["input_mode"]), matched_sum, cfg); pn[label] = (target, rate)
|
||||
src = v.cand.source_neuron_idx.to_numpy(np.int32); lookup = np.full(v.n, -1, np.int32); lookup[v.kc] = np.arange(len(v.kc), dtype=np.int32); loc = lookup[src]
|
||||
if np.any(loc < 0): raise RuntimeError("non-KC source in plastic mask")
|
||||
comp = v.cand.target_compartment_verbatim.astype(str).to_numpy() == compartment; k = float(cfg["eligibility_v2"]["eta_effective_mv"]) / float(cfg["model"]["w_syn_mv_per_anatomical_synapse"])
|
||||
if k <= 0 or k >= 1: raise RuntimeError("invalid frozen eta/Wsyn dimensionless conversion")
|
||||
bound = 1.0 - float(cfg["eligibility_v2"]["max_fractional_depression"]); checkpoints: dict[int, dict[int, dict[str, np.ndarray]]] = {}; histories = []; update_times = []; training_walls = []; reproduction = []
|
||||
zero = np.zeros(len(v.edge_ids), np.float32); target_mbon = np.unique(v.cand.loc[comp, "target_neuron_idx"].to_numpy(np.int32)); replay = ReplayEngine(sim, target_mbon)
|
||||
# Baseline replay uses the same deterministic evaluation KC spike trains for all mechanisms.
|
||||
evaluation = {}; pre_replay = {}
|
||||
for es in eval_seeds:
|
||||
for label, (targets, rates) in pn.items():
|
||||
run = sim.run_rates(targets, rates, es, duration, collect_kc_spikes=True); evaluation[(es, label)] = run["kc_spikes"]
|
||||
run["kc_spikes"].to_parquet(exp / "isolated-replay" / f"KC-spikes-{label}-seed-{es}.parquet", index=False)
|
||||
pre_replay[(es, label)] = replay.run(run["kc_spikes"], zero, duration)
|
||||
for ts in seeds:
|
||||
delta_add = zero.copy(); em = np.ones(len(v.edge_ids), np.float32); eb = np.ones(len(v.edge_ids), np.float32); checkpoints[ts] = {}; p4l = np.load(Path(cfg["paths"]["phase4l_root"]) / "plasticity-magnitude" / f"seed-{ts}-overlays.npz")
|
||||
if not np.array_equal(p4l["edge_id"], v.edge_ids): raise RuntimeError("Phase 4L additive checkpoint edge IDs differ")
|
||||
for pairing in range(1, maxpair + 1):
|
||||
run = sim.run_rates(*pn["ODOR_A"], ts + pairing, duration, delta=delta_add, dan_targets=dan, dan_pulses=[float(protocol["dan_time_ms"])])
|
||||
training_walls.append(run["wall_seconds"]); trace = run["kc_trace_at_dan"]; elig = trace[loc].astype(np.float32); active = comp & (elig > 0)
|
||||
old_m, old_b = em[active].copy(), eb[active].copy(); t0 = time.perf_counter()
|
||||
delta_add, _ = apply_v2(v, trace, compartment, 1.0, delta_add, cfg)
|
||||
factor = np.float32(1.0) - np.float32(k) * elig[active]; em[active] = np.maximum(np.float32(0.0), em[active] * factor)
|
||||
eb[active] = np.maximum(np.float32(bound), eb[active] * factor); update_times.append(time.perf_counter() - t0)
|
||||
for local, om, ob in zip(np.flatnonzero(active), old_m, old_b):
|
||||
histories.append({"train_seed": ts, "pairing": pairing, "edge_id": int(v.edge_ids[local]), "eligibility": float(elig[local]),
|
||||
"E_multiplicative_before": float(om), "E_multiplicative_after": float(em[local]), "E_bounded_before": float(ob), "E_bounded_after": float(eb[local])})
|
||||
if pairing in points:
|
||||
saved = p4l[f"pairing_{pairing}"].astype(np.float32); error = float(np.max(np.abs(saved - delta_add)))
|
||||
reproduction.append({"train_seed": ts, "pairing": pairing, "max_abs_error_mv": error, "exact": bool(np.array_equal(saved, delta_add))})
|
||||
if error > 1e-6: raise RuntimeError(f"Phase 4L additive training reproduction divergence seed={ts} pairing={pairing}: {error}")
|
||||
checkpoints[ts][pairing] = {"additive": delta_add.copy(), "multiplicative": em.copy(), "bounded_multiplicative": eb.copy(), "eligibility_at_last_DAN": elig.copy()}
|
||||
np.savez_compressed(exp / "efficacy-overlay" / f"E-seed-{ts}-pairings-{pairing}.npz", E_multiplicative=em, E_bounded_multiplicative=eb)
|
||||
p4l.close()
|
||||
pd.DataFrame(histories).to_parquet(exp / "efficacy-overlay/update-history.parquet", index=False)
|
||||
write_json(exp / "regression/phase4l-additive-reproduction.json", {"all_passed": all(x["max_abs_error_mv"] <= 1e-6 for x in reproduction), "checks": reproduction})
|
||||
if not np.all(np.isfinite([h["E_multiplicative_after"] for h in histories])) or not np.all(np.isfinite([h["E_bounded_after"] for h in histories])):
|
||||
raise RuntimeError("nonfinite efficacy")
|
||||
rows = []; replay_times = []
|
||||
for ts, es in zip(seeds, eval_seeds):
|
||||
for pairing in points:
|
||||
state = checkpoints[ts][pairing]
|
||||
for mechanism in MECHANISMS:
|
||||
if mechanism == "additive": delta = state["additive"]; efficacy = 1.0 + delta / anatomical
|
||||
else:
|
||||
efficacy = state[mechanism]; delta = anatomical * (efficacy - 1.0)
|
||||
if not np.all(np.isfinite(efficacy)) or np.any(efficacy < -1e-6) or np.any(efficacy > 1.0 + 1e-6): raise RuntimeError("invalid E state")
|
||||
changed = np.abs(delta) > 1e-8; e_changed = efficacy[changed]; weighted_E = float(np.sum(anatomical[changed] * e_changed) / np.sum(anatomical[changed])) if np.any(changed) else 1.0
|
||||
pairrow = {"train_seed": ts, "eval_seed": es, "pairings": pairing, "mechanism": mechanism, "updated_edges": int(changed.sum()),
|
||||
"E_mean_all_mask": float(np.mean(efficacy)), "E_median_all_mask": float(np.median(efficacy)), "E_min_all_mask": float(np.min(efficacy)),
|
||||
"E_mean_changed": float(np.mean(e_changed)) if len(e_changed) else 1.0, "E_median_changed": float(np.median(e_changed)) if len(e_changed) else 1.0,
|
||||
"E_anatomical_weighted_changed": weighted_E, "effective_weight_delta_sum_mv": float(delta.sum()),
|
||||
"effective_weight_reduction_percent_changed": float(100 * (1 - weighted_E)), "saturation_fraction_mask": float(np.count_nonzero(efficacy <= (bound + 1e-6)) / len(efficacy))}
|
||||
result = {}
|
||||
for label in ("ODOR_A", "ODOR_B"):
|
||||
t0 = time.perf_counter(); post = replay.run(evaluation[(es, label)], delta, duration); replay_times.append(time.perf_counter() - t0)
|
||||
result[label] = replay_effect(pre_replay[(es, label)], post, sim.dt)
|
||||
pairrow.update({f"A_{k}": value for k, value in result["ODOR_A"].items() if k != "isolated_input_scope"})
|
||||
pairrow.update({f"B_{k}": value for k, value in result["ODOR_B"].items() if k != "isolated_input_scope"})
|
||||
pairrow["A_specific_functional"] = compare_A_B(result["ODOR_A"], result["ODOR_B"])
|
||||
rows.append(pairrow)
|
||||
table = pd.DataFrame(rows); table.to_parquet(exp / "isolated-replay/mechanism-comparison.parquet", index=False)
|
||||
curve = []
|
||||
for (mechanism, pairing), frame in table.groupby(["mechanism", "pairings"]):
|
||||
curve.append({"mechanism": mechanism, "pairings": int(pairing), "seeds": int(len(frame)), "A_specific_functional_seeds": int(frame.A_specific_functional.sum()),
|
||||
"E_mean_changed": float(frame.E_mean_changed.mean()), "E_median_changed": float(frame.E_median_changed.mean()), "E_anatomical_weighted_changed": float(frame.E_anatomical_weighted_changed.mean()),
|
||||
"E_min_mask": float(frame.E_min_all_mask.min()), "effective_weight_reduction_percent_changed": float(frame.effective_weight_reduction_percent_changed.mean()),
|
||||
"A_transmission_reduction_percent": float(frame.A_transmitted_reduction_percent.mean()), "B_transmission_reduction_percent": float(frame.B_transmitted_reduction_percent.mean()),
|
||||
"A_total_KC_input_reduction_percent": float(frame.A_MBON_total_KC_input_reduction_percent.mean()), "B_total_KC_input_reduction_percent": float(frame.B_MBON_total_KC_input_reduction_percent.mean()),
|
||||
"A_MBON_delta_g_max_abs_mv": float(frame.A_delta_g_max_abs_mv.mean()), "A_MBON_delta_v_max_abs_mv": float(frame.A_delta_v_max_abs_mv.mean()),
|
||||
"A_MBON_spike_count_delta_mean": float(frame.A_MBON_spike_count_delta.mean()), "A_MBON_spike_L1_mean": float(frame.A_MBON_spike_L1.mean()),
|
||||
"A_MBON_first_latency_delta_ms_mean": float(frame.A_first_latency_delta_ms.mean()) if frame.A_first_latency_delta_ms.notna().any() else None,
|
||||
"B_MBON_spike_L1_mean": float(frame.B_MBON_spike_L1.mean()), "saturation_fraction_mask": float(frame.saturation_fraction_mask.mean())})
|
||||
gate = {}; majority = int(mcfg["gates"]["seed_majority"])
|
||||
for mech in MECHANISMS:
|
||||
successes = [x for x in curve if x["mechanism"] == mech and x["A_specific_functional_seeds"] >= majority]
|
||||
gate[mech] = {"pass": bool(successes), "first_passing_pairing": min((x["pairings"] for x in successes), default=None), "seed_majority": majority,
|
||||
"endpoint_100_specific_functional_seeds": next(x["A_specific_functional_seeds"] for x in curve if x["mechanism"] == mech and x["pairings"] == 100)}
|
||||
primary = gate["multiplicative"]["pass"] or gate["bounded_multiplicative"]["pass"]
|
||||
report = {"status": "PASS-KC-MBON-FUNCTIONAL-TRANSFER-v2" if primary else "FAIL-KC-MBON-FUNCTIONAL-TRANSFER-v2", "presynaptic_functional_transfer_pass": primary,
|
||||
"mechanism_gates": gate, "curve": curve, "same_KC_spikes_DAN_timing_mask_and_seeds": True, "phase4l_additive_reproduced": True,
|
||||
"postsynaptic_branch_run": False, "no_dense_time_by_edge_state": True,
|
||||
"performance": {"mean_training_brain_wall_seconds_per_pairing": float(np.mean(training_walls)), "mean_update_seconds_per_pairing": float(np.mean(update_times)),
|
||||
"mean_isolated_replay_seconds": float(np.mean(replay_times)), "E_overlay_memory_bytes_two_float32_arrays": 2 * len(v.edge_ids) * 4}}
|
||||
write_json(exp / "isolated-replay/summary.json", report)
|
||||
return report, checkpoints, replay
|
||||
|
||||
|
||||
def anatomy_vs_efficacy(v: Validator, sim: OdorSimulator, cfg: dict[str, Any], exp: Path, anatomical: np.ndarray,
|
||||
checkpoints: dict[int, dict[int, dict[str, np.ndarray]]], mcfg: dict[str, Any]) -> dict[str, Any]:
|
||||
targets = v.cand.target_neuron_idx.to_numpy(np.int32); base = v.base.detach().cpu().numpy(); positive = np.maximum(base, 0.0)
|
||||
target_total = np.bincount(v.rt.indices_np, weights=positive.astype(np.float64), minlength=v.n)
|
||||
frames = []
|
||||
for ts in map(int, mcfg["protocol"]["train_seeds"]):
|
||||
state = checkpoints[ts][100]
|
||||
for mechanism in ("multiplicative", "bounded_multiplicative"):
|
||||
E = state[mechanism]; frame = v.cand[["edge_id", "source_neuron_idx", "source_body_id", "target_neuron_idx", "target_body_id", "raw_synapse_count", "target_compartment_verbatim"]].copy()
|
||||
frame["train_seed"] = ts; frame["mechanism"] = mechanism; frame["anatomical_weight_mv"] = anatomical; frame["E_baseline"] = 1.0; frame["E_current"] = E
|
||||
frame["effective_weight_mv"] = anatomical * E; frame["target_total_positive_anatomical_input_mv"] = target_total[targets]
|
||||
frame["target_MBON_total_input_fraction_static"] = anatomical * E / np.maximum(target_total[targets], 1e-12)
|
||||
frame["eligibility_at_last_DAN"] = state["eligibility_at_last_DAN"]
|
||||
frame["DAN_channel"] = mcfg["protocol"]["matched_dan_type"]; frames.append(frame)
|
||||
out = pd.concat(frames, ignore_index=True); out.to_parquet(exp / "anatomy-vs-efficacy/anatomy-vs-efficacy.parquet", index=False)
|
||||
from scipy.stats import spearmanr
|
||||
by = out[(out.train_seed == int(mcfg["protocol"]["train_seeds"][0])) & (out.mechanism == "bounded_multiplicative")]
|
||||
rho = spearmanr(by.raw_synapse_count, by.effective_weight_mv).statistic
|
||||
report = {"rows": int(len(out)), "unique_edges": int(out.edge_id.nunique()), "raw_count_vs_effective_weight_spearman_seed1_bounded": float(rho),
|
||||
"high_anatomical_count_implies_high_current_effective_in_this_assumed_model_only": bool(rho > 0.9),
|
||||
"physiological_influence_inferred_from_EM_count": False, "static_fraction_is_not_dynamic_MBON_input": True}
|
||||
write_json(exp / "anatomy-vs-efficacy/summary.json", report); return report
|
||||
|
||||
|
||||
def run_wholebrain(v: Validator, sim: OdorSimulator, cfg: dict[str, Any], mcfg: dict[str, Any], exp: Path, panel: pd.DataFrame,
|
||||
vectors: dict[str, np.ndarray], selection: dict[str, Any], mapping: pd.DataFrame,
|
||||
anatomical: np.ndarray, checkpoints: dict[int, dict[int, dict[str, np.ndarray]]], isolated: dict[str, Any]) -> dict[str, Any]:
|
||||
passed = [x for x in ("multiplicative", "bounded_multiplicative") if isolated["mechanism_gates"][x]["pass"]]
|
||||
if not passed:
|
||||
out = {"status": "NOT-RUN-ISOLATED-ENDPOINT-GATE-BLOCKED", "tested_mechanisms": [], "feedback_masks_memory": None}; write_json(exp / "whole-brain-replay/summary.json", out); return out
|
||||
duration = float(mcfg["protocol"]["duration_ms"]); gloms = selection["glomeruli_order"]; matched = float(np.median(panel.mapped_response_sum)); pn = {}
|
||||
for label in ("ODOR_A", "ODOR_B"):
|
||||
oid = selection[label]["odor_id"]; pn[label] = pn_drive(oid, vectors[oid], mapping, gloms, str(mcfg["protocol"]["input_mode"]), matched, cfg)[:2]
|
||||
rows = []; walls = []; zero = np.zeros(len(v.edge_ids), np.float32)
|
||||
for ts, es in zip(map(int, mcfg["protocol"]["train_seeds"]), map(int, mcfg["protocol"]["evaluation_seeds"])):
|
||||
baseline = {label: sim.run_rates(*pn[label], es, duration, delta=zero) for label in pn}
|
||||
for mechanism in passed:
|
||||
E = checkpoints[ts][100][mechanism]; delta = anatomical * (E - 1)
|
||||
effects = {}
|
||||
for label in pn:
|
||||
post = sim.run_rates(*pn[label], es, duration, delta=delta); walls.append(post["wall_seconds"]); effects[label] = effect(baseline[label], post)
|
||||
rows.append({"train_seed": ts, "eval_seed": es, "mechanism": mechanism, "odor": label, "MBON_spike_L1": effects[label]["MBON_spike_l1"],
|
||||
"MBON_g_max_abs_mv": effects[label]["MBON_g_max_abs"], "MBON_v_max_abs_mv": effects[label]["MBON_v_max_abs"],
|
||||
"MBON_latency_delta_ms": effects[label]["MBON_first_latency_delta_ms"], "downstream_spike_L1": effects[label]["downstream_spike_l1"],
|
||||
"KC_spike_L1": int(np.abs(post["kc_counts"] - baseline[label]["kc_counts"]).sum()),
|
||||
"finite": bool(post["finite"]), "active_neuron_fraction": float(np.count_nonzero(post["counts"]) / v.n), "wall_seconds": post["wall_seconds"]})
|
||||
frame = pd.DataFrame(rows); frame.to_parquet(exp / "whole-brain-replay/results.parquet", index=False); gates = {}
|
||||
for mechanism in passed:
|
||||
sub = frame[frame.mechanism == mechanism]; a = sub[sub.odor == "ODOR_A"].sort_values("train_seed"); b = sub[sub.odor == "ODOR_B"].sort_values("train_seed")
|
||||
wins = int(((a.MBON_spike_L1.to_numpy() > b.MBON_spike_L1.to_numpy()) & (a.MBON_spike_L1.to_numpy() > 0)).sum())
|
||||
gates[mechanism] = {"A_specific_functional_seeds": wins, "pass": bool(wins >= int(mcfg["gates"]["seed_majority"]) and sub.finite.all() and sub.active_neuron_fraction.max() < .5),
|
||||
"A_MBON_spike_L1_mean": float(a.MBON_spike_L1.mean()), "B_MBON_spike_L1_mean": float(b.MBON_spike_L1.mean()),
|
||||
"A_downstream_spike_L1_mean": float(a.downstream_spike_L1.mean()), "B_downstream_spike_L1_mean": float(b.downstream_spike_L1.mean()),
|
||||
"A_KC_spike_L1_mean": float(a.KC_spike_L1.mean()), "B_KC_spike_L1_mean": float(b.KC_spike_L1.mean())}
|
||||
status = "PASS-WHOLE-BRAIN-KCMBON-TRANSFER" if any(x["pass"] for x in gates.values()) else "WHOLE-BRAIN-FEEDBACK-MASKS-MEMORY"
|
||||
out = {"status": status, "tested_mechanisms": passed, "gates": gates, "feedback_masks_memory": status != "PASS-WHOLE-BRAIN-KCMBON-TRANSFER", "mean_post_brain_wall_seconds": float(np.mean(walls))}
|
||||
write_json(exp / "whole-brain-replay/summary.json", out); return out
|
||||
|
||||
|
||||
def conditioning_gate(cfg: dict[str, Any], mcfg: dict[str, Any], exp: Path, isolated: dict[str, Any], whole: dict[str, Any], identity: dict[str, Any]) -> dict[str, Any]:
|
||||
encoder = json.loads((Path(cfg["paths"]["phase4l_root"]) / "kc-representation/summary.json").read_text()); temporal = json.loads((Path(cfg["paths"]["phase4j_root"]) / "eligibility-v2/unit-tests.json").read_text())
|
||||
gates = {"OlfactoryEncoder_v1": bool(encoder["candidate_generated"]), "TemporalEligibility_v2": bool(temporal["all_passed"]),
|
||||
"KC_sparse_code": bool(encoder["modes"]["physiological"]["gate"] and encoder["modes"]["matched_energy"]["gate"]),
|
||||
"A_B_specific_edges": bool(identity["A_specific_eligible_edges"] > 0 and identity["B_specific_eligible_edges"] > 0),
|
||||
"isolated_functional_transfer_v2": bool(isolated["presynaptic_functional_transfer_pass"]),
|
||||
"whole_brain_transfer": whole["status"] == "PASS-WHOLE-BRAIN-KCMBON-TRANSFER"}
|
||||
report = {"status": "OPEN" if all(gates.values()) else "NOT-RUN-GATE-BLOCKED", "gates": gates, "formal_conditioning_run": False,
|
||||
"acquisition": "NOT-RUN" if not all(gates.values()) else "PENDING", "controls": "NOT-RUN" if not all(gates.values()) else "PENDING"}
|
||||
write_json(exp / "conditioning/gate.json", report); return report
|
||||
|
||||
|
||||
def integrity(cfg: dict[str, Any], mcfg: dict[str, Any], base_config: Path, exp: Path, before_hash: dict[str, str]) -> dict[str, Any]:
|
||||
del mcfg
|
||||
gc.collect(); torch.cuda.empty_cache(); checks = frozen_checks(cfg); data = dataset_checks(cfg); regression = phase4c_regression(cfg, base_config, exp); after = frozen_hashes(cfg)
|
||||
report = {"all_passed": bool(checks["all_passed"] and data["all_passed"] and regression["all_passed"] and before_hash == after),
|
||||
"frozen_checks": checks, "phase1_to_3b_dataset": data, "phase4c_reference_regression": regression,
|
||||
"frozen_hashes_identical": before_hash == after, "before": before_hash, "after": after,
|
||||
"base_CSR_modified": False, "APL_modified": False, "eligibility_v2_modified": False, "plastic_mask_expanded": False,
|
||||
"OlfactoryEncoder_modified": False, "PlasticityAccumulation_modified": False}
|
||||
write_json(exp / "audit/integrity.json", report); return report
|
||||
|
||||
|
||||
def main() -> None:
|
||||
ap = argparse.ArgumentParser(); ap.add_argument("--config", type=Path, required=True); args = ap.parse_args(); mcfg = yaml.safe_load(args.config.read_text()); base_config = Path(mcfg["frozen_phase4l_config"])
|
||||
cfg = yaml.safe_load(base_config.read_text()); cfg["paths"]["phase4l_root"] = cfg["paths"]["experiment_root"]; cfg["paths"]["phase4l_model_root"] = mcfg["frozen_phase4l_config"].rsplit("/", 1)[0]
|
||||
exp, model = Path(mcfg["experiment_root"]), Path(mcfg["model_root"])
|
||||
for sub in ("literature-audit", "anatomical-weight", "efficacy-overlay", "presynaptic", "postsynaptic", "anatomy-vs-efficacy", "isolated-replay", "whole-brain-replay", "conditioning", "controls", "performance", "regression", "audit", "manifests", "compiler-reports/extension-build"):
|
||||
(exp / sub).mkdir(parents=True, exist_ok=True)
|
||||
model.mkdir(parents=True, exist_ok=True)
|
||||
if not (exp / "literature-audit/phase4m_literature_audit.json").is_file(): raise RuntimeError("literature audit must predate model implementation")
|
||||
prior = frozen_checks(cfg); before_hash = frozen_hashes(cfg); write_json(exp / "audit/frozen-before.json", {"checks": prior, "hashes": before_hash})
|
||||
shutil.copy2(args.config, exp / "audit/phase4m.yaml"); shutil.copy2(Path(__file__).resolve(), exp / "audit/phase4m.py")
|
||||
panel, vectors, selection, mapping = load_odor_inputs(cfg); v = Validator(cfg, base_config, exp); sim = OdorSimulator(v, cfg, exp); raw, anatomical = anatomical_audit(v, cfg, exp)
|
||||
isolated, checkpoints, replay = main_experiment(v, sim, cfg, mcfg, exp, panel, vectors, selection, mapping, anatomical)
|
||||
anatomy = anatomy_vs_efficacy(v, sim, cfg, exp, anatomical, checkpoints, mcfg)
|
||||
post = {"status": mcfg["postsynaptic"]["current_status"], "branch_defined": True, "run": False,
|
||||
"reason": "postsynaptic nAChR evidence exists in adult Drosophila, but a quantitative rule for exact MaleCNS y5B'2a target and its receptor map is unavailable",
|
||||
"combined_presynaptic_postsynaptic_run": False, "global_MBON_threshold_modified": False}
|
||||
write_json(exp / "postsynaptic/summary.json", post)
|
||||
whole = run_wholebrain(v, sim, cfg, mcfg, exp, panel, vectors, selection, mapping, anatomical, checkpoints, isolated)
|
||||
edge_identity = json.loads((Path(cfg["paths"]["phase4l_root"]) / "eligible-edges/summary.json").read_text())
|
||||
formal = conditioning_gate(cfg, mcfg, exp, isolated, whole, edge_identity)
|
||||
# Formal conditioning implementation is intentionally gated separately: this
|
||||
# diagnostic run never creates a memory claim from the efficacy comparison.
|
||||
if formal["status"] == "OPEN":
|
||||
formal["status"] = "NOT-RUN-IMPLEMENTATION-REQUIRES-CONTROLLED-FORMAL-PROTOCOL"
|
||||
formal["formal_conditioning_run"] = False
|
||||
formal["acquisition"] = "NOT-RUN"
|
||||
formal["controls"] = "NOT-RUN"
|
||||
write_json(exp / "conditioning/gate.json", formal)
|
||||
controls = {"isolated_neutral_B": "RUN", "matched_seed_same_KC_spikes": "RUN", "D_zero": "analytic identity E=1, no dopamine update", "random_reward": "NOT-RUN-FORMAL-CONDITIONING-BLOCKED", "unrelated_DAN": "NOT-RUN-FORMAL-CONDITIONING-BLOCKED", "frozen_no_plasticity": "isolated E=1 reference replay"}
|
||||
write_json(exp / "controls/summary.json", controls)
|
||||
checkpoints_bytes = sum(p.stat().st_size for p in (exp / "efficacy-overlay").glob("E-seed-*.npz")); performance = isolated["performance"] | {"overlay_checkpoint_bytes_total": checkpoints_bytes,
|
||||
"peak_RAM_kib": int(resource.getrusage(resource.RUSAGE_SELF).ru_maxrss), "peak_VRAM_bytes": int(torch.cuda.max_memory_allocated()),
|
||||
"whole_brain_mean_post_wall_seconds": whole.get("mean_post_brain_wall_seconds"), "dense_time_by_edge_state": False}
|
||||
write_json(exp / "performance/summary.json", performance)
|
||||
candidates = {"KCMBON-SynapticEfficacy-v1": True, "PresynapticEfficacy-v1": True,
|
||||
"KCMBON-FunctionalTransfer-v2": bool(isolated["presynaptic_functional_transfer_pass"]), "MaleCNS-OlfactoryAssociativeMemory-v1-candidate": False}
|
||||
del sim, v, replay, checkpoints; integrity_report = integrity(cfg, mcfg, base_config, exp, before_hash)
|
||||
summary = {"phase": "4M", "literature_audit": "literature-audit/phase4m_literature_audit.json", "E_math": mcfg["mechanisms"], "anatomical": json.loads((exp / "anatomical-weight/summary.json").read_text()),
|
||||
"isolated": isolated, "postsynaptic": post, "anatomy_vs_efficacy": anatomy, "whole_brain": whole, "conditioning": formal, "controls": controls,
|
||||
"performance": performance, "candidates": candidates, "integrity": integrity_report, "phase4n_started": False}
|
||||
write_json(exp / "audit/final-summary.json", summary); shutil.copy2(args.config, model / "phase4m.yaml"); shutil.copy2(Path(__file__).resolve(), model / "phase4m.py"); shutil.copy2(exp / "literature-audit/phase4m_literature_audit.json", model / "literature-audit.json")
|
||||
write_json(model / "VERSION.json", {"name": mcfg["name"], "candidates": candidates, "summary_sha256": sha256(exp / "audit/final-summary.json"), "base_CSR_read_only": True})
|
||||
seal(model, model / "MANIFEST.json", model / "checksums.sha256"); seal(exp, exp / "manifests/manifest.json", exp / "manifests/checksums.sha256")
|
||||
print(json.dumps({"phase": "4M", "isolated": isolated["status"], "whole_brain": whole["status"], "conditioning": formal["status"], "integrity": integrity_report["all_passed"]}, indent=2)); sys.stdout.flush()
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
@@ -0,0 +1,59 @@
|
||||
phase: 4M
|
||||
name: KCMBON-SynapticEfficacy-v1
|
||||
frozen_phase4l_config: /data/digital-drosophila/models/male-cns-lif/olfactory-grounding-v1/phase4l.yaml
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4m-kcmbon-efficacy
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/kcmbon-efficacy-v1
|
||||
|
||||
mechanisms:
|
||||
additive:
|
||||
label: Phase4L-additive-delta-weight-baseline
|
||||
update: frozen Phase4J apply_v2, including its pre-existing 25 percent per-edge floor
|
||||
multiplicative:
|
||||
label: PresynapticEfficacy-v1
|
||||
update: E_new = max(0, E_old * (1 - eta_effective_mv / W_syn_mv_per_contact * D * eligibility))
|
||||
interpretation: engineering per-anatomical-contact efficacy conversion, not measured release probability
|
||||
bounded_multiplicative:
|
||||
label: KCMBON-SynapticEfficacy-v1
|
||||
update: E_new = max(0.75, E_old * (1 - eta_effective_mv / W_syn_mv_per_contact * D * eligibility))
|
||||
lower_bound_provenance: frozen Phase4J max_fractional_depression 0.25
|
||||
|
||||
protocol:
|
||||
pairing_counts: [1, 2, 5, 10, 20, 50, 100]
|
||||
train_seeds: [51201, 51202, 51203, 51204, 51205]
|
||||
evaluation_seeds: [51301, 51302, 51303, 51304, 51305]
|
||||
target_compartment_verbatim: "y5B'2a"
|
||||
matched_dan_type: PAM15
|
||||
unrelated_compartment_verbatim: "y2a'1"
|
||||
unrelated_dan_type: PPL103
|
||||
dan_time_ms: 90.0
|
||||
duration_ms: 120.0
|
||||
formal_training_pairings: 100
|
||||
forced_odor: ODOR_A
|
||||
neutral_odor: ODOR_B
|
||||
input_mode: physiological
|
||||
no_random_terminal_heterogeneity: true
|
||||
|
||||
gates:
|
||||
seed_majority: 3
|
||||
functional_direction: MBON spike decrease or first-spike latency increase in A; B must not show equal-or-greater change
|
||||
selection: evaluate every predeclared pairing count; formal whole-brain test uses the predeclared 100-pairing endpoint only
|
||||
no_functional_gate_on_g_alone: true
|
||||
|
||||
postsynaptic:
|
||||
branch_name: MBON-PostsynapticPlasticity-v1
|
||||
current_status: AUDIT-ONLY-NOT-RUN-COMPARTMENT-SPECIFIC-QUANTITATIVE-RULE-UNKNOWN
|
||||
never_modify_global_threshold: true
|
||||
|
||||
prohibitions:
|
||||
change_base_CSR: false
|
||||
change_edge_existence: false
|
||||
change_raw_synapse_count: false
|
||||
change_global_Wsyn: false
|
||||
change_transmitter_sign: false
|
||||
change_LIF: false
|
||||
change_APL: false
|
||||
change_eligibility_v2: false
|
||||
change_olfactory_encoder: false
|
||||
expand_plastic_mask: false
|
||||
tune_from_phase4k_5_percent: false
|
||||
formal_conditioning_before_transfer_gates: false
|
||||
@@ -0,0 +1,75 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Seal Phase 4M after run-log closure and qualify whole-brain causal language."""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import shutil
|
||||
import subprocess
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
|
||||
from phase4h_audit import sha256, write_json
|
||||
from phase4k import seal
|
||||
from phase4m import frozen_checks, frozen_hashes
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(); parser.add_argument("--config", type=Path, required=True); args = parser.parse_args(); mcfg = yaml.safe_load(args.config.read_text())
|
||||
bpath = Path(mcfg["frozen_phase4l_config"]); cfg = yaml.safe_load(bpath.read_text()); cfg["paths"]["phase4l_root"] = cfg["paths"]["experiment_root"]; cfg["paths"]["phase4l_model_root"] = str(bpath.parent)
|
||||
exp, model = Path(mcfg["experiment_root"]), Path(mcfg["model_root"]); summary_path = exp / "audit/final-summary.json"; summary = json.loads(summary_path.read_text())
|
||||
if not summary["integrity"]["all_passed"]: raise RuntimeError("Phase 4M completion integrity was not PASS")
|
||||
checks = frozen_checks(cfg)
|
||||
if frozen_hashes(cfg) != summary["integrity"]["after"]: raise RuntimeError("frozen input changed after Phase 4M run")
|
||||
whole = summary["whole_brain"]; whole["operational_gate_label_only"] = True
|
||||
whole["feedback_causally_isolated_by_this_experiment"] = False
|
||||
whole["interpretation"] = ("The prescribed WHOLE-BRAIN-FEEDBACK-MASKS-MEMORY label denotes isolated transfer PASS with whole-brain A/B specificity FAIL. "
|
||||
"It does not prove feedback is the unique cause; shared plastic edges and baseline network input may also contribute.")
|
||||
frame = pd.read_parquet(exp / "whole-brain-replay/results.parquet")
|
||||
whole["endpoint_100_KC_recruitment_change"] = {mech: {odor: float(part.KC_spike_L1.mean()) for odor, part in group.groupby("odor")}
|
||||
for mech, group in frame.groupby("mechanism")}
|
||||
|
||||
edge = pd.read_parquet(Path(cfg["paths"]["phase4l_root"]) / "eligible-edges/edge-identity.parquet")
|
||||
categories = edge.odor_eligibility_category.astype(str).to_numpy(); static = {}; seed_rows = []
|
||||
for seed in map(int, mcfg["protocol"]["train_seeds"]):
|
||||
data = np.load(exp / "efficacy-overlay" / f"E-seed-{seed}-pairings-100.npz")
|
||||
for mech, key in (("multiplicative", "E_multiplicative"), ("bounded_multiplicative", "E_bounded_multiplicative")):
|
||||
E = data[key]
|
||||
for cat in ("A-specific", "B-specific", "shared", "inactive"):
|
||||
sel = categories == cat; seed_rows.append({"seed": seed, "mechanism": mech, "edge_category": cat,
|
||||
"changed_edges": int(np.count_nonzero(E[sel] < 1 - 1e-7)), "E_reduction_mean": float((1 - E[sel]).mean()) if sel.any() else 0.0,
|
||||
"E_reduction_sum": float((1 - E[sel]).sum())})
|
||||
data.close()
|
||||
audit = pd.DataFrame(seed_rows); audit.to_parquet(exp / "whole-brain-replay/endpoint-E-identity.parquet", index=False)
|
||||
for mech, group in audit.groupby("mechanism"):
|
||||
static[mech] = {cat: {"changed_edges_mean": float(g.changed_edges.mean()), "E_reduction_mean": float(g.E_reduction_mean.mean()),
|
||||
"E_reduction_sum_mean": float(g.E_reduction_sum.mean())} for cat, g in group.groupby("edge_category")}
|
||||
whole["endpoint_100_E_identity"] = static
|
||||
whole["direct_feedback_vs_shared_edge_causal_disambiguation"] = "NOT-ESTABLISHED; no feedback edge ablation was authorized"
|
||||
write_json(exp / "whole-brain-replay/summary.json", whole); summary["whole_brain"] = whole
|
||||
|
||||
iso = summary["isolated"]; iso["functional_transfer_gate_scope"] = "PASS at one or more predeclared pairing checkpoints, not necessarily retained at 100 pairings"
|
||||
iso["endpoint_100_majority_pass"] = {mech: bool(gate["endpoint_100_specific_functional_seeds"] >= int(mcfg["gates"]["seed_majority"]))
|
||||
for mech, gate in iso["mechanism_gates"].items()}
|
||||
iso["nonmonotonic_functional_output_despite_monotonic_E_depression"] = True
|
||||
write_json(exp / "isolated-replay/summary.json", iso); summary["isolated"] = iso
|
||||
summary["integrity"]["postrun_frozen_recheck_passed"] = checks["all_passed"]
|
||||
write_json(summary_path, summary)
|
||||
version = json.loads((model / "VERSION.json").read_text()); version["summary_sha256"] = sha256(summary_path); write_json(model / "VERSION.json", version)
|
||||
write_json(exp / "audit/post-run-qualification.json", {"only_phase4m_reports_edited": True, "brain_rerun": False,
|
||||
"run_log_sealed_after_process_exit": True, "whole_brain_feedback_label_is_operational_not_causal_proof": True,
|
||||
"frozen_recheck_passed": checks["all_passed"]})
|
||||
shutil.copy2(Path(__file__).resolve(), exp / "audit/phase4m_finalize.py"); shutil.copy2(Path(__file__).resolve(), model / "phase4m_finalize.py")
|
||||
seal(model, model / "MANIFEST.json", model / "checksums.sha256")
|
||||
seal(exp, exp / "manifests/manifest.json", exp / "manifests/checksums.sha256")
|
||||
for root, checksum in ((model, "checksums.sha256"), (exp, "manifests/checksums.sha256")):
|
||||
check = subprocess.run(["sha256sum", "-c", checksum], cwd=root, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, text=True)
|
||||
if check.returncode: raise RuntimeError(f"self-checksum failure {root}: {check.stderr[-1000:]}")
|
||||
print(json.dumps({"frozen": checks["all_passed"], "model_SHA256": "PASS", "experiment_SHA256": "PASS",
|
||||
"endpoint_100_isolated_majority": iso["endpoint_100_majority_pass"]}))
|
||||
|
||||
|
||||
if __name__ == "__main__": main()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user