Audit APL PN scope and localize official synaptic contacts without changing frozen dynamics
This commit is contained in:
Digital Drosophila Archive
@@ -0,0 +1,18 @@
|
||||
# 授权追加：官方突触位置审计
|
||||
|
||||
用户已明确授权下载约 6.78 GB 的官方 syn-partners v1.0 作为新增参考数据。
|
||||
本追加取代原 CPU 静态审计中“本轮不下载 bulk”的范围限制；不改变原审计结果、原 raw 或任何模型参数。
|
||||
原静态审计的 unknown 描述“追加数据前”的证据状态，最终结果另存而不覆盖。
|
||||
|
||||
在查看位置结果前定义：
|
||||
|
||||
1. 以官方 generation 和 MD5 校验下载，另生成本地 SHA-256；不下载 syn-points 全表或 EM。
|
||||
2. 检查实际 schema 后按原字段读取，不猜列名。坐标为 MaleCNS 8 nm voxel；距离乘 0.008 转 µm。
|
||||
3. 提取全部 APL 入/出突触对、APL 所连 PN→KC 对、七个既定观察 PN↔ALLN 对。
|
||||
4. 每个 canonical body pair 的实际突触对行数必须与 base CSR 原 synapse count 一致；有重复配对或 count 差异即停止机制解释。
|
||||
5. 依据真实 primary_post 字符串报告 ROI，不由神经元名称猜 compartment。
|
||||
6. 对 APL→PN postsynaptic point 与同一 PN 的 PN→KC presynaptic points 计算最近欧氏距离分布。距离只是空间邻近，不等于同 bouton、受体作用或功能因果。
|
||||
7. 分别报告 APL→PN、PN→KC、PN↔ALLN 的 ROI，判断 point-neuron 全局电流会不会混淆不同解剖区域。
|
||||
8. 无受体、时间常数、末梢释放系数的部分继续 unknown；不据此增删边、补抑制、改变阈值或训练。
|
||||
|
||||
4090 仅追加五组已有 frozen-v2 reference regression。原五组输入、参数、初态和数值语义不变；所有新输出留在本阶段目录。
|
||||
@@ -0,0 +1,37 @@
|
||||
# Phase 4AA — APL→PN scope audit（预先声明）
|
||||
|
||||
接续封存 Phase 4Z，仅做 CPU 解剖/实现范围审计。不重新运行旧科学阶段，不使用 GPU，不影响 ComfyUI。
|
||||
不训练、不改参数/符号、不增加边、不调整 APL calibration、eligibility-v2 或 14,232 plastic mask。
|
||||
|
||||
## 选择与检验
|
||||
|
||||
- 直接从实际 annotation_type == APL、annotation_class == ALPN / Kenyon_Cell 识别；空 class 不靠名称猜测。
|
||||
- 审计所有 APL 入/出边，不只选择七个观察 PN。七个 PN 来自封存 Phase 4Z 的全部真实目标 KC 入源。
|
||||
- 从 CSR 独立抽取，再流式过滤原始 Feather，逐 body pair / 原始 count 严格核对。
|
||||
- 区分 frozen fast-current out-mask、KC-only graded output、KC/PN-only graded input；输入被 graded 计算忽略不等于原图边被删除。
|
||||
- 核对 PN↔APL 双向边、PN→KC 与 PN→ALLN 出边；这只证明解剖路径，不宣称空间/生理因果。
|
||||
- 检查真实 schema 是否含接触位置、ROI、靶位点受体；SWC centerline 不等同 synaptic partner 坐标。
|
||||
- 保存 Phase 4Z 所用运行时代码相关段落与 SHA-256，核实覆盖数量完全一致。
|
||||
|
||||
## 未知与停止条件
|
||||
|
||||
缺失的接触 ROI、末梢位置、受体和作用系数均记 unknown。
|
||||
有配对边和论文支持不等于这只 MaleCNS 中某条连接已经定位到 calyx bouton。
|
||||
不得据本审计自动恢复 APL→PN whole-cell negative current 或扩大 graded target matrix。
|
||||
官方 syn-partners/syn-points 只获取 HEAD 元数据；不自动下载数 GB 数据，不请求 EM。
|
||||
若继续定位需 neuPrint 授权或新的官方附加数据，输出具体需求，不猜测补齐。
|
||||
|
||||
## 通过的含义
|
||||
|
||||
audit PASS = 原始边核对通过、覆盖重建与 Phase 4Z 相符、旧文件哈希不变、未知被保留。
|
||||
不等于 learning PASS，不产生新的正式模型候选。全项目顶层 gate 保持 5/10。
|
||||
复现用新的 experiment_root；禁止覆盖任何已经封存的输出目录。
|
||||
|
||||
## 证据
|
||||
|
||||
- 官方 MaleCNS 下载定义：https://male-cns.janelia.org/download/
|
||||
- APL 局部化：https://doi.org/10.7554/eLife.56954
|
||||
- PN bouton/KC dendrite 研究：https://doi.org/10.7554/eLife.74172
|
||||
- Google Research 项目说明：https://www.research.google/blog/a-connectomics-milestone-mapping-the-complete-male-fruit-fly-brain/
|
||||
|
||||
Google/Janelia 的解剖数据与论文用于约束来源和机制；不能替代受体/电导测量或学习任务验证。
|
||||
@@ -0,0 +1,111 @@
|
||||
# Phase 4AA — APL→PN 作用范围与官方突触定位审计
|
||||
|
||||
正式结论：**PASS-APL-PN-ANATOMICAL-LOCALIZATION / NO-DYNAMICS-CANDIDATE**。
|
||||
|
||||
本轮从 Phase 4Z 接续，先完成只读 CPU 范围审计；随后用户明确授权下载官方 syn-partners 并继续定位，也确认 ComfyUI 空闲以便运行 frozen regression。
|
||||
原始静态审计 `audit/results.json` 中的空间 unknown/blocked 是追加数据前的状态，保留不覆盖；最终状态以 `audit/final-summary.json` 和本报告为准。
|
||||
本轮无训练、无参数修改、无新的正式动力学候选。顶层验收仍为 5/10（50%），不是生物学复现比例。
|
||||
|
||||
## 核心新证据
|
||||
|
||||
以实际 annotation 与原始 CSR 识别 2 个 APL、4,064 个 KC、686 个 ALPN，不通过 neurotransmitter 推断 DAN 身份。
|
||||
所有 166,700 个 neurons、25,582,938 edges、124,177,617 anatomical contacts 不变。
|
||||
|
||||
| 已有 APL 实现范围 | 边数 | 原始 contacts |
|
||||
|---|---:|---:|
|
||||
| APL 全部出边 | 5,845 | 209,912 |
|
||||
| KC-only graded output 覆盖 | 4,633 | 196,200 |
|
||||
| 非 KC 出边，没有 graded 替代 | 1,212 | 13,712 |
|
||||
| 上述缺口中的 APL→PN | 268 | 4,600 |
|
||||
| APL 全部入边 | 6,185 | 234,025 |
|
||||
| 不被 KC/PN-only graded state 使用的入边 | 1,202 | 18,257 |
|
||||
|
||||
268 条 APL→PN 边涉及 259 个 PN；247 对同时存在 PN→APL 反向连接。
|
||||
“graded state 未使用”不等于原图入边被删除：普通 LIF APL 仍有这些入边，但其快速出边被既有 overlay 置零。基于 KC/PN 计算的独立 graded state 没有使用其余来源。
|
||||
这属于既有模型作用范围，不能把每个未覆盖来源都自动补入；需要额外生理依据。
|
||||
|
||||
## 官方附加数据与精确核对
|
||||
|
||||
用户授权下载的 `syn-partners-male-cns-v1.0-minconf-0.5.feather`：
|
||||
|
||||
- 6,777,179,098 bytes；官方 generation `1780494942562468`。
|
||||
- 官方 MD5：`58efcf712f8c4d4de5f2ad51e97def76`，验证通过。
|
||||
- SHA-256：`959d8ef4173b35382a3e6acfaf5167c795b6d10b877572d146af04e1b487bc07`。
|
||||
- 流式读取全部 311,833,243 条配对记录；实际字段和 dtype 已保存。
|
||||
- 来源：[MaleCNS 官方下载页](https://male-cns.janelia.org/download/)，许可证 CC BY 4.0。新增文件保存在本阶段 external-resources，不写 raw，不下载 EM。
|
||||
|
||||
| 提取范围 | 配对记录总数 | canonical 配对记录 | canonical neuron pairs |
|
||||
|---|---:|---:|---:|
|
||||
| APL 所有入/出接触 | 500,348 | 443,915 | 12,027 |
|
||||
| APL 所连 PN→KC | 385,229 | 385,229 | 21,978 |
|
||||
| 七个重点 PN→ALLN | 22,599 | 22,599 | 823 |
|
||||
| ALLN→七个重点 PN | 19,024 | 19,024 | 886 |
|
||||
|
||||
每组同时检查重复配对、完整 pair 集合及逐边 count：均与原始 CSR 完全一致。不存在只比较出现过的边而漏掉整条缺失边的情况。
|
||||
APL 配对中的非 canonical 部分保留来源记录，不加入 neuron-only graph。
|
||||
|
||||
## 真实 ROI 定位
|
||||
|
||||
4,600 个 APL→PN 接触都取得真实 pre/post 坐标：
|
||||
|
||||
- `CA(L)` 2,063；`CA(R)` 1,757；合计 3,820（83.04%）。
|
||||
- `CentralBrain-unspecified` 731（15.89%）：坐标已知，但官方 ROI 未细分，绝不按邻近位置重标为 calyx。
|
||||
- 其余实际标注合计 49，分布于 LH、PED、SCL、SLP、a'L；不丢弃非 calyx 结果。
|
||||
|
||||
CA 对应 calyx，官方作者 [Cell Type Explorer](https://reiserlab.github.io/celltype-explorer-drosophila-male-cns/types/KCg-d_R.html) 提供标签释义。
|
||||
七个重点 PN：
|
||||
|
||||
| neuron_idx / body_id | type | APL→PN contacts | CA(R) | ROI 未细分 |
|
||||
|---|---|---:|---:|---:|
|
||||
| 30 / 10039 | VL2a_adPN | 50 | 47 | 3 |
|
||||
| 179 / 10198 | VA7l_adPN | 7 | 7 | 0 |
|
||||
| 484 / 10522 | VL2p_adPN | 50 | 49 | 1 |
|
||||
| 1644 / 11761 | VP1d+VP4_l2PN1 | 78 | 70 | 8 |
|
||||
| 6065 / 16610 | VA7m_lPN | 14 | 13 | 1 |
|
||||
| 6842 / 17484 | DC2_adPN | 13 | 13 | 0 |
|
||||
| 133666 / 527577 | VA6_adPN | 35 | 29 | 6 |
|
||||
|
||||
合计 228/247（92.31%）在 CA(R)，19 个未细分。
|
||||
对同一 PN，APL postsynaptic sites 到最近 PN→KC presynaptic site 的距离，中位数范围为 0.545–0.861 µm；坐标按官方 8 nm voxel 转换。
|
||||
距离不是同 bouton 的证明，更不是受体或功能因果证明。保留全部分布和远距离样本，不设“最佳阈值”筛掉例外。
|
||||
259 个 APL 靶 PN 中，15 个在本 canonical 图中没有 PN→KC 出边，不能一概当作同一功能入口。
|
||||
|
||||
七个 PN→ALLN 的 22,599 个接触全部位于 AL(L/R)；ALLN→七个 PN 的 19,024 个接触中，19,021 位于 AL(L/R)，另有 1 个 LAL(R)、2 个未细分。
|
||||
同一 PN 的 calyx 相关接触与触角叶局部回路接触必须区分，不能用单一 whole-neuron 抑制项自动代表两种位置。
|
||||
|
||||
## 如何解释，不做什么
|
||||
|
||||
已有原始研究支持 APL 的局部抑制与 PN bouton/KC dendrite 相互作用，但不提供当前 MaleCNS 每条边的受体或强度。
|
||||
见 [Prisco et al., eLife](https://doi.org/10.7554/eLife.74172)、[Amin et al., eLife](https://doi.org/10.7554/eLife.56954)。
|
||||
本轮数据加强了“空间作用范围缺失”这一机制假设，**没有证明补上它就能修复 KC 持续放电或刺激特异性记忆**。
|
||||
仍未知：具体 bouton 身份、GABA 受体、末梢释放变化、时间常数、量化作用系数及 whole-brain 干预效果。
|
||||
|
||||
下一步应先设计独立的 compartment/terminal-specific input hypothesis 和预声明对照，不直接改变 frozen baseline；对未知系数不能用“学习效果更好”反向调参。
|
||||
优先处理 upstream encoding / 作用范围，而不是继续改 dopamine eta、扩大 plastic mask 或调 KC/MBON 阈值。
|
||||
|
||||
## 回归、完整性与资源
|
||||
|
||||
独立新目录重跑 deterministic-v2 的五组冻结试次（各一次）：
|
||||
|
||||
| trial | biological time | wall time | 事件 / counts / V / g |
|
||||
|---|---:|---:|---|
|
||||
| auditory 10 Hz | 1 s | 2.690 s | 全部精确一致 |
|
||||
| auditory 100 Hz | 1 s | 4.915 s | 全部精确一致 |
|
||||
| olfactory 50 Hz | 1 s | 4.961 s | 全部精确一致 |
|
||||
| mechanosensory 50 Hz | 1 s | 4.982 s | 全部精确一致 |
|
||||
| gustatory 50 Hz | 1 s | 4.977 s | 全部精确一致 |
|
||||
|
||||
均无 NaN/Inf，V/g 最大误差 0；4090 本进程 peak allocated VRAM 为 285,994,496 bytes（约 272.75 MiB），不包括 ComfyUI 已驻留显存。
|
||||
本轮不声称重做了 Phase 4P 的 20 次重复实验；旧 v1/v2 reference 均没有覆盖。
|
||||
GPU 编译加回归约 85.55 s；CPU 结构审计含两次旧文件哈希扫描约 86.92 s；下载加校验约 375.66 s；空间提取/分析约 37.98 s。
|
||||
230,239 个旧科学文件的 SHA-256 复核通过。初始 Git payload 的 681 项校验也通过；新阶段另做 manifest。
|
||||
未停止或卸载 ComfyUI，未更新任何环境包。
|
||||
|
||||
## 异常与处理记录
|
||||
|
||||
1. 初次建立实验目录遇到写权限限制，在新实验目录单独配置 owner；没有递归修改旧目录。
|
||||
2. GPU 首次启动 PATH 未包含原 venv 的 Ninja，未开始模拟；失败目录保留。仅补启动 PATH 后，在另一个新目录完成 5/5 回归。
|
||||
3. 下载首次校验选项分隔符错误，下载未开始；修正为 aria2 要求的语法后完成官方校验，没有坏文件被当作通过。
|
||||
4. 旧 NumPy→Torch 只读数组警告保留，未修改冻结加载器；文件哈希与数值回归通过。
|
||||
|
||||
本轮到此结束，不生成 learning candidate，不进入游戏、RL、视觉或动作 decoder。
|
||||
@@ -0,0 +1,245 @@
|
||||
"""CPU-only, read-only APL/PN implementation scope audit. No scientific modules imported."""
|
||||
import argparse
|
||||
import ast
|
||||
import concurrent.futures as cf
|
||||
from datetime import datetime, timezone
|
||||
import gzip
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import time
|
||||
import urllib.request
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow as pa
|
||||
import pyarrow.compute as pc
|
||||
import yaml
|
||||
|
||||
|
||||
def digest(path):
|
||||
h=hashlib.sha256()
|
||||
with path.open('rb') as f:
|
||||
for b in iter(lambda:f.read(8<<20),b''):h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def save(path,value):
|
||||
path.parent.mkdir(parents=True,exist_ok=True)
|
||||
with path.open('x',encoding='utf-8') as f:
|
||||
json.dump(value,f,ensure_ascii=False,indent=2,sort_keys=True,allow_nan=False)
|
||||
f.write('\n')
|
||||
|
||||
|
||||
def immutable_check(root,snapshot,workers):
|
||||
summary=json.loads((snapshot/'summary.json').read_text())
|
||||
assert digest(snapshot/'files.jsonl.gz')==summary['files_jsonl_gz_sha256']
|
||||
with gzip.open(snapshot/'files.jsonl.gz','rt') as f:rows=[json.loads(x) for x in f]
|
||||
def check(r):
|
||||
p=root/r['path']
|
||||
if r['kind']=='symlink':
|
||||
ok=p.is_symlink() and str(p.readlink())==r['target']
|
||||
else:
|
||||
ok=p.is_file() and p.stat().st_size==r['bytes'] and digest(p)==r['sha256']
|
||||
return None if ok else r['path']
|
||||
with cf.ThreadPoolExecutor(max_workers=workers) as pool:bad=[x for x in pool.map(check,rows) if x]
|
||||
return {'pass':not bad,'checked':len(rows),'mismatches':bad,'reference_sha256':summary['files_jsonl_gz_sha256'],
|
||||
'exclusions':summary['excluded_directory_names'],'new_stage_additions_allowed':True,
|
||||
'meaning':'All original listed bytes checked; only newly named stage outputs are additions.'}
|
||||
|
||||
|
||||
def population(meta):
|
||||
out=meta.annotation_class.fillna('UNKNOWN_CLASS').astype(str).to_numpy(object)
|
||||
for typ in ('APL','DPM'):out[meta.annotation_type.eq(typ).to_numpy()]=typ
|
||||
return out
|
||||
|
||||
|
||||
def edge_frame(ids,sources,targets,weights,meta,pops):
|
||||
src=sources[ids];dst=targets[ids]
|
||||
f=pd.DataFrame({'edge_id':ids,'source_neuron_idx':src,'target_neuron_idx':dst,'raw_synapse_count':weights[ids]})
|
||||
for pref,arr in [('source',src),('target',dst)]:
|
||||
f[pref+'_population']=pops[arr]
|
||||
for col in ('body_id','annotation_class','annotation_type','annotation_instance','annotation_somaSide','neurotransmitter','nt_ground_truth','annotation_receptorType'):
|
||||
f[pref+'_'+col]=meta[col].to_numpy()[arr]
|
||||
return f
|
||||
|
||||
|
||||
def raw_incident(path,bodyids):
|
||||
tables=[]
|
||||
with pa.memory_map(str(path),'r') as f:
|
||||
reader=pa.ipc.open_file(f)
|
||||
assert reader.schema.names==['body_pre','body_post','weight'],str(reader.schema)
|
||||
ids=pa.array(bodyids.astype(np.int64))
|
||||
for i in range(reader.num_record_batches):
|
||||
batch=reader.get_batch(i)
|
||||
keep=pc.or_(pc.is_in(batch.column(0),value_set=ids),pc.is_in(batch.column(1),value_set=ids))
|
||||
selected=batch.filter(keep)
|
||||
if len(selected):tables.append(pa.Table.from_batches([selected]))
|
||||
return pa.concat_tables(tables).to_pandas()
|
||||
|
||||
|
||||
def official_head(timeout):
|
||||
base='https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/'
|
||||
out=[]
|
||||
for name in ('syn-partners-male-cns-v1.0-minconf-0.5.feather','syn-points-male-cns-v1.0-minconf-0.5.feather'):
|
||||
row={'url':base+name,'retrieved_utc':datetime.now(timezone.utc).isoformat(),'method':'HEAD','body_downloaded':False}
|
||||
try:
|
||||
with urllib.request.urlopen(urllib.request.Request(base+name,method='HEAD'),timeout=timeout) as r:
|
||||
row.update(status=r.status,content_length=int(r.headers['Content-Length']),etag=r.headers.get('ETag'),
|
||||
generation=r.headers.get('x-goog-generation'),upstream_hash_header=r.headers.get('x-goog-hash'),
|
||||
last_modified=r.headers.get('Last-Modified'),accept_ranges=r.headers.get('Accept-Ranges'))
|
||||
except Exception as e:row.update(status='UNAVAILABLE',error=type(e).__name__+': '+str(e))
|
||||
out.append(row)
|
||||
return out
|
||||
|
||||
|
||||
def seal(root):
|
||||
paths=sorted(p for p in root.rglob('*') if p.is_file() and p.name not in ('manifest.json','checksums.sha256'))
|
||||
rows=[{'path':p.relative_to(root).as_posix(),'bytes':p.stat().st_size,'sha256':digest(p)} for p in paths]
|
||||
save(root/'manifest.json',{'files':rows,'created_utc':datetime.now(timezone.utc).isoformat()})
|
||||
with (root/'checksums.sha256').open('x') as f:
|
||||
for r in rows:f.write(r['sha256']+' '+r['path']+'\n')
|
||||
return len(rows)
|
||||
|
||||
|
||||
def run(cfg):
|
||||
pa.set_cpu_count(4);pa.set_io_thread_count(4)
|
||||
start=time.perf_counter();root=Path(cfg['root']);exp=Path(cfg['experiment_root']);model=Path(cfg['model_root'])
|
||||
assert exp.is_relative_to(root/'experiments') and model.is_relative_to(root/'models/male-cns-lif')
|
||||
if exp.exists() and any(exp.iterdir()):raise RuntimeError('Refusing nonempty experiment directory')
|
||||
exp.mkdir(parents=True,exist_ok=True)
|
||||
for d in ('protocol','audit','connectivity','metadata','spatial','external-resources','regression','performance'):(exp/d).mkdir()
|
||||
save(exp/'protocol/preregistered.json',{'config':cfg,'source_hashes':{p.name:digest(p) for p in model.iterdir() if p.is_file()},
|
||||
'created_utc':datetime.now(timezone.utc).isoformat(),'before_analysis':True})
|
||||
pre=immutable_check(root,root/cfg['snapshot'],cfg['scope']['snapshot_workers']);save(exp/'regression/preflight.json',pre)
|
||||
assert pre['pass'],'original bytes differ'
|
||||
canonical=root/cfg['canonical'];prior=root/cfg['prior'];expect=cfg['expected']
|
||||
meta=pd.read_parquet(canonical/'neuron-index/unified-metadata.parquet').sort_values('neuron_idx').reset_index(drop=True)
|
||||
ptr=np.load(canonical/'gpu/csr-indptr.int64.npy',mmap_mode='r');dst=np.load(canonical/'gpu/csr-indices.int32.npy',mmap_mode='r');w=np.load(canonical/'gpu/csr-weights.int64.npy',mmap_mode='r')
|
||||
assert len(meta)==expect['neurons'] and np.array_equal(meta.neuron_idx,np.arange(len(meta)))
|
||||
assert str(meta.neuron_idx.dtype)=='int32' and str(meta.body_id.dtype)=='uint64' and meta.body_id.is_unique
|
||||
assert len(ptr)==len(meta)+1 and ptr[0]==0 and ptr[-1]==len(w)==len(dst)==expect['edges']
|
||||
assert ptr.dtype==np.int64 and dst.dtype==np.int32 and w.dtype==np.int64
|
||||
assert int(w.sum())==expect['synapse_count_sum'] and np.all(np.diff(ptr)>=0) and w.min()>0
|
||||
src=np.repeat(np.arange(len(meta),dtype=np.int32),np.diff(ptr));pops=population(meta)
|
||||
apl=np.flatnonzero(meta.annotation_type.eq('APL'));kc=np.flatnonzero(meta.annotation_class.eq('Kenyon_Cell'));pn=np.flatnonzero(meta.annotation_class.eq('ALPN'))
|
||||
assert len(apl)==expect['APL'] and len(kc)==expect['KC']
|
||||
oi=np.flatnonzero(np.isin(src,apl));ii=np.flatnonzero(np.isin(dst,apl))
|
||||
outgoing=edge_frame(oi,src,dst,w,meta,pops);incoming=edge_frame(ii,src,dst,w,meta,pops)
|
||||
outgoing['fast_current_masked']=True;outgoing['graded_output_covered']=np.isin(dst[oi],kc)
|
||||
incoming['graded_input_used']=np.isin(src[ii],np.r_[kc,pn])
|
||||
# This describes the graded-state input only. Fast synapses into APL remain in CSR/runtime.
|
||||
watched=json.loads((prior/'protocol/watched.json').read_text())['all_incoming_ALPNs']
|
||||
focus=outgoing[outgoing.target_neuron_idx.isin(watched)].copy()
|
||||
old=pd.read_parquet(prior/'audit/APL-target-coverage.parquet')
|
||||
fields=['edge_id','source_neuron_idx','target_neuron_idx','raw_synapse_count']
|
||||
assert np.array_equal(outgoing[fields].sort_values('edge_id').to_numpy(np.int64),old[fields].sort_values('edge_id').to_numpy(np.int64))
|
||||
nonkc=outgoing[~outgoing.graded_output_covered]
|
||||
assert (len(outgoing),int(outgoing.raw_synapse_count.sum()),len(nonkc),int(nonkc.raw_synapse_count.sum()),len(focus),int(focus.raw_synapse_count.sum()))==(expect['APL_outgoing_edges'],expect['APL_outgoing_contacts'],expect['non_KC_edges'],expect['non_KC_contacts'],expect['watched_PN_edges'],expect['watched_PN_contacts'])
|
||||
print('CSR and Phase4Z coverage exact',flush=True)
|
||||
raw_path=root/'datasets/male-cns-v1.0/raw/connectivity/connectome-weights-male-cns-v1.0-minconf-0.5.feather'
|
||||
raw=raw_incident(raw_path,meta.iloc[apl].body_id.to_numpy())
|
||||
valid=raw.body_pre.isin(meta.body_id)&raw.body_post.isin(meta.body_id)
|
||||
actual=raw.loc[valid].sort_values(['body_pre','body_post']).reset_index(drop=True)
|
||||
assert not actual.duplicated(['body_pre','body_post']).any()
|
||||
incident=pd.concat([outgoing,incoming]).drop_duplicates('edge_id')
|
||||
expected=incident[['source_body_id','target_body_id','raw_synapse_count']].rename(columns={'source_body_id':'body_pre','target_body_id':'body_post','raw_synapse_count':'weight'}).sort_values(['body_pre','body_post']).reset_index(drop=True)
|
||||
assert np.array_equal(actual.to_numpy(np.int64),expected.to_numpy(np.int64))
|
||||
raw.to_parquet(exp/'connectivity/raw-APL-incident-including-noncanonical.parquet',index=False)
|
||||
print('Raw Feather APL incident edges exact',len(actual),flush=True)
|
||||
outgoing.to_parquet(exp/'connectivity/APL-outgoing.parquet',index=False);incoming.to_parquet(exp/'connectivity/APL-incoming.parquet',index=False)
|
||||
for label,frame,col in [('output',outgoing,'target_population'),('input',incoming,'source_population')]:
|
||||
grouped=frame.groupby(col,dropna=False).agg(edges=('edge_id','size'),contacts=('raw_synapse_count','sum')).reset_index()
|
||||
grouped.to_parquet(exp/'connectivity'/f'APL-{label}-population-summary.parquet',index=False)
|
||||
grouped.to_csv(exp/'connectivity'/f'APL-{label}-population-summary.csv',index=False)
|
||||
ignored=incoming[~incoming.graded_input_used]
|
||||
ignored.to_parquet(exp/'connectivity/graded-input-not-used.parquet',index=False)
|
||||
pairs=incoming[['source_neuron_idx','target_neuron_idx','raw_synapse_count','edge_id']].rename(columns={'source_neuron_idx':'target_neuron_idx','target_neuron_idx':'source_neuron_idx','raw_synapse_count':'reverse_contacts','edge_id':'reverse_edge_id'})
|
||||
pn_out=outgoing[outgoing.target_neuron_idx.isin(pn)].merge(pairs,on=['source_neuron_idx','target_neuron_idx'],how='left',validate='one_to_one')
|
||||
pn_out['has_PN_to_APL']=pn_out.reverse_edge_id.notna()
|
||||
pn_out['watched_in_Phase4Z']=pn_out.target_neuron_idx.isin(watched)
|
||||
pn_out.to_parquet(exp/'connectivity/APL-PN-reciprocity.parquet',index=False)
|
||||
pn_out.to_csv(exp/'connectivity/APL-PN-reciprocity.csv',index=False)
|
||||
downstream=[]
|
||||
for node in pn_out.target_neuron_idx.unique():
|
||||
ids=np.arange(ptr[node],ptr[node+1]);cl=pops[dst[ids]]
|
||||
for pop in ('Kenyon_Cell','ALLN','APL'):
|
||||
selected=ids[cl==pop]
|
||||
downstream.append({'PN_neuron_idx':int(node),'PN_body_id':int(meta.iloc[node].body_id),'PN_type':meta.iloc[node].annotation_type,
|
||||
'target_population':pop,'edges':len(selected),'contacts':int(w[selected].sum())})
|
||||
pd.DataFrame(downstream).to_parquet(exp/'connectivity/APL-target-PN-downstream.parquet',index=False)
|
||||
focus=pn_out[pn_out.watched_in_Phase4Z].copy()
|
||||
focus[['source_neuron_idx','source_body_id','source_annotation_instance','target_neuron_idx','target_body_id','target_annotation_type','raw_synapse_count','reverse_contacts','reverse_edge_id']].to_csv(exp/'connectivity/watched-PN-reciprocity.csv',index=False)
|
||||
# Reconstruct the exact frozen matrices on CPU; do not simulate or apply any weights.
|
||||
inp=np.zeros((len(apl),len(meta)),np.float32);output=np.zeros((len(apl),len(kc)),np.float32)
|
||||
kmap={int(n):i for i,n in enumerate(kc)};amap={int(n):i for i,n in enumerate(apl)}
|
||||
for r in incoming[incoming.graded_input_used].itertuples():inp[amap[r.target_neuron_idx],r.source_neuron_idx]+=r.raw_synapse_count
|
||||
for r in outgoing[outgoing.graded_output_covered].itertuples():output[amap[r.source_neuron_idx],kmap[r.target_neuron_idx]]+=r.raw_synapse_count
|
||||
input_scale=float(inp[inp>0].sum()/np.count_nonzero(np.any(inp>0,axis=0)));output_scale=float(output[output>0].mean())
|
||||
coverage={'input_scale':input_scale,'output_scale':output_scale,'input_matrix_shape':list(inp.shape),'output_matrix_shape':list(output.shape),
|
||||
'KC_targets_with_graded_input':int(np.count_nonzero(np.any(output>0,axis=0))),
|
||||
'normalized_input_matrix_sha256':hashlib.sha256((inp/input_scale).tobytes()).hexdigest(),
|
||||
'normalized_output_matrix_sha256':hashlib.sha256((output/output_scale).tobytes()).hexdigest()}
|
||||
save(exp/'audit/frozen-matrix-reconstruction.json',coverage)
|
||||
frozen=json.loads((prior/'audit/APL-target-coverage.json').read_text());source=Path(frozen['source'])
|
||||
assert digest(source)==frozen['source_sha256']
|
||||
body=source.read_text();tree=ast.parse(body)
|
||||
cls=next(n for n in tree.body if isinstance(n,ast.ClassDef) and n.name=='DeterministicAPLSim')
|
||||
with (exp/'audit/frozen-APL-class-source.txt').open('x') as f:f.write(ast.get_source_segment(body,cls)+'\n')
|
||||
save(exp/'audit/frozen-implementation-provenance.json',{'path':str(source),'sha256':digest(source),'same_as_Phase4Z':True,'models_imported':False})
|
||||
ids=np.unique(np.r_[apl,pn_out.target_neuron_idx]);meta.iloc[ids].to_parquet(exp/'metadata/APL-and-target-PNs.parquet',index=False)
|
||||
spatial=pn_out[['edge_id','source_neuron_idx','source_body_id','target_neuron_idx','target_body_id','target_annotation_type','raw_synapse_count']].copy()
|
||||
for col in ('synapse_location','synapse_ROI','PN_subcellular_target','postsynaptic_GABA_receptor','effective_inhibition_coefficient'):spatial[col]='unknown'
|
||||
spatial['reason']='Local body-pair counts and SWC centerlines do not contain synapse partner locations or target receptor measurements'
|
||||
spatial.to_parquet(exp/'spatial/APL-PN-unknowns.parquet',index=False)
|
||||
swc=[]
|
||||
for node in np.r_[apl,watched]:
|
||||
p=Path(meta.iloc[node].swc_path)
|
||||
a=np.loadtxt(p,comments='#',ndmin=2)
|
||||
assert a.shape[1]==7 and np.all(np.isfinite(a))
|
||||
swc.append({'neuron_idx':int(node),'body_id':int(meta.iloc[node].body_id),'path':str(p),'sha256':digest(p),
|
||||
'nodes':len(a),'columns':int(a.shape[1]),'synaptic_partner_identity_in_standard_SWC':False,
|
||||
'coordinates_used_to_infer_synapse_locations':False})
|
||||
save(exp/'spatial/watched-SWC-audit.json',swc)
|
||||
schemas=[]
|
||||
for p in sorted((root/'datasets/male-cns-v1.0/raw').glob('*/*.feather')):
|
||||
with pa.memory_map(str(p),'r') as f:sc=pa.ipc.open_file(f).schema
|
||||
schemas.append({'path':str(p),'sha256':digest(p),'fields':[{'name':x.name,'dtype':str(x.type)} for x in sc]})
|
||||
save(exp/'spatial/local-schemas.json',schemas)
|
||||
save(exp/'external-resources/official-data-access.json',official_head(cfg['scope']['official_network_timeout_s']))
|
||||
evidence={
|
||||
'official_definition':{'url':'https://male-cns.janelia.org/download/','dataset':'male-cns:v1.0','license':'CC-BY-4.0',
|
||||
'license_url':'https://creativecommons.org/licenses/by/4.0/','observation':'Official page specifies paired synapse coordinates/primary_post in syn-partners; local flat weights contain none.',
|
||||
'required_subset':'All outgoing APL partner records; exact body IDs from metadata; minimum confidence 0.5; include pre/post x/y/z, body IDs and primary_post.',
|
||||
'no_EM_needed':True,'coordinate_units':'synapse coordinates and original SWC in 8nm voxels; precomputed skeletons have different units'},
|
||||
'primary_physiology':[
|
||||
{'doi':'10.7554/eLife.74172','url':'https://elifesciences.org/articles/74172','claim':'Local APL inhibition at PN boutons and KC dendritic claws was studied; no edge-specific MaleCNS coefficient follows.'},
|
||||
{'doi':'10.7554/eLife.56954','url':'https://pmc.ncbi.nlm.nih.gov/articles/PMC7541083/','claim':'APL activity/inhibition can be spatially localized; global point-neuron inhibition is a modeling assumption.'}],
|
||||
'google_research':{'url':'https://www.research.google/blog/a-connectomics-milestone-mapping-the-complete-male-fruit-fly-brain/',
|
||||
'role':'Primary project provenance, not evidence of associative-learning performance'},
|
||||
'access_choices':['Authorized neuPrint male-cns:v1.0 targeted query, requiring user-configured API token',
|
||||
'Explicitly approved official syn-partners bulk download into a NEW external-reference directory'],
|
||||
'not_attempted':['bulk synapse download','EM download','credential discovery','unauthenticated query bypass','new dynamics','GPU regression']}
|
||||
save(exp/'external-resources/evidence-and-limitations.json',evidence)
|
||||
results={'phase':'4AA','status':'PASS-STATIC-SCOPE-AUDIT / BLOCKED-SPATIAL-PHYSIOLOGY','candidate':False,
|
||||
'neurons':len(meta),'graph_edges':len(w),'synapse_count_sum':int(w.sum()),'APL_neurons':len(apl),'KC_neurons':len(kc),'ALPN_neurons':len(pn),
|
||||
'APL_outgoing_edges':len(outgoing),'APL_outgoing_contacts':int(outgoing.raw_synapse_count.sum()),
|
||||
'non_KC_uncovered_edges':len(nonkc),'non_KC_uncovered_contacts':int(nonkc.raw_synapse_count.sum()),'non_KC_uncovered_targets':int(nonkc.target_neuron_idx.nunique()),
|
||||
'APL_to_PN_edges':len(pn_out),'APL_to_PN_contacts':int(pn_out.raw_synapse_count.sum()),'APL_target_PNs':int(pn_out.target_neuron_idx.nunique()),
|
||||
'APL_PN_pairs_with_reverse_edge':int(pn_out.has_PN_to_APL.sum()),'watched_PN_contacts':int(focus.raw_synapse_count.sum()),
|
||||
'APL_incoming_edges':len(incoming),'APL_incoming_contacts':int(incoming.raw_synapse_count.sum()),
|
||||
'incoming_edges_not_used_by_graded_state':len(ignored),'incoming_contacts_not_used_by_graded_state':int(ignored.raw_synapse_count.sum()),
|
||||
'raw_APL_incident_canonical_edges_checked':len(actual),'raw_APL_incident_noncanonical_rows':int((~valid).sum()),
|
||||
'raw_to_CSR_exact':True,'Phase4Z_static_coverage_exact':True,'APL_PN_pairs_with_confirmed_local_synapse_ROI':0,
|
||||
'APL_PN_pairs_with_confirmed_subcellular_target':0,'whole_cell_inhibition_repair_authorized_by_evidence':False,
|
||||
'new_simulation':False,'new_scientific_regression_run':False,'old_reference_files_rehashed':True,
|
||||
'weights_changed':False,'gpu_used':False,'ComfyUI_models_unloaded':False,'top_level_gates_passed':5,'top_level_gates_total':10}
|
||||
post=immutable_check(root,root/cfg['snapshot'],cfg['scope']['snapshot_workers']);save(exp/'regression/postflight.json',post);assert post['pass']
|
||||
results['frozen_integrity_pass']=post['pass'];save(exp/'audit/results.json',results)
|
||||
save(exp/'performance/resources.json',{'wall_seconds':time.perf_counter()-start,'max_RSS_KiB':resource.getrusage(resource.RUSAGE_SELF).ru_maxrss,'gpu_used':False,'includes_two_full_integrity_scans':True})
|
||||
print(json.dumps(results,indent=2),flush=True)
|
||||
|
||||
|
||||
if __name__=='__main__':
|
||||
ap=argparse.ArgumentParser();ap.add_argument('--config',type=Path,required=True);args=ap.parse_args()
|
||||
run(yaml.safe_load(args.config.read_text()))
|
||||
@@ -0,0 +1,31 @@
|
||||
phase: 4AA
|
||||
name: APL-PN-SCOPE-AUDIT
|
||||
root: /data/digital-drosophila
|
||||
model_root: /data/digital-drosophila/models/male-cns-lif/apl-pn-scope-audit-v1
|
||||
experiment_root: /data/digital-drosophila/experiments/phase4aa-apl-pn-scope
|
||||
canonical: datasets/male-cns-v1.0/processed/neuron-only
|
||||
prior: experiments/phase4z-pn-feedback-audit
|
||||
snapshot: registry/snapshots/bootstrap-20260929
|
||||
expected:
|
||||
neurons: 166700
|
||||
edges: 25582938
|
||||
synapse_count_sum: 124177617
|
||||
APL: 2
|
||||
KC: 4064
|
||||
APL_outgoing_edges: 5845
|
||||
APL_outgoing_contacts: 209912
|
||||
non_KC_edges: 1212
|
||||
non_KC_contacts: 13712
|
||||
watched_PN_edges: 7
|
||||
watched_PN_contacts: 247
|
||||
scope:
|
||||
cpu_only: true
|
||||
new_simulation: false
|
||||
conditioning: false
|
||||
parameter_changes: false
|
||||
sign_changes: false
|
||||
learned_candidate: false
|
||||
bulk_data_download: false
|
||||
new_spatial_positions_inferred_from_skeletons: false
|
||||
snapshot_workers: 4
|
||||
official_network_timeout_s: 30
|
||||
@@ -0,0 +1,12 @@
|
||||
b87a601d869f0a4cf7f88ccb2b7f5592b22da25d11d3e5bfb1be4bba05c079c0 LOCALIZATION-PROTOCOL.md
|
||||
e88f765be0fd91178eed1d862b1de8d8b307ba5ec3cc0ae0d0d93202f264acce PROTOCOL.md
|
||||
fa0bfd5a163a436e061c94e37cf6fe84ca75fdfa94d6f20ce85eda5058ca9ce7 REPORT.md
|
||||
4ef071ba88f4e1cd78f068be58414dd7f4099dfac69c68266c4f46e175eab9fb audit.py
|
||||
e0171bbd7dc7e693268eac00152e560e841f885c7d524c74d51fda7f7fbadbbf audit.yaml
|
||||
23db504ac8f31c62a434b24506cbe8456e2993e923bae4c89040cc6a1d8d06f7 fetch_syn_partners.py
|
||||
ae203b77c011d01ab8fb1cc76fcf4b0223a71e309d6995445fc7f33ce22a2f06 finalize.py
|
||||
aca6c69399538a41bc9148f2c837f0546e8a39d6b2b833585b9c55e53fa8414b localize.py
|
||||
554e9378e068fb13abb4f08dbf9ea710228fa535da2442edb913261fa22fcb20 regression.py
|
||||
238506a18fb81d544480c815d2517589e5b9d8c1932e86a5fe163a1b2955f712 regression_retry.py
|
||||
cb67bab972195090d5c4188169982abdc960479b8031477f1f39ac445814e88f test_audit.py
|
||||
732dcfe1a9c807356f7664393f6678743c61e3fe38a915ca0ce3f00ee7b17d9a validate_localization.py
|
||||
@@ -0,0 +1,49 @@
|
||||
"""User-authorized official additional reference download. Never writes raw."""
|
||||
import base64
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from audit import save
|
||||
|
||||
ROOT=Path('/data/digital-drosophila')
|
||||
OUT=ROOT/'experiments/phase4aa-apl-pn-scope/external-resources/synaptic-partners'
|
||||
NAME='syn-partners-male-cns-v1.0-minconf-0.5.feather'
|
||||
OBJECT='v1.0/connectome-data/flat-connectome/'+NAME
|
||||
META='https://storage.googleapis.com/storage/v1/b/flyem-male-cns/o/'+urllib.parse.quote(OBJECT,safe='')
|
||||
|
||||
def main():
|
||||
OUT.mkdir(parents=True,exist_ok=True)
|
||||
with urllib.request.urlopen(META,timeout=30) as r:meta=json.load(r)
|
||||
assert meta['name']==OBJECT and int(meta['size'])==6777179098
|
||||
assert meta['generation']=='1780494942562468','Stop on an unexpected upstream revision'
|
||||
md5=base64.b64decode(meta['md5Hash']).hex()
|
||||
if (OUT/'upstream-object.json').exists():
|
||||
prior=json.loads((OUT/'upstream-object.json').read_text())
|
||||
assert all(prior[k]==meta[k] for k in ('name','generation','size','md5Hash'))
|
||||
else:save(OUT/'upstream-object.json',meta)
|
||||
url='https://storage.googleapis.com/flyem-male-cns/'+OBJECT+'?generation='+meta['generation']
|
||||
if not (OUT/'authorization-and-source.json').exists():
|
||||
save(OUT/'authorization-and-source.json',{'user_authorized_additional_download':True,'new_reference_only':True,
|
||||
'source_page':'https://male-cns.janelia.org/download/','url':url,'metadata_url':META,
|
||||
'dataset':'male-cns:v1.0','license':'CC-BY-4.0','EM_downloaded':False,
|
||||
'raw_modified':False,'retrieved_utc':time.strftime('%Y-%m-%dT%H:%M:%SZ',time.gmtime())})
|
||||
cmd=['aria2c','--continue=true','--max-connection-per-server=8','--split=8','--min-split-size=16M',
|
||||
'--max-tries=8','--retry-wait=5','--connect-timeout=30','--timeout=60','--auto-file-renaming=false',
|
||||
'--allow-overwrite=false','--check-integrity=true','--file-allocation=none','--summary-interval=20',
|
||||
'--enable-color=false','--console-log-level=warn','--checksum=md5='+md5,'--dir='+str(OUT),'--out='+NAME,url]
|
||||
begin=time.perf_counter();subprocess.run(cmd,check=True)
|
||||
file=OUT/NAME;hs=hashlib.sha256();hm=hashlib.md5()
|
||||
with file.open('rb') as f:
|
||||
for b in iter(lambda:f.read(8<<20),b''):hs.update(b);hm.update(b)
|
||||
result={'bytes':file.stat().st_size,'sha256':hs.hexdigest(),'md5':hm.hexdigest(),'official_md5':md5,
|
||||
'generation':meta['generation'],'download_and_verification_seconds':time.perf_counter()-begin,
|
||||
'pass':file.stat().st_size==int(meta['size']) and hm.hexdigest()==md5,'path':str(file)}
|
||||
assert result['pass'],'Official MD5/size mismatch; retain file and stop'
|
||||
if not (OUT/'download-verification.json').exists():save(OUT/'download-verification.json',result)
|
||||
print(json.dumps(result),flush=True)
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,51 @@
|
||||
"""Seal only this newly created audit and copy small evidence into Git registry."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import time
|
||||
from audit import save,digest,seal,immutable_check
|
||||
|
||||
ROOT=Path('/data/digital-drosophila');MODEL=ROOT/'models/male-cns-lif/apl-pn-scope-audit-v1';EXP=ROOT/'experiments/phase4aa-apl-pn-scope'
|
||||
def main():
|
||||
assert not (EXP/'manifest.json').exists() and not (MODEL/'manifest.json').exists(),'Already sealed'
|
||||
static=json.loads((EXP/'audit/results.json').read_text())
|
||||
local=json.loads((EXP/'localization/summary.json').read_text())
|
||||
pairs=json.loads((EXP/'localization/complete-pair-set-validation.json').read_text())
|
||||
gpu=json.loads((EXP/'retry-with-correct-PATH/gpu-frozen-regression/summary.json').read_text())
|
||||
external=json.loads((EXP/'external-resources/synaptic-partners/download-verification.json').read_text())
|
||||
assert static['frozen_integrity_pass'] and local['pass'] and pairs['pass'] and gpu['pass'] and external['pass']
|
||||
post=immutable_check(ROOT,ROOT/'registry/snapshots/bootstrap-20260929',4)
|
||||
save(EXP/'regression/final-old-files-integrity.json',post);assert post['pass']
|
||||
result={'phase':'4AA','status':'PASS-APL-PN-ANATOMICAL-LOCALIZATION / NO-DYNAMICS-CANDIDATE',
|
||||
'candidate':False,'previous_static_spatial_block_resolved_by_user_authorized_reference_data':True,
|
||||
'old_static_audit_retained_without_overwrite':True,'canonical_neurons':166700,'canonical_edges':25582938,
|
||||
'APL_PN_edges':268,'APL_target_PNs':259,'APL_PN_contacts':4600,'calyx_contacts':3820,
|
||||
'calyx_fraction':3820/4600,'ROI_unspecified_contacts':731,'other_ROI_contacts':49,
|
||||
'watched_PN_contacts':247,'watched_calyx_contacts':228,'watched_unspecified_contacts':19,
|
||||
'complete_pair_checks':pairs,'frozen_trials_pass':gpu['pass'],'frozen_trials_count':len(gpu['trials']),
|
||||
'old_scientific_files_verified':post['checked'],'old_file_integrity_pass':post['pass'],
|
||||
'external_MD5_pass':external['pass'],'external_SHA256':external['sha256'],
|
||||
'brain_dynamics_changed':False,'training':False,'receptor_or_terminal_gain_calibrated':False,
|
||||
'bouton_identity_confirmed':False,'ComfyUI_stopped_or_unloaded':False,'top_level_gate_fraction':'5/10',
|
||||
'failures_preserved':['initial experiment-directory permission before any output','first GPU build lookup of Ninja failed before simulation','first aria2 checksum argument invalid before download'],
|
||||
'finalized_utc':time.strftime('%Y-%m-%dT%H:%M:%SZ',time.gmtime())}
|
||||
save(EXP/'audit/final-summary.json',result)
|
||||
with (MODEL/'REPORT.md').open('rb') as src,(EXP/'REPORT.md').open('xb') as dst:shutil.copyfileobj(src,dst)
|
||||
model_count=seal(MODEL);experiment_count=seal(EXP)
|
||||
verified=[]
|
||||
for base in (MODEL,EXP):
|
||||
manifest=json.loads((base/'manifest.json').read_text());bad=[r['path'] for r in manifest['files'] if digest(base/r['path'])!=r['sha256']]
|
||||
assert not bad;verified.append({'path':str(base),'files':len(manifest['files']),'pass':not bad})
|
||||
registry=ROOT/'registry/phase4aa-apl-pn-scope';registry.mkdir(exist_ok=False)
|
||||
records=[]
|
||||
for p in sorted(EXP.rglob('*')):
|
||||
if not p.is_file() or p.suffix not in ('.json','.csv','.md','.sha256') or p.stat().st_size>2*1024**2:continue
|
||||
dst=registry/'evidence'/p.relative_to(EXP);dst.parent.mkdir(parents=True,exist_ok=True)
|
||||
with p.open('rb') as src,dst.open('xb') as out:shutil.copyfileobj(src,out)
|
||||
records.append({'original_path':str(p.relative_to(ROOT)),'copy_path':str(dst.relative_to(registry)),'sha256':digest(p),'bytes':p.stat().st_size})
|
||||
save(registry/'evidence-index.json',records);save(registry/'seal-verification.json',verified)
|
||||
doc=ROOT/'docs/progress/phase4aa-apl-pn-scope.md';doc.parent.mkdir(parents=True,exist_ok=True)
|
||||
with (MODEL/'REPORT.md').open('rb') as src,doc.open('xb') as dst:shutil.copyfileobj(src,dst)
|
||||
print(json.dumps({'result':result['status'],'model_files':model_count,'experiment_files':experiment_count,'evidence_copies':len(records),'all_seals_pass':True}),flush=True)
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,112 @@
|
||||
"""Official paired-synapse localization; numerical counts must reconcile to CSR."""
|
||||
from pathlib import Path
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow as pa
|
||||
import pyarrow.compute as pc
|
||||
from scipy.spatial import cKDTree
|
||||
from audit import save,digest
|
||||
|
||||
ROOT=Path('/data/digital-drosophila');EXP=ROOT/'experiments/phase4aa-apl-pn-scope'
|
||||
|
||||
def main():
|
||||
pa.set_cpu_count(4);pa.set_io_thread_count(4);start=time.perf_counter()
|
||||
out=EXP/'localization';out.mkdir(exist_ok=False)
|
||||
official=EXP/'external-resources/synaptic-partners'
|
||||
verified=json.loads((official/'download-verification.json').read_text());assert verified['pass']
|
||||
path=Path(verified['path']);assert path.stat().st_size==verified['bytes']
|
||||
source_hash=digest(path);assert source_hash==verified['sha256']
|
||||
save(out/'protocol.json',{'source_sha256':source_hash,'source_generation':verified['generation'],
|
||||
'source_script_sha256':digest(Path(__file__)),'protocol_sha256':digest(Path(__file__).with_name('LOCALIZATION-PROTOCOL.md')),
|
||||
'coordinate_unit_nm':8,'distance_to_microns_multiplier':0.008,
|
||||
'nearest_neighbor_is_not_bouton_or_functional_identity':True,'before_analysis':True})
|
||||
canonical=ROOT/'datasets/male-cns-v1.0/processed/neuron-only'
|
||||
m=pd.read_parquet(canonical/'neuron-index/unified-metadata.parquet').sort_values('neuron_idx').reset_index(drop=True)
|
||||
ptr=np.load(canonical/'gpu/csr-indptr.int64.npy',mmap_mode='r');dst=np.load(canonical/'gpu/csr-indices.int32.npy',mmap_mode='r');w=np.load(canonical/'gpu/csr-weights.int64.npy',mmap_mode='r')
|
||||
body=m.body_id.to_numpy();apl=body[m.annotation_type.eq('APL')];kc=body[m.annotation_class.eq('Kenyon_Cell')];alln=body[m.annotation_class.eq('ALLN')]
|
||||
pnpairs=pd.read_parquet(EXP/'connectivity/APL-PN-reciprocity.parquet');pn=pnpairs.target_body_id.unique()
|
||||
watchidx=json.loads((ROOT/'experiments/phase4z-pn-feedback-audit/protocol/watched.json').read_text())['all_incoming_ALPNs']
|
||||
watch=body[watchidx]
|
||||
groups={k:[] for k in ('APL-incident','APL-target-PN-to-KC','watched-PN-to-ALLN','ALLN-to-watched-PN')}
|
||||
sets={name:pa.array(arr.astype(np.int64)) for name,arr in [('APL',apl),('KC',kc),('ALLN',alln),('PN',pn),('watched',watch)]}
|
||||
total=0
|
||||
with pa.memory_map(str(path),'r') as f:
|
||||
reader=pa.ipc.open_file(f);names=reader.schema.names
|
||||
required=['x_pre','y_pre','z_pre','body_pre','conf_pre','x_post','y_post','z_post','body_post','conf_post','primary_post']
|
||||
save(out/'actual-schema.json',{'fields':[{'name':x.name,'dtype':str(x.type)} for x in reader.schema],'record_batches':reader.num_record_batches})
|
||||
assert set(required).issubset(names),names
|
||||
for i in range(reader.num_record_batches):
|
||||
batch=reader.get_batch(i);total+=len(batch)
|
||||
pre=batch.column(names.index('body_pre'));post=batch.column(names.index('body_post'))
|
||||
masks={
|
||||
'APL-incident':pc.or_(pc.is_in(pre,value_set=sets['APL']),pc.is_in(post,value_set=sets['APL'])),
|
||||
'APL-target-PN-to-KC':pc.and_(pc.is_in(pre,value_set=sets['PN']),pc.is_in(post,value_set=sets['KC'])),
|
||||
'watched-PN-to-ALLN':pc.and_(pc.is_in(pre,value_set=sets['watched']),pc.is_in(post,value_set=sets['ALLN'])),
|
||||
'ALLN-to-watched-PN':pc.and_(pc.is_in(pre,value_set=sets['ALLN']),pc.is_in(post,value_set=sets['watched']))}
|
||||
for name,mask in masks.items():
|
||||
sub=batch.filter(mask)
|
||||
if len(sub):groups[name].append(pa.Table.from_batches([sub]))
|
||||
if (i+1)%500==0:print('streamed_batches',i+1,'rows',total,flush=True)
|
||||
lookup=pd.Series(m.neuron_idx.to_numpy(),index=body)
|
||||
dfs={};checks=[]
|
||||
keys=['body_pre','x_pre','y_pre','z_pre','body_post','x_post','y_post','z_post']
|
||||
for name,tables in groups.items():
|
||||
a=pa.concat_tables(tables).to_pandas();assert not a.duplicated(keys).any(),name+' duplicate partner row'
|
||||
a['source_neuron_idx']=a.body_pre.map(lookup);a['target_neuron_idx']=a.body_post.map(lookup)
|
||||
canonical_rows=a.source_neuron_idx.notna()&a.target_neuron_idx.notna()
|
||||
pairs=a.loc[canonical_rows].groupby(['source_neuron_idx','target_neuron_idx']).size()
|
||||
problems=[]
|
||||
for (s,t),count in pairs.items():
|
||||
s,t=int(s),int(t);eid=np.arange(ptr[s],ptr[s+1]);found=eid[dst[eid]==t]
|
||||
if len(found)!=1 or int(w[found[0]])!=count:problems.append({'source':s,'target':t,'actual_partner_count':int(count),'CSR_weight':int(w[found[0]]) if len(found)==1 else None})
|
||||
result={'group':name,'rows':len(a),'canonical_rows':int(canonical_rows.sum()),'canonical_pairs':len(pairs),'duplicate_partner_rows':0,'mismatches':problems,'pass':not problems}
|
||||
checks.append(result);save(out/(name+'-count-check.json'),result);assert result['pass'],name+' count mismatch'
|
||||
a.to_parquet(out/(name+'.parquet'),index=False);dfs[name]=a
|
||||
print(json.dumps(result),flush=True)
|
||||
a=dfs['APL-incident'];forward=a[a.body_pre.isin(apl)&a.body_post.isin(pn)].copy();reverse=a[a.body_pre.isin(pn)&a.body_post.isin(apl)].copy()
|
||||
assert len(forward)==int(pnpairs.raw_synapse_count.sum())==4600
|
||||
frames={'APL-to-PN':forward,'PN-to-APL':reverse,'PN-to-KC':dfs['APL-target-PN-to-KC'],
|
||||
'watched-PN-to-ALLN':dfs['watched-PN-to-ALLN'],'ALLN-to-watched-PN':dfs['ALLN-to-watched-PN']}
|
||||
roi_rows=[];pair_rows=[]
|
||||
for name,frame in frames.items():
|
||||
for roi,g in frame.groupby('primary_post',dropna=False,observed=True):
|
||||
roi_rows.append({'group':name,'primary_post':str(roi),'contacts':len(g),'fraction':len(g)/len(frame),'source_bodies':g.body_pre.nunique(),'target_bodies':g.body_post.nunique()})
|
||||
for (pre,post,roi),g in frame.groupby(['body_pre','body_post','primary_post'],dropna=False,observed=True):
|
||||
pair_rows.append({'group':name,'body_pre':int(pre),'body_post':int(post),'primary_post':str(roi),'contacts':len(g)})
|
||||
rois=pd.DataFrame(roi_rows);rois.to_csv(out/'population-ROI-summary.csv',index=False);rois.to_parquet(out/'population-ROI-summary.parquet',index=False)
|
||||
pd.DataFrame(pair_rows).to_parquet(out/'pair-ROI-counts.parquet',index=False)
|
||||
# Neuropil annotation is a location label, not evidence for receptor or inhibitory coefficient.
|
||||
distances=[];pnkc=dfs['APL-target-PN-to-KC']
|
||||
for b,g in forward.groupby('body_post'):
|
||||
q=pnkc[pnkc.body_pre==b]
|
||||
if len(q):
|
||||
sites=q[['x_pre','y_pre','z_pre']].drop_duplicates().to_numpy(np.float64)
|
||||
query=g[['x_post','y_post','z_post']].to_numpy(np.float64)
|
||||
dist=cKDTree(sites).query(query,workers=1)[0]*0.008
|
||||
distances.append({'PN_body_id':int(b),'PN_neuron_idx':int(lookup.loc[b]),'APL_postsynaptic_contacts':len(g),'PN_to_KC_unique_presynaptic_sites':len(sites),
|
||||
'nearest_distance_microns_min':float(dist.min()),'nearest_distance_microns_median':float(np.median(dist)),
|
||||
'nearest_distance_microns_p95':float(np.quantile(dist,.95)),'nearest_distance_microns_max':float(dist.max()),
|
||||
'bouton_identity_confirmed':False,'watched':int(lookup.loc[b]) in watchidx})
|
||||
else:distances.append({'PN_body_id':int(b),'PN_neuron_idx':int(lookup.loc[b]),'APL_postsynaptic_contacts':len(g),'PN_to_KC_unique_presynaptic_sites':0,
|
||||
'bouton_identity_confirmed':False,'watched':int(lookup.loc[b]) in watchidx})
|
||||
pd.DataFrame(distances).to_csv(out/'spatial-neighborhoods.csv',index=False)
|
||||
watched_rows=[]
|
||||
for node in watchidx:
|
||||
b=body[node];row={'neuron_idx':node,'body_id':int(b),'type':str(m.iloc[node].annotation_type)}
|
||||
for name,frame in frames.items():
|
||||
sub=frame[frame.body_post==b] if name in ('APL-to-PN','ALLN-to-watched-PN') else frame[frame.body_pre==b]
|
||||
row[name]={str(k):int(v) for k,v in sub.groupby('primary_post',observed=True).size().items()}
|
||||
watched_rows.append(row)
|
||||
save(out/'watched-PN-ROI.json',watched_rows)
|
||||
summary={'pass':all(x['pass'] for x in checks),'source_sha256':source_hash,'full_source_rows_streamed':total,
|
||||
'groups':checks,'APL_to_PN_contacts_localized':len(forward),'APL_to_PN_pairs_localized':len(pnpairs),
|
||||
'APL_target_PNs_localized':int(forward.body_post.nunique()),'ROI_counts':rois.to_dict('records'),
|
||||
'spatial_location_from_official_partners':True,'bouton_identity_confirmed':False,'target_receptor_confirmed':False,
|
||||
'terminal_inhibition_coefficient_known':False,'new_dynamics':False,'whole_brain_functional_intervention_run':False,
|
||||
'wall_seconds':time.perf_counter()-start}
|
||||
save(out/'summary.json',summary);print(json.dumps(summary),flush=True)
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,65 @@
|
||||
{
|
||||
"created_utc": "2026-09-29T15:56:36.952596+00:00",
|
||||
"files": [
|
||||
{
|
||||
"bytes": 1607,
|
||||
"path": "LOCALIZATION-PROTOCOL.md",
|
||||
"sha256": "b87a601d869f0a4cf7f88ccb2b7f5592b22da25d11d3e5bfb1be4bba05c079c0"
|
||||
},
|
||||
{
|
||||
"bytes": 2468,
|
||||
"path": "PROTOCOL.md",
|
||||
"sha256": "e88f765be0fd91178eed1d862b1de8d8b307ba5ec3cc0ae0d0d93202f264acce"
|
||||
},
|
||||
{
|
||||
"bytes": 7290,
|
||||
"path": "REPORT.md",
|
||||
"sha256": "fa0bfd5a163a436e061c94e37cf6fe84ca75fdfa94d6f20ce85eda5058ca9ce7"
|
||||
},
|
||||
{
|
||||
"bytes": 18939,
|
||||
"path": "audit.py",
|
||||
"sha256": "4ef071ba88f4e1cd78f068be58414dd7f4099dfac69c68266c4f46e175eab9fb"
|
||||
},
|
||||
{
|
||||
"bytes": 906,
|
||||
"path": "audit.yaml",
|
||||
"sha256": "e0171bbd7dc7e693268eac00152e560e841f885c7d524c74d51fda7f7fbadbbf"
|
||||
},
|
||||
{
|
||||
"bytes": 2939,
|
||||
"path": "fetch_syn_partners.py",
|
||||
"sha256": "23db504ac8f31c62a434b24506cbe8456e2993e923bae4c89040cc6a1d8d06f7"
|
||||
},
|
||||
{
|
||||
"bytes": 4144,
|
||||
"path": "finalize.py",
|
||||
"sha256": "ae203b77c011d01ab8fb1cc76fcf4b0223a71e309d6995445fc7f33ce22a2f06"
|
||||
},
|
||||
{
|
||||
"bytes": 8867,
|
||||
"path": "localize.py",
|
||||
"sha256": "aca6c69399538a41bc9148f2c837f0546e8a39d6b2b833585b9c55e53fa8414b"
|
||||
},
|
||||
{
|
||||
"bytes": 4455,
|
||||
"path": "regression.py",
|
||||
"sha256": "554e9378e068fb13abb4f08dbf9ea710228fa535da2442edb913261fa22fcb20"
|
||||
},
|
||||
{
|
||||
"bytes": 394,
|
||||
"path": "regression_retry.py",
|
||||
"sha256": "238506a18fb81d544480c815d2517589e5b9d8c1932e86a5fe163a1b2955f712"
|
||||
},
|
||||
{
|
||||
"bytes": 1520,
|
||||
"path": "test_audit.py",
|
||||
"sha256": "cb67bab972195090d5c4188169982abdc960479b8031477f1f39ac445814e88f"
|
||||
},
|
||||
{
|
||||
"bytes": 2229,
|
||||
"path": "validate_localization.py",
|
||||
"sha256": "732dcfe1a9c807356f7664393f6678743c61e3fe38a915ca0ce3f00ee7b17d9a"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,72 @@
|
||||
"""Supplement: existing five frozen trials on 4090, new output paths only."""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
|
||||
ROOT=Path('/data/digital-drosophila')
|
||||
MODEL=ROOT/'models/male-cns-lif/apl-pn-scope-audit-v1'
|
||||
EXP=ROOT/'experiments/phase4aa-apl-pn-scope'
|
||||
sys.path.insert(0,str(MODEL))
|
||||
from audit import save,digest,immutable_check
|
||||
|
||||
def queues():
|
||||
result=[]
|
||||
for port in (8188,8189,8190):
|
||||
with urllib.request.urlopen(f'http://192.168.10.51:{port}/queue',timeout=5) as r:q=json.load(r)
|
||||
result.append({'port':port,'running':len(q.get('queue_running',[])),'pending':len(q.get('queue_pending',[]))})
|
||||
return result
|
||||
|
||||
def main():
|
||||
out=EXP/'gpu-frozen-regression';out.mkdir(exist_ok=False)
|
||||
q=queues();assert all(x['running']==0 and x['pending']==0 for x in q),'ComfyUI busy; defer'
|
||||
runtime=ROOT/'models/male-cns-lif/deterministic-runtime-v2'
|
||||
cfg=yaml.safe_load((runtime/'phase4p.yaml').read_text())
|
||||
os.environ['CUDA_VISIBLE_DEVICES']=cfg['device']['visible_gpu_uuid']
|
||||
os.environ['TORCH_CUDA_ARCH_LIST']='8.9'
|
||||
os.environ['CUDA_HOME']='/usr/local/cuda'
|
||||
os.environ['MAX_JOBS']='2'
|
||||
os.environ['PYTHONDONTWRITEBYTECODE']='1'
|
||||
sys.path.insert(0,str(ROOT/'models/male-cns-lif'))
|
||||
sys.path.insert(0,str(runtime))
|
||||
import torch
|
||||
import phase4p
|
||||
free,total=torch.cuda.mem_get_info();assert free>2*1024**3,'Insufficient free VRAM; no service eviction'
|
||||
save(out/'protocol-addendum.json',{'scope':'Five frozen reference v2 trials only; supplement to CPU structural audit',
|
||||
'reason':'User confirmed ComfyUI is not in use; baseline behavior check before any future mechanism intervention',
|
||||
'new_mechanism_or_training':False,'ComfyUI_queues':q,'available_vram_bytes':free,
|
||||
'reference':'experiments/phase4p-deterministic-runtime/phase4c-v2/reference',
|
||||
'historical_files_overwritten':False,'source_sha256':digest(Path(__file__)),
|
||||
'runtime_sha256':digest(runtime/'phase4p.py'),'config_sha256':digest(runtime/'phase4p.yaml')})
|
||||
torch.cuda.reset_peak_memory_stats();start=time.perf_counter();rt=phase4p.CanonicalRuntime(cfg,runtime,out)
|
||||
rows=[]
|
||||
for spec in cfg['reference_trials']:
|
||||
q=queues();assert all(x['running']==0 and x['pending']==0 for x in q),'ComfyUI work arrived; stop between trials'
|
||||
ref=ROOT/'experiments/phase4p-deterministic-runtime/phase4c-v2/reference'/spec['name']
|
||||
events=pd.read_parquet(ref/'spike-events.parquet')
|
||||
expected={'step':events.step.to_numpy(),'neuron_idx':events.neuron_idx.to_numpy(),
|
||||
'counts':np.load(ref/'per-neuron-spike-counts.int32.npy'),'v':np.load(ref/'final-voltage.float32.npy'),'g':np.load(ref/'final-g.float32.npy')}
|
||||
result=rt.run(np.flatnonzero(rt.pops[spec['population']]),spec['rate_hz'],spec['seed'],spec['duration_ms'])
|
||||
exact={k:bool(np.array_equal(result[k],v)) for k,v in expected.items()}
|
||||
finite=bool(np.isfinite(result['v']).all() and np.isfinite(result['g']).all())
|
||||
row={'trial':spec['name'],'exact':exact,'finite':finite,'pass':all(exact.values()) and finite,
|
||||
'wall_seconds':result['metrics']['wall_seconds'],'events':len(result['step']),
|
||||
'events_sha256':phase4p.sha_bytes(result['step'],result['neuron_idx']),
|
||||
'max_abs_V_error':float(np.max(np.abs(result['v'].astype(float)-expected['v']))),
|
||||
'max_abs_g_error':float(np.max(np.abs(result['g'].astype(float)-expected['g'])))}
|
||||
save(out/(spec['name']+'.json'),row);rows.append(row);print(json.dumps(row),flush=True)
|
||||
assert row['pass'],'Frozen regression diverged'
|
||||
save(out/'summary.json',{'pass':all(r['pass'] for r in rows),'trials':rows,'total_wall_seconds_including_build':time.perf_counter()-start,
|
||||
'peak_allocated_VRAM_bytes':int(torch.cuda.max_memory_allocated()),'device':torch.cuda.get_device_name(),
|
||||
'ComfyUI_models_unloaded':False,'new_dynamics':False,'original_reference_v1_not_overwritten':True,
|
||||
'oracle':'deterministic v2 / original v1 preserved; not claiming v1 atomic runtime is deterministic'})
|
||||
# No old scientific artifact may change even after imports/build/run.
|
||||
post=immutable_check(ROOT,ROOT/'registry/snapshots/bootstrap-20260929',4)
|
||||
save(out/'post-regression-integrity.json',post);assert post['pass']
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,10 @@
|
||||
"""Retry regression with the existing venv/toolkit executables on PATH; no install."""
|
||||
from pathlib import Path
|
||||
import os
|
||||
import regression
|
||||
|
||||
root=Path('/data/digital-drosophila')
|
||||
os.environ['PATH']=str(root/'models/male-cns-lif/.venv/bin')+':/usr/local/cuda/bin:'+os.environ['PATH']
|
||||
regression.EXP=regression.EXP/'retry-with-correct-PATH'
|
||||
regression.EXP.mkdir(exist_ok=False)
|
||||
regression.main()
|
||||
@@ -0,0 +1,29 @@
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
import unittest
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import pyarrow as pa
|
||||
import pyarrow.feather as feather
|
||||
from audit import raw_incident,population,save
|
||||
|
||||
class AuditTests(unittest.TestCase):
|
||||
def test_raw_incident_union_keeps_self_loop_once(self):
|
||||
with tempfile.TemporaryDirectory(prefix='apl-scope-test-') as d:
|
||||
p=Path(d)/'raw.feather'
|
||||
frame=pd.DataFrame({'body_pre':[10,10,20,40],'body_post':[10,30,10,50],'weight':[1,2,3,4]},dtype=np.int64)
|
||||
feather.write_feather(pa.Table.from_pandas(frame,preserve_index=False),p,chunksize=2)
|
||||
result=raw_incident(p,np.array([10],np.uint64))
|
||||
self.assertTrue(np.array_equal(result.to_numpy(),frame.iloc[:3].to_numpy()))
|
||||
with self.assertRaises(AssertionError):
|
||||
feather.write_feather(pa.table({'body_pre':[10],'wrong':[30],'weight':[1]}),p)
|
||||
raw_incident(p,np.array([10],np.uint64))
|
||||
def test_population_does_not_equate_dopamine_to_DAN(self):
|
||||
m=pd.DataFrame({'annotation_class':[None,None,'ALPN','DAN'], 'annotation_type':['APL','DPM','VA6_adPN','PAM01']})
|
||||
self.assertEqual(list(population(m)),['APL','DPM','ALPN','DAN'])
|
||||
def test_exclusive_output(self):
|
||||
with tempfile.TemporaryDirectory(prefix='apl-scope-test-') as d:
|
||||
p=Path(d)/'x.json';save(p,{'original':True})
|
||||
with self.assertRaises(FileExistsError):save(p,{'original':False})
|
||||
|
||||
if __name__=='__main__':unittest.main()
|
||||
@@ -0,0 +1,34 @@
|
||||
"""Independent complete-pair-set and weight check (including entirely missing pairs)."""
|
||||
from pathlib import Path
|
||||
import json
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from audit import save
|
||||
|
||||
ROOT=Path('/data/digital-drosophila');EXP=ROOT/'experiments/phase4aa-apl-pn-scope'
|
||||
c=ROOT/'datasets/male-cns-v1.0/processed/neuron-only'
|
||||
m=pd.read_parquet(c/'neuron-index/unified-metadata.parquet').sort_values('neuron_idx').reset_index(drop=True)
|
||||
ptr=np.load(c/'gpu/csr-indptr.int64.npy',mmap_mode='r');dst=np.load(c/'gpu/csr-indices.int32.npy',mmap_mode='r');w=np.load(c/'gpu/csr-weights.int64.npy',mmap_mode='r')
|
||||
src=np.repeat(np.arange(len(m),dtype=np.int32),np.diff(ptr))
|
||||
apl=np.flatnonzero(m.annotation_type.eq('APL'));kc=np.flatnonzero(m.annotation_class.eq('Kenyon_Cell'));alln=np.flatnonzero(m.annotation_class.eq('ALLN'))
|
||||
pn=pd.read_parquet(EXP/'connectivity/APL-PN-reciprocity.parquet').target_neuron_idx.unique()
|
||||
watch=json.loads((ROOT/'experiments/phase4z-pn-feedback-audit/protocol/watched.json').read_text())['all_incoming_ALPNs']
|
||||
selectors={
|
||||
'APL-incident':np.isin(src,apl)|np.isin(dst,apl),
|
||||
'APL-target-PN-to-KC':np.isin(src,pn)&np.isin(dst,kc),
|
||||
'watched-PN-to-ALLN':np.isin(src,watch)&np.isin(dst,alln),
|
||||
'ALLN-to-watched-PN':np.isin(src,alln)&np.isin(dst,watch)}
|
||||
results=[]
|
||||
for name,mask in selectors.items():
|
||||
frame=pd.read_parquet(EXP/'localization'/(name+'.parquet'))
|
||||
frame=frame[frame.source_neuron_idx.notna()&frame.target_neuron_idx.notna()]
|
||||
counts=frame.groupby(['source_neuron_idx','target_neuron_idx']).size().reset_index(name='weight')
|
||||
actual=counts.to_numpy(np.int64)
|
||||
ids=np.flatnonzero(mask)
|
||||
expected=np.column_stack((src[ids],dst[ids],w[ids])).astype(np.int64)
|
||||
actual=actual[np.lexsort((actual[:,1],actual[:,0]))];expected=expected[np.lexsort((expected[:,1],expected[:,0]))]
|
||||
passed=np.array_equal(actual,expected)
|
||||
results.append({'group':name,'expected_CSR_pairs':len(expected),'observed_pairs':len(actual),'complete_pair_set_and_weight_exact':passed})
|
||||
assert passed,name
|
||||
save(EXP/'localization/complete-pair-set-validation.json',{'pass':all(r['complete_pair_set_and_weight_exact'] for r in results),'groups':results})
|
||||
print(json.dumps(results,indent=2))
|
||||
Reference in New Issue
Block a user