Audit calyx-specific PN contact scope without changing frozen baseline

This commit is contained in:
Digital Drosophila Archive
2026-09-30 00:27:22 +08:00
parent d6843e675d
commit 3213aada79
77 changed files with 4152 additions and 14 deletions
@@ -0,0 +1,86 @@
# Phase 4AB — calyx contact-scope counterfactuals
This is an exploratory mechanism diagnostic, NOT an APL physiological model,
learning experiment, parameter calibration or new formal runtime candidate.
All previous scientific artifacts remain read-only. Only new output directories
are permitted. No additional download is required.
## Question and predeclared controls
Does removal of anatomically labelled calyx PN→KC input alter persistent KC
activity without also removing the same PNs' antennal-lobe output?
Select PN/ROI pairs from actual Phase 4AA APL→ALPN synaptic contacts whose
`primary_post` is exactly `CA(L)` or `CA(R)`. Select PN→KC contacts only when
their source PN and actual `primary_post` match that pair. This is **ROI-level
co-location**, not proof of the same bouton, receptor, physiological coefficient,
or even a per-presynaptic-site ROI annotation. `primary_post` labels the
postsynaptic site. Never reclassify `CentralBrain-unspecified` by proximity.
Compare four conditions on the same eligible PN set:
1. Frozen reference; no changed weights or injected current.
2. Block only the matched CA PN→KC contact component.
3. Block all PN→KC output of those PNs, irrespective of ROI.
4. Block all fast outgoing synaptic weights of those PNs (broad scope control).
These are complete-block engineering interventions, not estimates of biological
APL inhibition. There is no strength sweep or winner selection. Condition 4
does **not** silence PN somata or alter their force requests. The frozen graded
APL input matrix remains active in all four conditions; condition 4 is therefore
not a complete PN knockout. All fast edge slots remain present. Output masks are
not plastic masks, and nothing is written back to base CSR or an old model.
For condition 2, the in-memory diagnostic effective weight is
`float32(float64(frozen_effective_weight) * (raw_count - CA_count) / raw_count)`.
The linear per-contact decomposition is an engineering consequence of the
existing linear weight model, not a measured transmitter-release law.
At strength zero there is no arithmetic or write at all; identity must be exact.
Conditions 3/4 set only the declared fast effective weights to zero. Anatomical
counts, transmitter signs, LIF/refractory/delay, APL matrices and calibration,
eligibility-v2 and the 14,232 plastic-edge mask remain frozen.
## Trials and measurements, declared before results
Use all six old calibration odors, seed 131101, the frozen Phase 4V DoOR→PN
request generator, 120 ms stimulus followed by observation to 1,000 ms. Do not
consume held-out odors. Reset all states between trials. Repeat every condition
for odor 0 to test exact determinism. No DAN reinforcement or weight training.
Record all spike events, all final counts/V/g, watched PN/KC/ALLN states, original
force requests and their accepted/preexisting/blocked status, KC active fraction,
late KC82312/VA6/ALLN activity (500–1,000 ms), and class-wise counts. All conditions
share requested inputs; delivered events need not match because recurrent state
and refractory periods can change. Explicitly report this distinction.
Unmodified trials must reproduce Phase 4Z whole events, watched states and
Phase 4Y final arrays bit-for-bit. Each first-odor repeat must be exact. Run the
five frozen Phase 4C trials against the deterministic-v2 archived oracle in new
output paths. Original v1 files remain preserved, not relabelled as deterministic.
Additionally, replay the first frozen trial with its recorded events fixed, using
the frozen ordered reduction and 18-step queue to validate locality: no terminal
mask affects non-KC arrivals; broad controls may do so. This fixed-event test
only measures the direct operator, not whole-brain causality.
## Interpretation gate
Reduced KC activity alone is not repaired encoding or memory. A broad blockade
can destroy useful drive. If CA blockade fails to remove the late KC response,
report that it is insufficient under this model; do not increase gain or relabel
unknown contacts. If it succeeds while ALPN/ALLN persist, report scope-specific
model counterfactual evidence, not proof that biological APL has that strength.
No result in this phase can generate a learned/physiological candidate.
## Source boundaries
- MaleCNS official v1.0 syn-partners, CC BY 4.0, already verified in Phase 4AA:
https://male-cns.janelia.org/download/
- Prisco et al. (2021), doi:10.7554/eLife.74172: local calyx APL interactions;
does not assign MaleCNS edge-specific inhibition coefficients.
- Amin et al. (2020), doi:10.7554/eLife.56954: spatially localized APL activity
and inhibition; does not validate the present engineering complete-block rule.
Unknown: bouton identity, receptor distribution, strength/time course of terminal
release modulation, and whether adding such physiology would rescue memory.
No game, decoder, RL, visual input, threshold/eta tuning, or next-phase training.
@@ -0,0 +1,39 @@
# Terminal-scope diagnostic v1 (Phase 4AB)
Independent engineering counterfactual harness, **not a formal brain model**.
Read `PROTOCOL.md` before interpreting any result. Frozen input datasets and
scientific sources are mandatory; no data are fetched by this stage.
## Sequence
Use the existing scientific environment with `PYTHONDONTWRITEBYTECODE=1`,
`OMP_NUM_THREADS=4`, `OPENBLAS_NUM_THREADS=4`, `MKL_NUM_THREADS=4`:
1. `python -B -m unittest -v test_scope`
2. `python -B diagnostic.py prepare` (new empty experiment directory only)
3. `python -B validate.py`
4. `python -B launch.py` (explicit frozen module paths, single 4090)
5. `python -B environment.py`
6. `python -B analyze.py`
7. `python -B diagnostic.py finalize`
8. `python -B publish.py` (new small Git evidence copies only)
The completed output is sealed. **Do not rerun these commands into its directory.**
Unit tests may be rerun read-only. An independent full replication requires a new
isolated experiment/model directory, with its paths changed consistently in the
copied configuration/entry points before preregistration. Retain this original
source and every old output. The current protocol also includes four in-run exact
repeats, six historical exact comparisons and five archived reference checks.
`launch.py` fixes the initial import-path failure without editing the five
preregistered source/config files. `analyze.py` is a post-hoc summary only and
does not launch any additional experiment or fit any parameter. Additional helper
hashes and imported code hashes are captured in the final manifest/protocol.
The existing environment is reused, never upgraded. ComfyUI queues are checked
between trials; a busy service causes deferral, not eviction. Performance includes
observation overhead. The historical APL graded input/output remains unchanged.
All-fast-PN-output blockade is not somatic silencing and not a complete knockout:
PN spikes still contribute to the frozen graded APL input. `primary_post` is a
postsynaptic ROI label, not a reconstructed bouton identity. Unknown stays unknown.
@@ -0,0 +1,89 @@
"""Post-hoc interpretation only; no further trials, selection or parameter changes."""
from pathlib import Path
import sys
import numpy as np
import pandas as pd
ROOT = Path('/data/digital-drosophila'); MODEL = ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'
EXP = ROOT/'experiments/phase4ab-terminal-scope'
sys.path.insert(0, str(MODEL))
import diagnostic
cfg, _, _, _, save, digest, _, _ = diagnostic.load_context()
assert (EXP/'audit/run-complete.json').exists()
m = pd.read_parquet(ROOT/cfg['canonical']/'neuron-index/unified-metadata.parquet').sort_values('neuron_idx').reset_index(drop=True)
trials = pd.read_csv(EXP/'audit/trials.csv'); watched = pd.read_csv(EXP/'audit/watched-counts.csv')
remaining, comparisons = [], []
for oi in cfg['odor_indices']:
names = {c: f'{oi}-{cfg["seed"]}-{c}' for c in cfg['conditions']}
narrow = pd.read_parquet(EXP/'events'/f'{names["calyx_PN_KC_contact_block"]}.parquet')
allkc = pd.read_parquet(EXP/'events'/f'{names["all_PN_KC_block"]}.parquet')
a = np.load(EXP/'states'/f'{names["calyx_PN_KC_contact_block"]}.npz')
b = np.load(EXP/'states'/f'{names["all_PN_KC_block"]}.npz')
comparisons.append({'odor_index': oi, 'terminal_vs_allKC_events_exact': narrow.equals(allkc),
'counts_exact': bool(np.array_equal(a['counts'], b['counts'])),
'max_V_difference': float(np.abs(a['final_v'].astype(float)-b['final_v']).max()),
'max_g_difference': float(np.abs(a['final_g'].astype(float)-b['final_g']).max())})
kc = np.flatnonzero(m.annotation_class.eq('Kenyon_Cell'))
late = narrow[(narrow.step >= 5000) & narrow.neuron_idx.isin(kc)]
for node, group in late.groupby('neuron_idx'):
row = m.iloc[node]
remaining.append({'odor_index': oi, 'neuron_idx': int(node), 'body_id': int(row.body_id),
'type': str(row.annotation_type), 'late_spikes': len(group),
'first_late_spike_ms': float(group.step.min()/10), 'last_spike_ms': float(group.step.max()/10),
'source_identity_not_inferred_from_this_count': True})
pd.DataFrame(remaining).to_csv(EXP/'audit/remaining-late-KCs.csv', index=False)
pd.DataFrame(comparisons).to_csv(EXP/'audit/narrow-vs-allKC.csv', index=False)
rows = []
for name, g in trials.groupby('condition', sort=False):
row = {'condition': name}
for key in ['whole_spikes', 'KC_active', 'KC_spikes', 'KC_late_spikes', 'KC82312_late_spikes', 'VA6_late_spikes', 'wall_seconds']:
row[key+'_min'] = float(g[key].min()); row[key+'_max'] = float(g[key].max())
rows.append(row)
pd.DataFrame(rows).to_csv(EXP/'audit/condition-ranges.csv', index=False)
ca = trials[trials.condition == 'calyx_PN_KC_contact_block']
wide = trials[trials.condition == 'whole_PN_output_block']
local = watched[(watched.population == 'ALLN') & watched.trial.str.endswith('calyx_PN_KC_contact_block')]
result = {'posthoc_summary_not_new_experiment': True,
'all_six_CA_block_targetKC_late_zero': bool(ca.KC82312_late_spikes.eq(0).all()),
'all_six_CA_block_VA6_late_persists': bool(ca.VA6_late_spikes.gt(0).all()),
'all_six_broad_block_KC_and_VA6_late_zero': bool((wide.KC_late_spikes.eq(0) & wide.VA6_late_spikes.eq(0)).all()),
'CA_block_active_KC_range': [int(ca.KC_active.min()), int(ca.KC_active.max())],
'CA_block_ALLN_watched_late_rate_range_hz': [float(local.late_rate_hz.min()), float(local.late_rate_hz.max())],
'CA_vs_allKC_spike_sequence_exact_for_all_odors': bool(all(x['terminal_vs_allKC_events_exact'] for x in comparisons)),
'not_a_physiological_APL_gain_estimate': True, 'not_a_memory_or_separability_success': True,
'new_candidate': False, 'source_sha256': digest(Path(__file__))}
save(EXP/'audit/mechanism-interpretation.json', result)
def span(frame, key):
lo, hi = frame[key].min(), frame[key].max()
return str(int(lo)) if lo == hi else f'{int(lo)}–{int(hi)}'
ref = trials[trials.condition == 'frozen_reference']
lines = ['# Phase 4AB 机制解读', '',
'正式状态:**COMPLETE-SCOPE-COUNTERFACTUAL / NO-DYNAMICS-CANDIDATE**。', '',
'这一轮区分了“PN→KC 传入位置”与“维持 PN 放电的上游反馈”。没有修复或批准新的 APL 生理模型,也没有训练。', '',
'| 诊断范围 | 活跃 KC(6 个旧气味) | 目标 KC 晚期 spikes | VA6 PN 晚期 spikes |',
'|---|---:|---:|---:|']
labels = {'frozen_reference': '原模型', 'calyx_PN_KC_contact_block': '仅明确 calyx 接触',
'all_PN_KC_block': '同批 PN 的全部 KC 输出', 'whole_PN_output_block': '同批 PN 的全部快速输出'}
for name in cfg['conditions']:
g = trials[trials.condition == name]
lines.append(f'| {labels[name]} | {span(g,"KC_active")} | {span(g,"KC82312_late_spikes")} | {span(g,"VA6_late_spikes")} |')
lines += ['', '晚期窗口是 500–1,000 ms。每个气味沿用 seed 131101;总计 24 组诊断、4 次精确重复,不是 5-seed 学习验证。', '',
'## 能说明什么', '',
f'仅 calyx 接触阻断后,目标 KC82312 的晚期 spikes 范围为 {span(ca,"KC82312_late_spikes")},VA6 仍为 {span(ca,"VA6_late_spikes")}。',
f'同时 KC 总活跃数降为 {span(ca,"KC_active")},原模型为 {span(ref,"KC_active")}。不能把这种接近静默的结果包装成编码或记忆改善。',
'即使两个局部阻断条件的事件序列相同,其电压/g 仍可能不同;逐气味比较见 narrow-vs-allKC.csv。',
'宽范围 PN 快速输出阻断影响反馈路径,不等价于局部 calyx 调制;它也没有关闭 PN 胞体或 frozen graded APL 的 PN 输入。', '',
'## 仍不能说明什么', '',
'完全阻断只是工程极端对照,不是 APL 的实测释放抑制比例。没有估计 receptor、bouton、局部扩散范围或时间常数。',
'未使用 held-out 气味,未检验学习、保持、反转或动作输出。不得选取这次“让活动更少”的操作作为正式脑模型。',
'剩余 late KC 的 ID/type 已单独保存;其来源不能仅凭残余计数猜测。', '',
'## 下一步边界', '',
'先继续 upstream feedback / compartment-specific physiology 的证据约束,区分 calyx 信号传递与触角叶持续放电,再确定可检验的局部动力学假设。',
'不能靠增加整体抑制、调 eta/threshold、扩大 plastic mask 或让 KC 全静默宣称修复。当前不进入学习、游戏或视觉。', '',
'相关原始研究:[Prisco et al., 2021](https://doi.org/10.7554/eLife.74172)、[Amin et al., 2020](https://doi.org/10.7554/eLife.56954)。它们支持研究局部 APL 作用,不直接验证本实验完全阻断幅度。', '',
'顶层验收仍为 5/10(50%);本轮推进的是机制定位与可重复诊断能力。最终完整性状态见 regression/final-integrity.json。']
with (EXP/'INTERPRETATION.zh.md').open('x', encoding='utf-8') as out:
out.write('\n'.join(lines)+'\n')
print(result)
@@ -0,0 +1,11 @@
451b53b128eb0ad67acb81f060f3676794a68c98637dd2c0eddd137da509fccf PROTOCOL.md
2842b5dd94ed30fc01d1a8b8d95e235de78695d3d1c1076402499276fb903261 README.md
33ba5f323e9ed3343f43a8d64fb7e862c8a4f1118b9b441f4745e26cc14303a8 analyze.py
e22088d9847504a50c0e684d2cee9420f484ab1ebef9d6057921062d8f785e73 config.yaml
c8ec2ae6e72d35fedbd42acc6e6b5e7af2634d5b9805dc20e777b172c55d5aa4 diagnostic.py
be92ce24e63910f67da0238b19e3abb882e1afd317561fccf01734f1b4a56f25 environment.py
0b2f71365267886933ba5becb52f4123d3d4e95e9c5bc0cf19c9bf0563831625 launch.py
bc194bb857336908b1c3fe71cc0e7b7fe4152794745bd3443533d47da886f5ec publish.py
8f5774496dcceaee1c2793a5308754cbd0864c4a38d507481071d988f9de424d scope.py
38a1c6e653ae97e77e5feda3ee8f2f8a3c366755bfbbd57ca1373b9b8f971bbb test_scope.py
971136a667367a84cfc50ebc3fa73a749eddb8b8e1ca6931708ba67bfc12f0c1 validate.py
@@ -0,0 +1,22 @@
phase: 4AB
root: /data/digital-drosophila
model: models/male-cns-lif/terminal-scope-diagnostic-v1
experiment: experiments/phase4ab-terminal-scope
prior_model: models/male-cns-lif/apl-pn-scope-audit-v1
prior_experiment: experiments/phase4aa-apl-pn-scope
runtime: models/male-cns-lif/deterministic-runtime-v2
snapshot: registry/snapshots/bootstrap-20260929
canonical: datasets/male-cns-v1.0/processed/neuron-only
event_prior: experiments/phase4z-pn-feedback-audit
state_prior: experiments/phase4y-dpm-hypothesis
input_model: models/male-cns-lif/csus-audit-v1
roi_labels: [CA(L), CA(R)]
seed: 131101
odor_indices: [0, 1, 2, 3, 4, 5]
repeat_odor_index: 0
late_start_ms: 500
target_KC: 82312
watched_VA6: 133666
conditions: [frozen_reference, calyx_PN_KC_contact_block, all_PN_KC_block, whole_PN_output_block]
workers: 4
no_candidate: true
@@ -0,0 +1,369 @@
"""New-path-only Phase 4AB scope diagnostics; never edits frozen scientific files."""
import argparse
from datetime import datetime, timezone
import json
import os
from pathlib import Path
import subprocess
import sys
import time
import urllib.request
import numpy as np
import pandas as pd
import yaml
from scope import contact_scope, edge_ids_for_pairs, blocked_weights
def load_context():
cfg = yaml.safe_load(Path(__file__).with_name('config.yaml').read_text())
root = Path(cfg['root']); model = root/cfg['model']; exp = root/cfg['experiment']
sys.path.insert(0, str(root/cfg['prior_model']))
from audit import save, digest, immutable_check, seal
return cfg, root, model, exp, save, digest, immutable_check, seal
def manifest_check(folder, digest):
manifest = folder/'manifest.json'
rows = json.loads(manifest.read_text())['files']
bad = []
for row in rows:
path = folder/row['path']
if not path.is_file() or path.stat().st_size != row['bytes'] or digest(path) != row['sha256']:
bad.append(row['path'])
return {'pass': not bad, 'checked': len(rows), 'mismatches': bad,
'manifest_sha256': digest(manifest), 'checksums_sha256': digest(folder/'checksums.sha256')}
def integrity(cfg, root, digest, immutable_check):
result = {'original': immutable_check(root, root/cfg['snapshot'], cfg['workers'])}
for key in ('prior_model', 'prior_experiment'):
result[key] = manifest_check(root/cfg[key], digest)
assert all(x['pass'] for x in result.values()), result
return result
def queues():
out = []
for port in (8188, 8189, 8190):
with urllib.request.urlopen(f'http://192.168.10.51:{port}/queue', timeout=5) as r:
q = json.load(r)
out.append({'port': port, 'running': len(q['queue_running']), 'pending': len(q['queue_pending'])})
assert all(x['running'] == x['pending'] == 0 for x in out), 'ComfyUI busy: defer, never evict'
return out
def prepare():
cfg, root, model, exp, save, digest, immutable_check, _ = load_context()
assert exp.parent == root/'experiments' and model.parent == root/'models/male-cns-lif'
if exp.exists() and any(exp.iterdir()): raise RuntimeError('Refusing nonempty experiment')
exp.mkdir(exist_ok=True)
for d in ('protocol', 'audit', 'masks', 'events', 'states', 'observations', 'requests', 'replay', 'regression', 'performance'):
(exp/d).mkdir()
save(exp/'protocol/preregistered.json', {'config': cfg, 'before_analysis': True,
'timestamp_utc': datetime.now(timezone.utc).isoformat(),
'parent_commit': subprocess.check_output(['git', '-C', str(root), 'rev-parse', 'HEAD'], text=True).strip(),
'source_hashes': {p.name: digest(p) for p in sorted(model.iterdir()) if p.is_file()}})
save(exp/'regression/preflight.json', integrity(cfg, root, digest, immutable_check))
can = root/cfg['canonical']; prior = root/cfg['prior_experiment']
m = pd.read_parquet(can/'neuron-index/unified-metadata.parquet').sort_values('neuron_idx').reset_index(drop=True)
ptr = np.load(can/'gpu/csr-indptr.int64.npy', mmap_mode='r')
dst = np.load(can/'gpu/csr-indices.int32.npy', mmap_mode='r')
raw = np.load(can/'gpu/csr-weights.int64.npy', mmap_mode='r')
assert len(m) == 166700 and len(dst) == 25582938 and int(raw.sum()) == 124177617
assert m.body_id.dtype == np.uint64 and np.array_equal(m.neuron_idx, np.arange(len(m)))
src = np.repeat(np.arange(len(m), dtype=np.int32), np.diff(ptr))
apl = np.flatnonzero(m.annotation_type.eq('APL')); pn = np.flatnonzero(m.annotation_class.eq('ALPN'))
kc = np.flatnonzero(m.annotation_class.eq('Kenyon_Cell'))
contacts = pd.read_parquet(prior/'localization/APL-incident.parquet')
ap = contacts[contacts.source_neuron_idx.isin(apl) & contacts.target_neuron_idx.isin(pn)].copy()
pk = pd.read_parquet(prior/'localization/APL-target-PN-to-KC.parquet')
assert len(ap) == 4600 and ap.target_neuron_idx.nunique() == 259 and len(pk) == 385229
assert pk.source_neuron_idx.isin(pn).all() and pk.target_neuron_idx.isin(kc).all()
pairs, allowed, keep = contact_scope(ap, pk, cfg['roi_labels'])
ids = edge_ids_for_pairs(pairs, ptr, dst)
assert np.array_equal(raw[ids], pairs.raw_partner_count)
full_expected = np.flatnonzero(np.isin(src, ap.target_neuron_idx.unique()) & np.isin(dst, kc))
assert np.array_equal(np.sort(ids), full_expected), 'Missing whole PN->KC pairs'
pairs['edge_id'] = ids
pairs['source_body_id'] = m.body_id.to_numpy()[pairs.source_neuron_idx]
pairs['target_body_id'] = m.body_id.to_numpy()[pairs.target_neuron_idx]
ca = pairs[pairs.blocked_contact_count > 0].sort_values('edge_id').copy()
eligible = np.sort(ca.source_neuron_idx.unique()).astype(np.int32)
all_pk = np.flatnonzero(np.isin(src, eligible) & np.isin(dst, kc))
whole = np.flatnonzero(np.isin(src, eligible))
assert set(ca.edge_id).issubset(all_pk) and set(all_pk).issubset(whole)
assert not np.any(np.isin(src[ca.edge_id], apl)) and not np.any(ca.retained_contact_count < 0)
for name, eids, count in [
('calyx_PN_KC_contact_block', ca.edge_id.to_numpy(np.int64), ca.blocked_contact_count.to_numpy(np.int64)),
('all_PN_KC_block', all_pk, raw[all_pk]), ('whole_PN_output_block', whole, raw[whole])]:
frame = pd.DataFrame({'edge_id': eids, 'source_neuron_idx': src[eids], 'target_neuron_idx': dst[eids],
'raw_synapse_count': raw[eids], 'blocked_contact_count': count})
frame['source_body_id'] = m.body_id.to_numpy()[src[eids]]
frame['target_body_id'] = m.body_id.to_numpy()[dst[eids]]
frame.to_parquet(exp/'masks'/f'{name}.parquet', index=False)
pairs.to_parquet(exp/'audit/all-PN-KC-contact-partitions.parquet', index=False)
allowed.to_parquet(exp/'audit/observed-APL-PN-ROI-pairs.parquet', index=False)
pns = m.iloc[np.sort(ap.target_neuron_idx.unique().astype(np.int32))].copy()
pns['eligible_for_diagnostic'] = pns.neuron_idx.isin(eligible)
pns.to_parquet(exp/'audit/all-259-APL-target-PNs.parquet', index=False)
roi = pk.assign(in_scope=keep).groupby(['primary_post', 'in_scope'], observed=True, dropna=False).size().reset_index(name='contacts')
roi.to_csv(exp/'audit/ROI-partition.csv', index=False)
summary = {'neurons': len(m), 'edges': len(dst), 'raw_contacts': int(raw.sum()), 'APL_target_PNs': len(pns),
'eligible_PNs': len(eligible), 'ineligible_PNs': int((~pns.eligible_for_diagnostic).sum()),
'calyx_mask_edges': len(ca), 'calyx_blocked_contacts': int(ca.blocked_contact_count.sum()),
'calyx_affected_KCs': int(ca.target_neuron_idx.nunique()), 'mixed_CA_nonCA_edges': int((ca.retained_contact_count > 0).sum()),
'calyx_other_contacts_retained_on_affected_edges': int(ca.retained_contact_count.sum()),
'all_PN_KC_edges': len(all_pk), 'all_PN_KC_contacts': int(raw[all_pk].sum()),
'whole_PN_edges': len(whole), 'whole_PN_contacts': int(raw[whole].sum()),
'unknown_ROI_contacts_in_original_PN_KC': int(pk.primary_post.eq('CentralBrain-unspecified').sum()),
'unknown_contacts_relabelled': 0, 'same_bouton_confirmed': False, 'biophysical_coefficient_known': False}
save(exp/'audit/scope-summary.json', summary)
inputs = [prior/'localization/APL-incident.parquet', prior/'localization/APL-target-PN-to-KC.parquet',
prior/'external-resources/synaptic-partners/download-verification.json',
can/'neuron-index/unified-metadata.parquet'] + list((can/'gpu').glob('csr-*.npy'))
save(exp/'protocol/input-hashes.json', [{'path': str(p.relative_to(root)), 'sha256': digest(p), 'bytes': p.stat().st_size} for p in inputs])
save(exp/'protocol/mask-hashes.json', {p.name: digest(p) for p in sorted((exp/'masks').iterdir())})
print(json.dumps(summary), flush=True)
def run():
cfg, root, model, exp, save, digest, _, _ = load_context()
pre = json.loads((exp/'protocol/preregistered.json').read_text())
assert all(digest(model/n) == h for n, h in pre['source_hashes'].items()), 'Source changed since preregistration'
assert not (exp/'regression/run-start.json').exists(), 'Refusing rerun into existing outputs'
assert all(digest(exp/'masks'/n) == h for n, h in json.loads((exp/'protocol/mask-hashes.json').read_text()).items())
q = queues(); runtime = root/cfg['runtime']
pc = yaml.safe_load((runtime/'phase4p.yaml').read_text())
os.environ['CUDA_VISIBLE_DEVICES'] = pc['device']['visible_gpu_uuid']
os.environ['CUDA_HOME'] = '/usr/local/cuda'; os.environ['TORCH_CUDA_ARCH_LIST'] = '8.9'; os.environ['MAX_JOBS'] = '2'
os.environ['PATH'] = str(root/'models/male-cns-lif/.venv/bin') + ':/usr/local/cuda/bin:' + os.environ['PATH']
sys.path.insert(0, str(root/'models/male-cns-lif')); sys.path.insert(0, str(runtime))
import torch
import phase4p, phase4v, phase4z
from phase4p_revalidate import DeterministicAPLSim, door_inputs, make_drive
from phase4r import arrhash
free, total = torch.cuda.mem_get_info(); assert free > 2*1024**3
save(exp/'regression/run-start.json', {'ComfyUI_queues': q, 'free_VRAM': free, 'total_VRAM': total,
'device': torch.cuda.get_device_name(), 'timestamp_utc': datetime.now(timezone.utc).isoformat()})
start = time.perf_counter(); torch.cuda.reset_peak_memory_stats()
rt = phase4p.CanonicalRuntime(pc, runtime, exp)
sim = DeterministicAPLSim(rt); sim.install(None)
assert len(sim.identity) == 14232 and sim.n == 166700 and len(rt.raw_np) == 25582938 and rt.delay_steps == 18
frozen = rt.weights.cpu().numpy().copy(); anatomical_hash = arrhash(rt.raw_np)
original_install = sim.install
apl_input_hash = arrhash(sim.input_gpu.cpu().numpy(), sim.output_gpu.cpu().numpy())
base_hash = arrhash(sim.base.cpu().numpy())
masks = {n: pd.read_parquet(exp/'masks'/f'{n}.parquet') for n in cfg['conditions'][1:]}
updates = {}
for name, mask in masks.items():
ids = mask.edge_id.to_numpy(np.int64)
values = blocked_weights(frozen[ids], rt.raw_np[ids], mask.blocked_contact_count.to_numpy(np.int64))
assert np.all(np.abs(values) <= np.abs(frozen[ids]))
updates[name] = (torch.as_tensor(ids, device=rt.device), torch.as_tensor(values, device=rt.device))
mask.assign(frozen_effective_weight=frozen[ids], diagnostic_effective_weight=values,
diagnostic_delta=(values.astype(float)-frozen[ids])).to_parquet(exp/'audit'/f'{name}-effective-overlay.parquet', index=False)
condition = 'frozen_reference'
def install(E):
assert E is None, 'No learning allowed'
original_install(None)
if condition != 'frozen_reference':
ix, values = updates[condition]; rt.weights[ix] = values
sim.install = install
vc = yaml.safe_load((root/cfg['input_model']/'phase4v.yaml').read_text())
assert vc['short_duration_ms'] == 120 and vc['observation_duration_ms'] == 1000
oldwatch = json.loads((root/cfg['event_prior']/'protocol/watched.json').read_text())
watch = np.asarray(oldwatch['all_watched'], np.int32)
targetkc = np.unique(sim.identity.loc[sim.identity.compartment.eq(vc['target_compartment']), 'source_neuron_idx'].to_numpy(np.int32))
oc, panel, vectors, selection, mapping = door_inputs(); panel = panel[panel.split == 'calibration'].reset_index(drop=True)
assert len(panel) == 6
save(exp/'protocol/calibration-panel.json', panel.to_dict('records'))
codes = phase4z.role_codes(sim.meta)
trials, regress, request_summary, populations, watched, hashes = [], [], [], [], [], []
baseline0 = None
try:
for oi in cfg['odor_indices']:
odor = panel.iloc[oi]
targets, rates = make_drive(oc, panel, vectors, selection, mapping, odor.odor_id, 'physiological')
forcing_seed = vc['seed_namespace'] + oi*vc['odor_seed_offset'] + cfg['seed']
force = phase4v.forcing(sim, targets, rates, forcing_seed, vc, 'offset'); ff = force.cpu().numpy()
hashes.append({'odor_index': oi, 'seed': forcing_seed, 'forcing': arrhash(ff), 'targets_rates': arrhash(targets, rates)})
np.savez_compressed(exp/'requests'/f'{oi}-scheduled.npz', forcing=ff, targets=targets, rates=rates)
requests0 = None
for condition in cfg['conditions']:
queues(); name = f'{oi}-{cfg["seed"]}-{condition}'
obs = phase4z.Observer(rt, watch, ff, targets); obs.attach()
try: result = phase4v.observed(sim, targets, force, targetkc, vc)
finally: obs.detach()
assert result['finite'] and np.isfinite(result['final_v']).all() and np.isfinite(result['final_g']).all()
ev = obs.events(); states = np.asarray(obs.states); req = pd.DataFrame(obs.request_rows)
assert len(req) == int(ff.sum())
assert np.array_equal(np.bincount(ev.neuron_idx, minlength=sim.n), result['counts'])
expected_weights = frozen.copy()
if condition != 'frozen_reference':
ids = masks[condition].edge_id.to_numpy(np.int64)
expected_weights[ids] = updates[condition][1].cpu().numpy()
assert np.array_equal(rt.weights.cpu().numpy(), expected_weights), 'Undeclared weight change'
row = {'trial': name, 'declared_weight_scope_exact': True}
if condition == 'frozen_reference':
old_ev = pd.read_parquet(root/cfg['event_prior']/'events'/f'{name}-whole-brain.parquet')
old_state = np.load(root/cfg['event_prior']/'states'/f'{name}-watched.npz')
old_final = np.load(root/cfg['state_prior']/'states'/f'{name}.npz')
row['prior_exact'] = {'events': ev.equals(old_ev), 'watched_states': np.array_equal(states, old_state['state']),
**{key: bool(np.array_equal(result[key], old_final[key])) for key in ('counts', 'final_v', 'final_g', 'trace')}}
assert all(row['prior_exact'].values()), row
requests0 = req
if oi == cfg['repeat_odor_index']: baseline0 = ev, states, req, result
paired = requests0.merge(req, on=['step', 'neuron_idx'], suffixes=('_ref', '_diagnostic'), validate='one_to_one', how='outer', indicator=True)
assert paired['_merge'].eq('both').all()
request_summary.append({'trial': name, 'scheduled': len(req), 'identical_request_keys': True,
'accepted': int(req.status.eq('accepted').sum()), 'blocked': int(req.status.eq('blocked').sum()), 'preexisting': int(req.status.eq('preexisting').sum()),
'status_differences_vs_reference': int((paired.status_ref != paired.status_diagnostic).sum()),
'actual_output_differences_vs_reference': int((paired.actual_output_spike_ref != paired.actual_output_spike_diagnostic).sum())})
late = ev[ev.step >= round(cfg['late_start_ms']/rt.dt)]
period_count = np.bincount(late.neuron_idx, minlength=sim.n)
for code in np.unique(codes):
nodes = np.flatnonzero(codes == code)
populations.append({'trial': name, 'population': str(code), 'neurons': len(nodes), 'active': int((result['counts'][nodes] > 0).sum()),
'spikes': int(result['counts'][nodes].sum()), 'late_spikes': int(period_count[nodes].sum())})
for node in watch:
watched.append({'trial': name, 'neuron_idx': int(node), 'body_id': int(sim.meta.iloc[node].body_id),
'type': str(sim.meta.iloc[node].annotation_type), 'population': str(codes[node]),
'spikes': int(result['counts'][node]), 'late_spikes': int(period_count[node]),
'late_rate_hz': float(period_count[node]/((1000-cfg['late_start_ms'])/1000))})
trials.append({'trial': name, 'odor_index': oi, 'odor_name': str(odor.odor_name), 'seed': cfg['seed'], 'condition': condition,
'whole_spikes': len(ev), 'KC_active': int(np.count_nonzero(result['counts'][sim.kc])),
'KC_active_fraction': float(np.count_nonzero(result['counts'][sim.kc])/len(sim.kc)),
'KC_spikes': int(result['counts'][sim.kc].sum()), 'KC_late_spikes': int(period_count[sim.kc].sum()),
'KC82312_late_spikes': int(period_count[cfg['target_KC']]), 'VA6_late_spikes': int(period_count[cfg['watched_VA6']]),
'finite': True, 'wall_seconds': result['wall_seconds'], 'biological_ms': 1000})
ev.to_parquet(exp/'events'/f'{name}.parquet', index=False)
req.to_parquet(exp/'requests'/f'{name}.parquet', index=False)
pd.DataFrame(result['samples']).to_parquet(exp/'observations'/f'{name}.parquet', index=False)
np.savez_compressed(exp/'states'/f'{name}.npz', counts=result['counts'], final_v=result['final_v'], final_g=result['final_g'], trace=result['trace'])
np.savez_compressed(exp/'states'/f'{name}-watched.npz', state=states, watch=watch)
if oi == cfg['repeat_odor_index']:
queues(); again = phase4z.Observer(rt, watch, ff, targets); again.attach()
try: repeat = phase4v.observed(sim, targets, force, targetkc, vc)
finally: again.detach()
row['repeat_exact'] = {'events': ev.equals(again.events()), 'states': np.array_equal(states, again.states),
'requests': req.equals(pd.DataFrame(again.request_rows)),
**{key: bool(np.array_equal(result[key], repeat[key])) for key in ('counts', 'final_v', 'final_g', 'trace')}}
assert all(row['repeat_exact'].values()), row
regress.append(row); save(exp/'regression'/f'{name}.json', row)
print(json.dumps(trials[-1]), flush=True)
# Fixed-event replay is an independent direct-operator check, not the dynamic whole-brain result.
replay_check(cfg, exp, sim, masks, frozen, baseline0, watch, phase4z, save)
finally:
sim.install = original_install; original_install(None)
assert np.array_equal(rt.weights.cpu().numpy(), frozen)
assert arrhash(rt.raw_np) == anatomical_hash and arrhash(sim.base.cpu().numpy()) == base_hash
assert arrhash(sim.input_gpu.cpu().numpy(), sim.output_gpu.cpu().numpy()) == apl_input_hash
save(exp/'regression/restored.json', {'base': True, 'raw': True, 'APL_matrices': True, 'frozen_effective_weights': True})
for name, rows in [('trials', trials), ('request-comparison', request_summary), ('population-counts', populations), ('watched-counts', watched)]:
pd.DataFrame(rows).to_csv(exp/'audit'/f'{name}.csv', index=False)
save(exp/'protocol/forcing-hashes.json', hashes)
# Separate reference trials do not include the graded APL overlay, exactly as archived.
rt.weights.copy_(sim.base)
reference = []
for spec in pc['reference_trials']:
queues(); ref = root/'experiments/phase4p-deterministic-runtime/phase4c-v2/reference'/spec['name']
ev = pd.read_parquet(ref/'spike-events.parquet')
expected = {'step': ev.step.to_numpy(), 'neuron_idx': ev.neuron_idx.to_numpy(),
'counts': np.load(ref/'per-neuron-spike-counts.int32.npy'), 'v': np.load(ref/'final-voltage.float32.npy'), 'g': np.load(ref/'final-g.float32.npy')}
r = rt.run(np.flatnonzero(rt.pops[spec['population']]), spec['rate_hz'], spec['seed'], spec['duration_ms'])
eq = {key: bool(np.array_equal(r[key], value)) for key, value in expected.items()}
assert all(eq.values()), eq
reference.append({'trial': spec['name'], 'exact': eq, 'wall_seconds': r['metrics']['wall_seconds']})
original_install(None)
save(exp/'regression/five-frozen-trials.json', {'pass': True, 'trials': reference, 'original_v1_preserved': True, 'oracle': 'frozen deterministic-v2'})
save(exp/'performance/summary.json', {'wall_seconds_including_build': time.perf_counter()-start,
'peak_VRAM_bytes': int(torch.cuda.max_memory_allocated()), 'device': torch.cuda.get_device_name(),
'diagnostic_overhead_included': True, 'not_runtime_benchmark': True, 'ComfyUI_models_unloaded': False})
save(exp/'audit/run-complete.json', {'trials': len(trials), 'exact_repeat_trials': 4, 'old_trial_reproductions': 6,
'frozen_reference_trials': 5, 'all_finite': True, 'candidate': False, 'training': False})
def replay_check(cfg, exp, sim, masks, frozen, baseline, watch, phase4z, save):
ev, state, req, result = baseline; rt = sim.rt; T = len(state)
ids = np.flatnonzero(np.isin(rt.indices_np, watch))
edges = pd.DataFrame({'edge_id': ids, 'source_neuron_idx': sim.sources[ids], 'target_neuron_idx': rt.indices_np[ids], 'weight': frozen[ids]})
arrivals = ev.merge(edges, left_on='neuron_idx', right_on='source_neuron_idx')
arrivals['arrival_step'] = arrivals.step + rt.delay_steps
arrivals = arrivals[arrivals.arrival_step < T].sort_values(['target_neuron_idx', 'arrival_step', 'edge_id'], kind='stable')
rows = []
for wi, node in enumerate(watch):
subset = arrivals[arrivals.target_neuron_idx == node].copy()
after = np.r_[state[1:, 1, wi], result['final_g'][node]]
overlay = after.astype(float)-state[:, 5, wi].astype(float)
request = np.zeros(T, bool); rq = req[req.neuron_idx == node]; request[rq.step.to_numpy(np.int32)] = True
base_arrival = state[:, 3, wi]
for name in cfg['conditions']:
weights = subset.weight.to_numpy(np.float32).copy()
if name != 'frozen_reference':
frame = masks[name]; eids = frame.edge_id.to_numpy(np.int64)
values = blocked_weights(frozen[eids], rt.raw_np[eids], frame.blocked_contact_count.to_numpy(np.int64))
lookup = pd.Series(values, index=eids)
mapped = subset.edge_id.map(lookup); keep = mapped.notna().to_numpy()
weights[keep] = mapped[keep].to_numpy(np.float32)
subset['diagnostic_weight'] = weights
a = np.zeros(T, np.float32)
for step, group in subset.groupby('arrival_step', sort=False):
a[int(step)] = np.cumsum(group.diagnostic_weight.to_numpy(np.float32), dtype=np.float32)[-1]
rr = phase4z.replay(a, overlay, request, rt)
if name == 'frozen_reference':
exact = np.array_equal(a, base_arrival) and np.array_equal(rr[0], ev.loc[ev.neuron_idx == node, 'step']) and np.array_equal(rr[1], state[:, 4:7, wi])
assert exact and rr[2] == result['final_v'][node] and rr[3] == result['final_g'][node]
unchanged = np.array_equal(a, base_arrival)
if name == 'calyx_PN_KC_contact_block' and node not in sim.kc: assert unchanged
rows.append({'condition': name, 'neuron_idx': int(node), 'arrival_unchanged': bool(unchanged),
'arrival_L1_difference': float(np.abs(a.astype(float)-base_arrival).sum()),
'conditional_late_spikes': int((rr[0] >= round(cfg['late_start_ms']/rt.dt)).sum()),
'all_other_recorded_events_and_APL_fixed': True, 'whole_brain_causality_claim': False})
pd.DataFrame(rows).to_csv(exp/'replay/fixed-event-scope.csv', index=False)
save(exp/'regression/fixed-event-replay.json', {'baseline_exact_neurons': len(watch), 'terminal_nonKC_arrivals_exact': True})
def finalize():
cfg, root, model, exp, save, digest, immutable_check, seal = load_context()
assert (exp/'audit/run-complete.json').exists()
pre = json.loads((exp/'protocol/preregistered.json').read_text())
assert all(digest(model/n) == h for n, h in pre['source_hashes'].items())
post = integrity(cfg, root, digest, immutable_check)
assert post == json.loads((exp/'regression/preflight.json').read_text())
save(exp/'regression/final-integrity.json', post)
q = queues(); save(exp/'audit/final-service-health.json', {'ComfyUI_queues': q, 'services_stopped': False})
trials = pd.read_csv(exp/'audit/trials.csv'); scope = json.loads((exp/'audit/scope-summary.json').read_text())
requests = pd.read_csv(exp/'audit/request-comparison.csv')
lines = ['# Phase 4AB — calyx contact-scope diagnostic', '',
'Status: COMPLETE-SCOPE-COUNTERFACTUAL / NO-DYNAMICS-CANDIDATE.',
'Diagnostic interventions only; no calibrated biological inhibition, no training, no parameter winner.', '',
f"Eligible PNs: {scope['eligible_PNs']}/259. CA mask: {scope['calyx_mask_edges']} edges / {scope['calyx_blocked_contacts']} contacts.",
f"Affected KCs: {scope['calyx_affected_KCs']}. Mixed-ROI affected edges: {scope['mixed_CA_nonCA_edges']}; retained other contacts: {scope['calyx_other_contacts_retained_on_affected_edges']}.",
f"Broader PN→KC control: {scope['all_PN_KC_edges']} edges / {scope['all_PN_KC_contacts']} contacts. All-fast-PN-output control: {scope['whole_PN_edges']} edges / {scope['whole_PN_contacts']} contacts.", '',
'| Odor | Condition | Active KC | KC late spikes | KC82312 late | VA6 late |',
'|---|---|---:|---:|---:|---:|']
for r in trials.itertuples():
lines.append(f'| {r.odor_name} | {r.condition} | {r.KC_active} | {r.KC_late_spikes} | {r.KC82312_late_spikes} | {r.VA6_late_spikes} |')
lines += ['', 'Late means 500–1,000 ms; 120 ms old calibration odor input; one seed, six old odors. No held-out generalization claim.',
f"Identical scheduled requests across conditions. Delivered request-associated output differences versus reference: {int(requests.actual_output_differences_vs_reference.sum())} (summed over all comparisons). See per-trial table; this is state/refractory feedback, not a changed RNG.", '',
'## Interpretation limits', '',
'CA selection uses exact primary_post labels plus matched APL→PN/ROI metadata, not bouton identities or inferred receptors. Unknown ROIs are not reclassified.',
'Whole-PN control removes fast output, not somatic spikes or the frozen graded APL input. It is not a complete PN knockout.',
'Reduction/silencing is not restored information or associative memory. No condition becomes a formal model. Fixed-event replay freezes other recorded events/APL; dynamic trials allow natural recurrent feedback.',
'Local-inhibition motivation: [Prisco et al.](https://doi.org/10.7554/eLife.74172), [Amin et al.](https://doi.org/10.7554/eLife.56954). These sources do not establish the complete-block operator as physiological.', '',
'## Verification', '',
'Six original trials reproduced exactly (whole spike events, watched state and final counts/V/g). Four first-odor repeats exact. Five deterministic-v2 frozen reference trials exact. Finite state throughout.',
'Original 230,239 files and both Phase 4AA manifests rechecked before/after. Base CSR/count/sign, frozen LIF/APL, eligibility-v2 and 14,232 plastic mask preserved.',
'Timing includes diagnostic CPU/GPU observation overhead and is not a runtime speed benchmark. ComfyUI was not stopped or unloaded.',
'Top-level acceptance gates remain 5/10 (50%); anatomical diagnosis is not evidence of learning success.']
(exp/'REPORT.md').write_text('\n'.join(lines)+'\n', encoding='utf-8')
save(exp/'audit/final-summary.json', {'status': 'COMPLETE-SCOPE-COUNTERFACTUAL / NO-DYNAMICS-CANDIDATE',
'integrity_pass': True, 'reference_pass': True, 'scope': scope, 'candidate': False, 'training': False,
'preserved_acceptance_gates': '5/10', 'frozen_baseline_modified': False})
for folder in (model, exp):
count = seal(folder); assert manifest_check(folder, digest)['pass']; print('SEALED', folder, count, flush=True)
if __name__ == '__main__':
parser = argparse.ArgumentParser(); parser.add_argument('task', choices=['prepare', 'run', 'finalize'])
globals()[parser.parse_args().task]()
@@ -0,0 +1,22 @@
"""Record executable/package/hardware versions without installing anything."""
import importlib.metadata
import platform
from pathlib import Path
import subprocess
import sys
MODEL = Path('/data/digital-drosophila/models/male-cns-lif/terminal-scope-diagnostic-v1')
EXP = Path('/data/digital-drosophila/experiments/phase4ab-terminal-scope')
sys.path.insert(0, str(MODEL))
import diagnostic
_, _, _, _, save, digest, _, _ = diagnostic.load_context()
packages = {}
for name in ('torch', 'numpy', 'pandas', 'pyarrow', 'scipy', 'PyYAML', 'ninja'):
try: packages[name] = importlib.metadata.version(name)
except importlib.metadata.PackageNotFoundError: packages[name] = None
gpu = subprocess.check_output(['nvidia-smi', '--query-gpu=index,name,uuid,driver_version,memory.total', '--format=csv,noheader'], text=True)
nvcc = subprocess.check_output(['/usr/local/cuda/bin/nvcc', '--version'], text=True)
save(EXP/'protocol/environment.json', {'python': sys.version, 'executable': sys.executable,
'platform': platform.platform(), 'packages': packages, 'GPU_inventory': gpu, 'nvcc': nvcc,
'installed_or_upgraded_packages': False, 'source_sha256': digest(Path(__file__))})
print(packages)
@@ -0,0 +1,32 @@
"""Explicit frozen module paths; repairs launch only, never scientific source."""
import os
from pathlib import Path
import sys
ROOT = Path('/data/digital-drosophila')
MODEL = ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'
EXP = ROOT/'experiments/phase4ab-terminal-scope'
PARENT = ROOT/'models/male-cns-lif'
names = ['deterministic-runtime-v2', 'credit-v3', 'timing-audit-v1', 'timing-credit-v1',
'timing-stability-v1', 'odor-timing-generalization-v1', 'csus-audit-v1',
'late-kc-audit-v1', 'dpm-hypothesis-v1', 'pn-feedback-audit-v1', 'transmitter-audit-v1']
sys.path[:0] = [str(MODEL), str(PARENT)] + [str(PARENT/n) for n in names]
os.environ['PYTHONDONTWRITEBYTECODE'] = '1'
import diagnostic
_, _, _, _, save, digest, _, _ = diagnostic.load_context()
save(EXP/'protocol/launch-addendum.json', {
'previous_attempt': 'ModuleNotFoundError: phase4v before run-start and before simulation',
'only_fix': 'Add explicit existing frozen dependency directories to module lookup',
'model_source_edited': False, 'protocol_parameters_changed': False,
'launcher_sha256': digest(Path(__file__)), 'module_paths': [str(PARENT/n) for n in names]})
try:
diagnostic.run()
finally:
loaded = {}
for name, mod in list(sys.modules.items()):
filename = getattr(mod, '__file__', None)
if filename:
p = Path(filename)
if p.is_file() and p.is_relative_to(PARENT) and '.venv' not in p.parts:
loaded[name] = {'path': str(p.relative_to(ROOT)), 'sha256': digest(p)}
save(EXP/'protocol/imported-code-hashes.json', loaded)
@@ -0,0 +1,60 @@
{
"created_utc": "2026-09-29T16:26:06.650270+00:00",
"files": [
{
"bytes": 4956,
"path": "PROTOCOL.md",
"sha256": "451b53b128eb0ad67acb81f060f3676794a68c98637dd2c0eddd137da509fccf"
},
{
"bytes": 2083,
"path": "README.md",
"sha256": "2842b5dd94ed30fc01d1a8b8d95e235de78695d3d1c1076402499276fb903261"
},
{
"bytes": 6922,
"path": "analyze.py",
"sha256": "33ba5f323e9ed3343f43a8d64fb7e862c8a4f1118b9b441f4745e26cc14303a8"
},
{
"bytes": 835,
"path": "config.yaml",
"sha256": "e22088d9847504a50c0e684d2cee9420f484ab1ebef9d6057921062d8f785e73"
},
{
"bytes": 28229,
"path": "diagnostic.py",
"sha256": "c8ec2ae6e72d35fedbd42acc6e6b5e7af2634d5b9805dc20e777b172c55d5aa4"
},
{
"bytes": 1193,
"path": "environment.py",
"sha256": "be92ce24e63910f67da0238b19e3abb882e1afd317561fccf01734f1b4a56f25"
},
{
"bytes": 1608,
"path": "launch.py",
"sha256": "0b2f71365267886933ba5becb52f4123d3d4e95e9c5bc0cf19c9bf0563831625"
},
{
"bytes": 1644,
"path": "publish.py",
"sha256": "bc194bb857336908b1c3fe71cc0e7b7fe4152794745bd3443533d47da886f5ec"
},
{
"bytes": 2797,
"path": "scope.py",
"sha256": "8f5774496dcceaee1c2793a5308754cbd0864c4a38d507481071d988f9de424d"
},
{
"bytes": 2615,
"path": "test_scope.py",
"sha256": "38a1c6e653ae97e77e5feda3ee8f2f8a3c366755bfbbd57ca1373b9b8f971bbb"
},
{
"bytes": 2032,
"path": "validate.py",
"sha256": "971136a667367a84cfc50ebc3fa73a749eddb8b8e1ca6931708ba67bfc12f0c1"
}
]
}
@@ -0,0 +1,29 @@
"""Copy only new small sealed evidence into Git; never copy bulk datasets."""
import json
from pathlib import Path
import shutil
import sys
ROOT = Path('/data/digital-drosophila'); MODEL = ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'
EXP = ROOT/'experiments/phase4ab-terminal-scope'
sys.path.insert(0, str(MODEL))
import diagnostic
_, _, _, _, save, digest, _, _ = diagnostic.load_context()
for folder in (MODEL, EXP):
assert diagnostic.manifest_check(folder, digest)['pass']
registry = ROOT/'registry/phase4ab-terminal-scope'; registry.mkdir(exist_ok=False)
records = []
for p in sorted(EXP.rglob('*')):
if not p.is_file() or p.suffix not in ('.json', '.csv', '.md', '.sha256') or p.stat().st_size > 4*1024**2:
continue
dst = registry/'evidence'/p.relative_to(EXP); dst.parent.mkdir(parents=True, exist_ok=True)
with p.open('rb') as src, dst.open('xb') as out: shutil.copyfileobj(src, out)
assert digest(p) == digest(dst)
records.append({'source': str(p.relative_to(ROOT)), 'copy': str(dst.relative_to(registry)),
'sha256': digest(p), 'bytes': p.stat().st_size})
save(registry/'evidence-index.json', records)
save(registry/'seal-verification.json', {str(p.relative_to(ROOT)): diagnostic.manifest_check(p, digest) for p in (MODEL, EXP)})
for source, dest in [('INTERPRETATION.zh.md', 'phase4ab-terminal-scope.md'), ('REPORT.md', 'phase4ab-technical-results.md')]:
target = ROOT/'docs/progress'/dest
with (EXP/source).open('rb') as src, target.open('xb') as out: shutil.copyfileobj(src, out)
print('Published', len(records), 'small evidence copies; no data copied to Git')
@@ -0,0 +1,53 @@
"""Pure CPU contact accounting and immutable diagnostic weight operations."""
import numpy as np
import pandas as pd
def contact_scope(apl_pn, pn_kc, rois):
"""Only exact, observed PN+postsynaptic-ROI pairs; no side/name inference."""
required = {'source_neuron_idx', 'target_neuron_idx', 'primary_post'}
if not required.issubset(apl_pn) or not required.issubset(pn_kc):
raise ValueError('Actual schema lacks required identifiers/primary_post')
permitted = apl_pn.loc[apl_pn.primary_post.isin(rois),
['target_neuron_idx', 'primary_post']].drop_duplicates()
permitted = permitted.rename(columns={'target_neuron_idx': 'source_neuron_idx'})
keys = pd.MultiIndex.from_frame(permitted.astype({'primary_post': 'string'}))
observed = pd.MultiIndex.from_frame(pn_kc[['source_neuron_idx', 'primary_post']]
.astype({'primary_post': 'string'}))
keep = observed.isin(keys)
pairs = ['source_neuron_idx', 'target_neuron_idx']
total = pn_kc.groupby(pairs, observed=True).size().rename('raw_partner_count')
ca = pn_kc.loc[keep].groupby(pairs, observed=True).size().rename('blocked_contact_count')
frame = total.to_frame().join(ca).fillna(0).reset_index()
for key in pairs + ['raw_partner_count', 'blocked_contact_count']:
frame[key] = frame[key].astype(np.int64)
frame['retained_contact_count'] = frame.raw_partner_count - frame.blocked_contact_count
assert (frame.retained_contact_count >= 0).all()
return frame, permitted, keep
def edge_ids_for_pairs(frame, ptr, dst):
out = []
for s, t in frame[['source_neuron_idx', 'target_neuron_idx']].itertuples(index=False, name=None):
ids = np.arange(ptr[s], ptr[s + 1], dtype=np.int64)
found = ids[dst[ids] == t]
if len(found) != 1:
raise ValueError(f'Expected exactly one existing CSR edge for {s}->{t}')
out.append(found[0])
return np.asarray(out, np.int64)
def blocked_weights(base, raw, contacts):
base = np.asarray(base)
raw = np.asarray(raw)
contacts = np.asarray(contacts)
if base.dtype != np.float32 or raw.dtype != np.int64 or contacts.dtype != np.int64:
raise TypeError('Expected float32 effective weight and int64 anatomical counts')
if base.shape != raw.shape or raw.shape != contacts.shape:
raise ValueError('Shape mismatch')
if not np.isfinite(base).all() or np.any(raw <= 0) or np.any(contacts < 0) or np.any(contacts > raw):
raise ValueError('Invalid contact partition or weights')
out = base.copy()
m = contacts > 0 # No-op copies original bits; no rounding at strength zero.
out[m] = (base[m].astype(np.float64) * (raw[m] - contacts[m]) / raw[m]).astype(np.float32)
return out
@@ -0,0 +1,52 @@
import unittest
import numpy as np
import pandas as pd
from scope import contact_scope, edge_ids_for_pairs, blocked_weights
class ScopeTests(unittest.TestCase):
def test_roi_and_side_not_guessed(self):
apl = pd.DataFrame({'source_neuron_idx': [0, 0, 1], 'target_neuron_idx': [3, 3, 4],
'primary_post': ['CA(R)', 'CentralBrain-unspecified', 'CA(L)']})
pn = pd.DataFrame({'source_neuron_idx': [3, 3, 3, 4, 4], 'target_neuron_idx': [5]*5,
'primary_post': ['CA(R)', 'CA(L)', None, 'CA(R)', 'CA(L)']})
pairs, allowed, keep = contact_scope(apl, pn, ['CA(L)', 'CA(R)'])
self.assertEqual(keep.tolist(), [True, False, False, False, True])
self.assertEqual(pairs.blocked_contact_count.tolist(), [1, 1])
self.assertEqual(pairs.retained_contact_count.tolist(), [2, 1])
self.assertEqual(len(allowed), 2)
def test_zero_identity_and_nonmutation(self):
w = np.array([.275, -3.14159, 0, -0.0], np.float32)
raw = np.array([1, 43, 10, 1], np.int64)
old = w.tobytes(), raw.tobytes()
result = blocked_weights(w, raw, np.zeros(4, np.int64))
self.assertEqual(w.tobytes(), result.tobytes())
self.assertEqual((w.tobytes(), raw.tobytes()), old)
def test_partial_and_full(self):
w = np.array([2., -4., 0., 1.], np.float32)
raw = np.array([4, 4, 3, 7], np.int64)
result = blocked_weights(w, raw, np.array([1, 4, 2, 0], np.int64))
np.testing.assert_array_equal(result, np.array([1.5, -0., 0., 1.], np.float32))
np.testing.assert_array_equal(w, np.array([2., -4., 0., 1.], np.float32))
def test_invalid_partition_rejected(self):
for bad in [-1, 3]:
with self.assertRaises(ValueError):
blocked_weights(np.array([1.], np.float32), np.array([2], np.int64), np.array([bad], np.int64))
def test_existing_edge_only(self):
p = pd.DataFrame({'source_neuron_idx': [0, 1], 'target_neuron_idx': [2, 0]})
ptr = np.array([0, 2, 3, 3], np.int64); dst = np.array([1, 2, 0], np.int32)
np.testing.assert_array_equal(edge_ids_for_pairs(p, ptr, dst), [1, 2])
p.loc[1, 'target_neuron_idx'] = 2
with self.assertRaises(ValueError): edge_ids_for_pairs(p, ptr, dst)
def test_duplicates_rejected(self):
p = pd.DataFrame({'source_neuron_idx': [0], 'target_neuron_idx': [1]})
with self.assertRaises(ValueError):
edge_ids_for_pairs(p, np.array([0, 2, 2], np.int64), np.array([1, 1], np.int32))
if __name__ == '__main__': unittest.main()
@@ -0,0 +1,38 @@
"""Additional adversarial operator tests and durable unit-test record."""
import io
import json
from pathlib import Path
import sys
import unittest
import numpy as np
MODEL = Path('/data/digital-drosophila/models/male-cns-lif/terminal-scope-diagnostic-v1')
EXP = Path('/data/digital-drosophila/experiments/phase4ab-terminal-scope')
sys.path.insert(0, str(MODEL))
from scope import blocked_weights
import diagnostic
_, _, _, _, save, digest, _, _ = diagnostic.load_context()
stream = io.StringIO()
suite = unittest.defaultTestLoader.discover(str(MODEL), pattern='test_scope.py')
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)
assert result.wasSuccessful(), stream.getvalue()
rng = np.random.default_rng(41337)
raw = rng.integers(1, 100000, size=10000, dtype=np.int64)
contacts = np.asarray([rng.integers(0, n+1) for n in raw], dtype=np.int64)
base = (raw.astype(np.float32)*np.float32(.275)*rng.choice([-1, 0, 1], size=len(raw))).astype(np.float32)
raw_bytes, base_bytes = raw.tobytes(), base.tobytes()
out = blocked_weights(base, raw, contacts)
assert np.isfinite(out).all() and np.all(np.abs(out) <= np.abs(base))
assert np.all(out[contacts == raw] == 0)
assert out[contacts == 0].tobytes() == base[contacts == 0].tobytes()
assert np.all(np.sign(out[out != 0]) == np.sign(base[out != 0]))
assert raw_bytes == raw.tobytes() and base_bytes == base.tobytes()
for bad in [np.array([np.inf], np.float32), np.array([np.nan], np.float32)]:
try: blocked_weights(bad, np.array([2], np.int64), np.array([1], np.int64))
except ValueError: pass
else: raise AssertionError('nonfinite accepted')
save(EXP/'regression/unit-tests.json', {'pass': True, 'unit_tests': result.testsRun, 'fuzz_cases': len(raw),
'fuzz_seed': 41337, 'finite': True, 'never_sign_flip_or_amplify': True, 'input_nonmutation': True,
'zero_identity_bitwise': True, 'unittest_log': stream.getvalue(), 'source_sha256': digest(Path(__file__))})
print('PASS', result.testsRun, 'unit tests +', len(raw), 'randomized operator cases')