Audit calyx-specific PN contact scope without changing frozen baseline
This commit is contained in:
@@ -0,0 +1,86 @@
|
||||
# Phase 4AB — calyx contact-scope counterfactuals
|
||||
|
||||
This is an exploratory mechanism diagnostic, NOT an APL physiological model,
|
||||
learning experiment, parameter calibration or new formal runtime candidate.
|
||||
All previous scientific artifacts remain read-only. Only new output directories
|
||||
are permitted. No additional download is required.
|
||||
|
||||
## Question and predeclared controls
|
||||
|
||||
Does removal of anatomically labelled calyx PN→KC input alter persistent KC
|
||||
activity without also removing the same PNs' antennal-lobe output?
|
||||
|
||||
Select PN/ROI pairs from actual Phase 4AA APL→ALPN synaptic contacts whose
|
||||
`primary_post` is exactly `CA(L)` or `CA(R)`. Select PN→KC contacts only when
|
||||
their source PN and actual `primary_post` match that pair. This is **ROI-level
|
||||
co-location**, not proof of the same bouton, receptor, physiological coefficient,
|
||||
or even a per-presynaptic-site ROI annotation. `primary_post` labels the
|
||||
postsynaptic site. Never reclassify `CentralBrain-unspecified` by proximity.
|
||||
|
||||
Compare four conditions on the same eligible PN set:
|
||||
|
||||
1. Frozen reference; no changed weights or injected current.
|
||||
2. Block only the matched CA PN→KC contact component.
|
||||
3. Block all PN→KC output of those PNs, irrespective of ROI.
|
||||
4. Block all fast outgoing synaptic weights of those PNs (broad scope control).
|
||||
|
||||
These are complete-block engineering interventions, not estimates of biological
|
||||
APL inhibition. There is no strength sweep or winner selection. Condition 4
|
||||
does **not** silence PN somata or alter their force requests. The frozen graded
|
||||
APL input matrix remains active in all four conditions; condition 4 is therefore
|
||||
not a complete PN knockout. All fast edge slots remain present. Output masks are
|
||||
not plastic masks, and nothing is written back to base CSR or an old model.
|
||||
|
||||
For condition 2, the in-memory diagnostic effective weight is
|
||||
`float32(float64(frozen_effective_weight) * (raw_count - CA_count) / raw_count)`.
|
||||
The linear per-contact decomposition is an engineering consequence of the
|
||||
existing linear weight model, not a measured transmitter-release law.
|
||||
At strength zero there is no arithmetic or write at all; identity must be exact.
|
||||
Conditions 3/4 set only the declared fast effective weights to zero. Anatomical
|
||||
counts, transmitter signs, LIF/refractory/delay, APL matrices and calibration,
|
||||
eligibility-v2 and the 14,232 plastic-edge mask remain frozen.
|
||||
|
||||
## Trials and measurements, declared before results
|
||||
|
||||
Use all six old calibration odors, seed 131101, the frozen Phase 4V DoOR→PN
|
||||
request generator, 120 ms stimulus followed by observation to 1,000 ms. Do not
|
||||
consume held-out odors. Reset all states between trials. Repeat every condition
|
||||
for odor 0 to test exact determinism. No DAN reinforcement or weight training.
|
||||
|
||||
Record all spike events, all final counts/V/g, watched PN/KC/ALLN states, original
|
||||
force requests and their accepted/preexisting/blocked status, KC active fraction,
|
||||
late KC82312/VA6/ALLN activity (500–1,000 ms), and class-wise counts. All conditions
|
||||
share requested inputs; delivered events need not match because recurrent state
|
||||
and refractory periods can change. Explicitly report this distinction.
|
||||
|
||||
Unmodified trials must reproduce Phase 4Z whole events, watched states and
|
||||
Phase 4Y final arrays bit-for-bit. Each first-odor repeat must be exact. Run the
|
||||
five frozen Phase 4C trials against the deterministic-v2 archived oracle in new
|
||||
output paths. Original v1 files remain preserved, not relabelled as deterministic.
|
||||
|
||||
Additionally, replay the first frozen trial with its recorded events fixed, using
|
||||
the frozen ordered reduction and 18-step queue to validate locality: no terminal
|
||||
mask affects non-KC arrivals; broad controls may do so. This fixed-event test
|
||||
only measures the direct operator, not whole-brain causality.
|
||||
|
||||
## Interpretation gate
|
||||
|
||||
Reduced KC activity alone is not repaired encoding or memory. A broad blockade
|
||||
can destroy useful drive. If CA blockade fails to remove the late KC response,
|
||||
report that it is insufficient under this model; do not increase gain or relabel
|
||||
unknown contacts. If it succeeds while ALPN/ALLN persist, report scope-specific
|
||||
model counterfactual evidence, not proof that biological APL has that strength.
|
||||
No result in this phase can generate a learned/physiological candidate.
|
||||
|
||||
## Source boundaries
|
||||
|
||||
- MaleCNS official v1.0 syn-partners, CC BY 4.0, already verified in Phase 4AA:
|
||||
https://male-cns.janelia.org/download/
|
||||
- Prisco et al. (2021), doi:10.7554/eLife.74172: local calyx APL interactions;
|
||||
does not assign MaleCNS edge-specific inhibition coefficients.
|
||||
- Amin et al. (2020), doi:10.7554/eLife.56954: spatially localized APL activity
|
||||
and inhibition; does not validate the present engineering complete-block rule.
|
||||
|
||||
Unknown: bouton identity, receptor distribution, strength/time course of terminal
|
||||
release modulation, and whether adding such physiology would rescue memory.
|
||||
No game, decoder, RL, visual input, threshold/eta tuning, or next-phase training.
|
||||
@@ -0,0 +1,39 @@
|
||||
# Terminal-scope diagnostic v1 (Phase 4AB)
|
||||
|
||||
Independent engineering counterfactual harness, **not a formal brain model**.
|
||||
Read `PROTOCOL.md` before interpreting any result. Frozen input datasets and
|
||||
scientific sources are mandatory; no data are fetched by this stage.
|
||||
|
||||
## Sequence
|
||||
|
||||
Use the existing scientific environment with `PYTHONDONTWRITEBYTECODE=1`,
|
||||
`OMP_NUM_THREADS=4`, `OPENBLAS_NUM_THREADS=4`, `MKL_NUM_THREADS=4`:
|
||||
|
||||
1. `python -B -m unittest -v test_scope`
|
||||
2. `python -B diagnostic.py prepare` (new empty experiment directory only)
|
||||
3. `python -B validate.py`
|
||||
4. `python -B launch.py` (explicit frozen module paths, single 4090)
|
||||
5. `python -B environment.py`
|
||||
6. `python -B analyze.py`
|
||||
7. `python -B diagnostic.py finalize`
|
||||
8. `python -B publish.py` (new small Git evidence copies only)
|
||||
|
||||
The completed output is sealed. **Do not rerun these commands into its directory.**
|
||||
Unit tests may be rerun read-only. An independent full replication requires a new
|
||||
isolated experiment/model directory, with its paths changed consistently in the
|
||||
copied configuration/entry points before preregistration. Retain this original
|
||||
source and every old output. The current protocol also includes four in-run exact
|
||||
repeats, six historical exact comparisons and five archived reference checks.
|
||||
|
||||
`launch.py` fixes the initial import-path failure without editing the five
|
||||
preregistered source/config files. `analyze.py` is a post-hoc summary only and
|
||||
does not launch any additional experiment or fit any parameter. Additional helper
|
||||
hashes and imported code hashes are captured in the final manifest/protocol.
|
||||
|
||||
The existing environment is reused, never upgraded. ComfyUI queues are checked
|
||||
between trials; a busy service causes deferral, not eviction. Performance includes
|
||||
observation overhead. The historical APL graded input/output remains unchanged.
|
||||
|
||||
All-fast-PN-output blockade is not somatic silencing and not a complete knockout:
|
||||
PN spikes still contribute to the frozen graded APL input. `primary_post` is a
|
||||
postsynaptic ROI label, not a reconstructed bouton identity. Unknown stays unknown.
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Post-hoc interpretation only; no further trials, selection or parameter changes."""
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
ROOT = Path('/data/digital-drosophila'); MODEL = ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'
|
||||
EXP = ROOT/'experiments/phase4ab-terminal-scope'
|
||||
sys.path.insert(0, str(MODEL))
|
||||
import diagnostic
|
||||
cfg, _, _, _, save, digest, _, _ = diagnostic.load_context()
|
||||
assert (EXP/'audit/run-complete.json').exists()
|
||||
m = pd.read_parquet(ROOT/cfg['canonical']/'neuron-index/unified-metadata.parquet').sort_values('neuron_idx').reset_index(drop=True)
|
||||
trials = pd.read_csv(EXP/'audit/trials.csv'); watched = pd.read_csv(EXP/'audit/watched-counts.csv')
|
||||
remaining, comparisons = [], []
|
||||
for oi in cfg['odor_indices']:
|
||||
names = {c: f'{oi}-{cfg["seed"]}-{c}' for c in cfg['conditions']}
|
||||
narrow = pd.read_parquet(EXP/'events'/f'{names["calyx_PN_KC_contact_block"]}.parquet')
|
||||
allkc = pd.read_parquet(EXP/'events'/f'{names["all_PN_KC_block"]}.parquet')
|
||||
a = np.load(EXP/'states'/f'{names["calyx_PN_KC_contact_block"]}.npz')
|
||||
b = np.load(EXP/'states'/f'{names["all_PN_KC_block"]}.npz')
|
||||
comparisons.append({'odor_index': oi, 'terminal_vs_allKC_events_exact': narrow.equals(allkc),
|
||||
'counts_exact': bool(np.array_equal(a['counts'], b['counts'])),
|
||||
'max_V_difference': float(np.abs(a['final_v'].astype(float)-b['final_v']).max()),
|
||||
'max_g_difference': float(np.abs(a['final_g'].astype(float)-b['final_g']).max())})
|
||||
kc = np.flatnonzero(m.annotation_class.eq('Kenyon_Cell'))
|
||||
late = narrow[(narrow.step >= 5000) & narrow.neuron_idx.isin(kc)]
|
||||
for node, group in late.groupby('neuron_idx'):
|
||||
row = m.iloc[node]
|
||||
remaining.append({'odor_index': oi, 'neuron_idx': int(node), 'body_id': int(row.body_id),
|
||||
'type': str(row.annotation_type), 'late_spikes': len(group),
|
||||
'first_late_spike_ms': float(group.step.min()/10), 'last_spike_ms': float(group.step.max()/10),
|
||||
'source_identity_not_inferred_from_this_count': True})
|
||||
pd.DataFrame(remaining).to_csv(EXP/'audit/remaining-late-KCs.csv', index=False)
|
||||
pd.DataFrame(comparisons).to_csv(EXP/'audit/narrow-vs-allKC.csv', index=False)
|
||||
|
||||
rows = []
|
||||
for name, g in trials.groupby('condition', sort=False):
|
||||
row = {'condition': name}
|
||||
for key in ['whole_spikes', 'KC_active', 'KC_spikes', 'KC_late_spikes', 'KC82312_late_spikes', 'VA6_late_spikes', 'wall_seconds']:
|
||||
row[key+'_min'] = float(g[key].min()); row[key+'_max'] = float(g[key].max())
|
||||
rows.append(row)
|
||||
pd.DataFrame(rows).to_csv(EXP/'audit/condition-ranges.csv', index=False)
|
||||
ca = trials[trials.condition == 'calyx_PN_KC_contact_block']
|
||||
wide = trials[trials.condition == 'whole_PN_output_block']
|
||||
local = watched[(watched.population == 'ALLN') & watched.trial.str.endswith('calyx_PN_KC_contact_block')]
|
||||
result = {'posthoc_summary_not_new_experiment': True,
|
||||
'all_six_CA_block_targetKC_late_zero': bool(ca.KC82312_late_spikes.eq(0).all()),
|
||||
'all_six_CA_block_VA6_late_persists': bool(ca.VA6_late_spikes.gt(0).all()),
|
||||
'all_six_broad_block_KC_and_VA6_late_zero': bool((wide.KC_late_spikes.eq(0) & wide.VA6_late_spikes.eq(0)).all()),
|
||||
'CA_block_active_KC_range': [int(ca.KC_active.min()), int(ca.KC_active.max())],
|
||||
'CA_block_ALLN_watched_late_rate_range_hz': [float(local.late_rate_hz.min()), float(local.late_rate_hz.max())],
|
||||
'CA_vs_allKC_spike_sequence_exact_for_all_odors': bool(all(x['terminal_vs_allKC_events_exact'] for x in comparisons)),
|
||||
'not_a_physiological_APL_gain_estimate': True, 'not_a_memory_or_separability_success': True,
|
||||
'new_candidate': False, 'source_sha256': digest(Path(__file__))}
|
||||
save(EXP/'audit/mechanism-interpretation.json', result)
|
||||
|
||||
def span(frame, key):
|
||||
lo, hi = frame[key].min(), frame[key].max()
|
||||
return str(int(lo)) if lo == hi else f'{int(lo)}–{int(hi)}'
|
||||
ref = trials[trials.condition == 'frozen_reference']
|
||||
lines = ['# Phase 4AB 机制解读', '',
|
||||
'正式状态:**COMPLETE-SCOPE-COUNTERFACTUAL / NO-DYNAMICS-CANDIDATE**。', '',
|
||||
'这一轮区分了“PN→KC 传入位置”与“维持 PN 放电的上游反馈”。没有修复或批准新的 APL 生理模型,也没有训练。', '',
|
||||
'| 诊断范围 | 活跃 KC(6 个旧气味) | 目标 KC 晚期 spikes | VA6 PN 晚期 spikes |',
|
||||
'|---|---:|---:|---:|']
|
||||
labels = {'frozen_reference': '原模型', 'calyx_PN_KC_contact_block': '仅明确 calyx 接触',
|
||||
'all_PN_KC_block': '同批 PN 的全部 KC 输出', 'whole_PN_output_block': '同批 PN 的全部快速输出'}
|
||||
for name in cfg['conditions']:
|
||||
g = trials[trials.condition == name]
|
||||
lines.append(f'| {labels[name]} | {span(g,"KC_active")} | {span(g,"KC82312_late_spikes")} | {span(g,"VA6_late_spikes")} |')
|
||||
lines += ['', '晚期窗口是 500–1,000 ms。每个气味沿用 seed 131101;总计 24 组诊断、4 次精确重复,不是 5-seed 学习验证。', '',
|
||||
'## 能说明什么', '',
|
||||
f'仅 calyx 接触阻断后,目标 KC82312 的晚期 spikes 范围为 {span(ca,"KC82312_late_spikes")},VA6 仍为 {span(ca,"VA6_late_spikes")}。',
|
||||
f'同时 KC 总活跃数降为 {span(ca,"KC_active")},原模型为 {span(ref,"KC_active")}。不能把这种接近静默的结果包装成编码或记忆改善。',
|
||||
'即使两个局部阻断条件的事件序列相同,其电压/g 仍可能不同;逐气味比较见 narrow-vs-allKC.csv。',
|
||||
'宽范围 PN 快速输出阻断影响反馈路径,不等价于局部 calyx 调制;它也没有关闭 PN 胞体或 frozen graded APL 的 PN 输入。', '',
|
||||
'## 仍不能说明什么', '',
|
||||
'完全阻断只是工程极端对照,不是 APL 的实测释放抑制比例。没有估计 receptor、bouton、局部扩散范围或时间常数。',
|
||||
'未使用 held-out 气味,未检验学习、保持、反转或动作输出。不得选取这次“让活动更少”的操作作为正式脑模型。',
|
||||
'剩余 late KC 的 ID/type 已单独保存;其来源不能仅凭残余计数猜测。', '',
|
||||
'## 下一步边界', '',
|
||||
'先继续 upstream feedback / compartment-specific physiology 的证据约束,区分 calyx 信号传递与触角叶持续放电,再确定可检验的局部动力学假设。',
|
||||
'不能靠增加整体抑制、调 eta/threshold、扩大 plastic mask 或让 KC 全静默宣称修复。当前不进入学习、游戏或视觉。', '',
|
||||
'相关原始研究:[Prisco et al., 2021](https://doi.org/10.7554/eLife.74172)、[Amin et al., 2020](https://doi.org/10.7554/eLife.56954)。它们支持研究局部 APL 作用,不直接验证本实验完全阻断幅度。', '',
|
||||
'顶层验收仍为 5/10(50%);本轮推进的是机制定位与可重复诊断能力。最终完整性状态见 regression/final-integrity.json。']
|
||||
with (EXP/'INTERPRETATION.zh.md').open('x', encoding='utf-8') as out:
|
||||
out.write('\n'.join(lines)+'\n')
|
||||
print(result)
|
||||
@@ -0,0 +1,11 @@
|
||||
451b53b128eb0ad67acb81f060f3676794a68c98637dd2c0eddd137da509fccf PROTOCOL.md
|
||||
2842b5dd94ed30fc01d1a8b8d95e235de78695d3d1c1076402499276fb903261 README.md
|
||||
33ba5f323e9ed3343f43a8d64fb7e862c8a4f1118b9b441f4745e26cc14303a8 analyze.py
|
||||
e22088d9847504a50c0e684d2cee9420f484ab1ebef9d6057921062d8f785e73 config.yaml
|
||||
c8ec2ae6e72d35fedbd42acc6e6b5e7af2634d5b9805dc20e777b172c55d5aa4 diagnostic.py
|
||||
be92ce24e63910f67da0238b19e3abb882e1afd317561fccf01734f1b4a56f25 environment.py
|
||||
0b2f71365267886933ba5becb52f4123d3d4e95e9c5bc0cf19c9bf0563831625 launch.py
|
||||
bc194bb857336908b1c3fe71cc0e7b7fe4152794745bd3443533d47da886f5ec publish.py
|
||||
8f5774496dcceaee1c2793a5308754cbd0864c4a38d507481071d988f9de424d scope.py
|
||||
38a1c6e653ae97e77e5feda3ee8f2f8a3c366755bfbbd57ca1373b9b8f971bbb test_scope.py
|
||||
971136a667367a84cfc50ebc3fa73a749eddb8b8e1ca6931708ba67bfc12f0c1 validate.py
|
||||
@@ -0,0 +1,22 @@
|
||||
phase: 4AB
|
||||
root: /data/digital-drosophila
|
||||
model: models/male-cns-lif/terminal-scope-diagnostic-v1
|
||||
experiment: experiments/phase4ab-terminal-scope
|
||||
prior_model: models/male-cns-lif/apl-pn-scope-audit-v1
|
||||
prior_experiment: experiments/phase4aa-apl-pn-scope
|
||||
runtime: models/male-cns-lif/deterministic-runtime-v2
|
||||
snapshot: registry/snapshots/bootstrap-20260929
|
||||
canonical: datasets/male-cns-v1.0/processed/neuron-only
|
||||
event_prior: experiments/phase4z-pn-feedback-audit
|
||||
state_prior: experiments/phase4y-dpm-hypothesis
|
||||
input_model: models/male-cns-lif/csus-audit-v1
|
||||
roi_labels: [CA(L), CA(R)]
|
||||
seed: 131101
|
||||
odor_indices: [0, 1, 2, 3, 4, 5]
|
||||
repeat_odor_index: 0
|
||||
late_start_ms: 500
|
||||
target_KC: 82312
|
||||
watched_VA6: 133666
|
||||
conditions: [frozen_reference, calyx_PN_KC_contact_block, all_PN_KC_block, whole_PN_output_block]
|
||||
workers: 4
|
||||
no_candidate: true
|
||||
@@ -0,0 +1,369 @@
|
||||
"""New-path-only Phase 4AB scope diagnostics; never edits frozen scientific files."""
|
||||
import argparse
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
from scope import contact_scope, edge_ids_for_pairs, blocked_weights
|
||||
|
||||
|
||||
def load_context():
|
||||
cfg = yaml.safe_load(Path(__file__).with_name('config.yaml').read_text())
|
||||
root = Path(cfg['root']); model = root/cfg['model']; exp = root/cfg['experiment']
|
||||
sys.path.insert(0, str(root/cfg['prior_model']))
|
||||
from audit import save, digest, immutable_check, seal
|
||||
return cfg, root, model, exp, save, digest, immutable_check, seal
|
||||
|
||||
|
||||
def manifest_check(folder, digest):
|
||||
manifest = folder/'manifest.json'
|
||||
rows = json.loads(manifest.read_text())['files']
|
||||
bad = []
|
||||
for row in rows:
|
||||
path = folder/row['path']
|
||||
if not path.is_file() or path.stat().st_size != row['bytes'] or digest(path) != row['sha256']:
|
||||
bad.append(row['path'])
|
||||
return {'pass': not bad, 'checked': len(rows), 'mismatches': bad,
|
||||
'manifest_sha256': digest(manifest), 'checksums_sha256': digest(folder/'checksums.sha256')}
|
||||
|
||||
|
||||
def integrity(cfg, root, digest, immutable_check):
|
||||
result = {'original': immutable_check(root, root/cfg['snapshot'], cfg['workers'])}
|
||||
for key in ('prior_model', 'prior_experiment'):
|
||||
result[key] = manifest_check(root/cfg[key], digest)
|
||||
assert all(x['pass'] for x in result.values()), result
|
||||
return result
|
||||
|
||||
|
||||
def queues():
|
||||
out = []
|
||||
for port in (8188, 8189, 8190):
|
||||
with urllib.request.urlopen(f'http://192.168.10.51:{port}/queue', timeout=5) as r:
|
||||
q = json.load(r)
|
||||
out.append({'port': port, 'running': len(q['queue_running']), 'pending': len(q['queue_pending'])})
|
||||
assert all(x['running'] == x['pending'] == 0 for x in out), 'ComfyUI busy: defer, never evict'
|
||||
return out
|
||||
|
||||
|
||||
def prepare():
|
||||
cfg, root, model, exp, save, digest, immutable_check, _ = load_context()
|
||||
assert exp.parent == root/'experiments' and model.parent == root/'models/male-cns-lif'
|
||||
if exp.exists() and any(exp.iterdir()): raise RuntimeError('Refusing nonempty experiment')
|
||||
exp.mkdir(exist_ok=True)
|
||||
for d in ('protocol', 'audit', 'masks', 'events', 'states', 'observations', 'requests', 'replay', 'regression', 'performance'):
|
||||
(exp/d).mkdir()
|
||||
save(exp/'protocol/preregistered.json', {'config': cfg, 'before_analysis': True,
|
||||
'timestamp_utc': datetime.now(timezone.utc).isoformat(),
|
||||
'parent_commit': subprocess.check_output(['git', '-C', str(root), 'rev-parse', 'HEAD'], text=True).strip(),
|
||||
'source_hashes': {p.name: digest(p) for p in sorted(model.iterdir()) if p.is_file()}})
|
||||
save(exp/'regression/preflight.json', integrity(cfg, root, digest, immutable_check))
|
||||
can = root/cfg['canonical']; prior = root/cfg['prior_experiment']
|
||||
m = pd.read_parquet(can/'neuron-index/unified-metadata.parquet').sort_values('neuron_idx').reset_index(drop=True)
|
||||
ptr = np.load(can/'gpu/csr-indptr.int64.npy', mmap_mode='r')
|
||||
dst = np.load(can/'gpu/csr-indices.int32.npy', mmap_mode='r')
|
||||
raw = np.load(can/'gpu/csr-weights.int64.npy', mmap_mode='r')
|
||||
assert len(m) == 166700 and len(dst) == 25582938 and int(raw.sum()) == 124177617
|
||||
assert m.body_id.dtype == np.uint64 and np.array_equal(m.neuron_idx, np.arange(len(m)))
|
||||
src = np.repeat(np.arange(len(m), dtype=np.int32), np.diff(ptr))
|
||||
apl = np.flatnonzero(m.annotation_type.eq('APL')); pn = np.flatnonzero(m.annotation_class.eq('ALPN'))
|
||||
kc = np.flatnonzero(m.annotation_class.eq('Kenyon_Cell'))
|
||||
contacts = pd.read_parquet(prior/'localization/APL-incident.parquet')
|
||||
ap = contacts[contacts.source_neuron_idx.isin(apl) & contacts.target_neuron_idx.isin(pn)].copy()
|
||||
pk = pd.read_parquet(prior/'localization/APL-target-PN-to-KC.parquet')
|
||||
assert len(ap) == 4600 and ap.target_neuron_idx.nunique() == 259 and len(pk) == 385229
|
||||
assert pk.source_neuron_idx.isin(pn).all() and pk.target_neuron_idx.isin(kc).all()
|
||||
pairs, allowed, keep = contact_scope(ap, pk, cfg['roi_labels'])
|
||||
ids = edge_ids_for_pairs(pairs, ptr, dst)
|
||||
assert np.array_equal(raw[ids], pairs.raw_partner_count)
|
||||
full_expected = np.flatnonzero(np.isin(src, ap.target_neuron_idx.unique()) & np.isin(dst, kc))
|
||||
assert np.array_equal(np.sort(ids), full_expected), 'Missing whole PN->KC pairs'
|
||||
pairs['edge_id'] = ids
|
||||
pairs['source_body_id'] = m.body_id.to_numpy()[pairs.source_neuron_idx]
|
||||
pairs['target_body_id'] = m.body_id.to_numpy()[pairs.target_neuron_idx]
|
||||
ca = pairs[pairs.blocked_contact_count > 0].sort_values('edge_id').copy()
|
||||
eligible = np.sort(ca.source_neuron_idx.unique()).astype(np.int32)
|
||||
all_pk = np.flatnonzero(np.isin(src, eligible) & np.isin(dst, kc))
|
||||
whole = np.flatnonzero(np.isin(src, eligible))
|
||||
assert set(ca.edge_id).issubset(all_pk) and set(all_pk).issubset(whole)
|
||||
assert not np.any(np.isin(src[ca.edge_id], apl)) and not np.any(ca.retained_contact_count < 0)
|
||||
for name, eids, count in [
|
||||
('calyx_PN_KC_contact_block', ca.edge_id.to_numpy(np.int64), ca.blocked_contact_count.to_numpy(np.int64)),
|
||||
('all_PN_KC_block', all_pk, raw[all_pk]), ('whole_PN_output_block', whole, raw[whole])]:
|
||||
frame = pd.DataFrame({'edge_id': eids, 'source_neuron_idx': src[eids], 'target_neuron_idx': dst[eids],
|
||||
'raw_synapse_count': raw[eids], 'blocked_contact_count': count})
|
||||
frame['source_body_id'] = m.body_id.to_numpy()[src[eids]]
|
||||
frame['target_body_id'] = m.body_id.to_numpy()[dst[eids]]
|
||||
frame.to_parquet(exp/'masks'/f'{name}.parquet', index=False)
|
||||
pairs.to_parquet(exp/'audit/all-PN-KC-contact-partitions.parquet', index=False)
|
||||
allowed.to_parquet(exp/'audit/observed-APL-PN-ROI-pairs.parquet', index=False)
|
||||
pns = m.iloc[np.sort(ap.target_neuron_idx.unique().astype(np.int32))].copy()
|
||||
pns['eligible_for_diagnostic'] = pns.neuron_idx.isin(eligible)
|
||||
pns.to_parquet(exp/'audit/all-259-APL-target-PNs.parquet', index=False)
|
||||
roi = pk.assign(in_scope=keep).groupby(['primary_post', 'in_scope'], observed=True, dropna=False).size().reset_index(name='contacts')
|
||||
roi.to_csv(exp/'audit/ROI-partition.csv', index=False)
|
||||
summary = {'neurons': len(m), 'edges': len(dst), 'raw_contacts': int(raw.sum()), 'APL_target_PNs': len(pns),
|
||||
'eligible_PNs': len(eligible), 'ineligible_PNs': int((~pns.eligible_for_diagnostic).sum()),
|
||||
'calyx_mask_edges': len(ca), 'calyx_blocked_contacts': int(ca.blocked_contact_count.sum()),
|
||||
'calyx_affected_KCs': int(ca.target_neuron_idx.nunique()), 'mixed_CA_nonCA_edges': int((ca.retained_contact_count > 0).sum()),
|
||||
'calyx_other_contacts_retained_on_affected_edges': int(ca.retained_contact_count.sum()),
|
||||
'all_PN_KC_edges': len(all_pk), 'all_PN_KC_contacts': int(raw[all_pk].sum()),
|
||||
'whole_PN_edges': len(whole), 'whole_PN_contacts': int(raw[whole].sum()),
|
||||
'unknown_ROI_contacts_in_original_PN_KC': int(pk.primary_post.eq('CentralBrain-unspecified').sum()),
|
||||
'unknown_contacts_relabelled': 0, 'same_bouton_confirmed': False, 'biophysical_coefficient_known': False}
|
||||
save(exp/'audit/scope-summary.json', summary)
|
||||
inputs = [prior/'localization/APL-incident.parquet', prior/'localization/APL-target-PN-to-KC.parquet',
|
||||
prior/'external-resources/synaptic-partners/download-verification.json',
|
||||
can/'neuron-index/unified-metadata.parquet'] + list((can/'gpu').glob('csr-*.npy'))
|
||||
save(exp/'protocol/input-hashes.json', [{'path': str(p.relative_to(root)), 'sha256': digest(p), 'bytes': p.stat().st_size} for p in inputs])
|
||||
save(exp/'protocol/mask-hashes.json', {p.name: digest(p) for p in sorted((exp/'masks').iterdir())})
|
||||
print(json.dumps(summary), flush=True)
|
||||
|
||||
|
||||
def run():
|
||||
cfg, root, model, exp, save, digest, _, _ = load_context()
|
||||
pre = json.loads((exp/'protocol/preregistered.json').read_text())
|
||||
assert all(digest(model/n) == h for n, h in pre['source_hashes'].items()), 'Source changed since preregistration'
|
||||
assert not (exp/'regression/run-start.json').exists(), 'Refusing rerun into existing outputs'
|
||||
assert all(digest(exp/'masks'/n) == h for n, h in json.loads((exp/'protocol/mask-hashes.json').read_text()).items())
|
||||
q = queues(); runtime = root/cfg['runtime']
|
||||
pc = yaml.safe_load((runtime/'phase4p.yaml').read_text())
|
||||
os.environ['CUDA_VISIBLE_DEVICES'] = pc['device']['visible_gpu_uuid']
|
||||
os.environ['CUDA_HOME'] = '/usr/local/cuda'; os.environ['TORCH_CUDA_ARCH_LIST'] = '8.9'; os.environ['MAX_JOBS'] = '2'
|
||||
os.environ['PATH'] = str(root/'models/male-cns-lif/.venv/bin') + ':/usr/local/cuda/bin:' + os.environ['PATH']
|
||||
sys.path.insert(0, str(root/'models/male-cns-lif')); sys.path.insert(0, str(runtime))
|
||||
import torch
|
||||
import phase4p, phase4v, phase4z
|
||||
from phase4p_revalidate import DeterministicAPLSim, door_inputs, make_drive
|
||||
from phase4r import arrhash
|
||||
free, total = torch.cuda.mem_get_info(); assert free > 2*1024**3
|
||||
save(exp/'regression/run-start.json', {'ComfyUI_queues': q, 'free_VRAM': free, 'total_VRAM': total,
|
||||
'device': torch.cuda.get_device_name(), 'timestamp_utc': datetime.now(timezone.utc).isoformat()})
|
||||
start = time.perf_counter(); torch.cuda.reset_peak_memory_stats()
|
||||
rt = phase4p.CanonicalRuntime(pc, runtime, exp)
|
||||
sim = DeterministicAPLSim(rt); sim.install(None)
|
||||
assert len(sim.identity) == 14232 and sim.n == 166700 and len(rt.raw_np) == 25582938 and rt.delay_steps == 18
|
||||
frozen = rt.weights.cpu().numpy().copy(); anatomical_hash = arrhash(rt.raw_np)
|
||||
original_install = sim.install
|
||||
apl_input_hash = arrhash(sim.input_gpu.cpu().numpy(), sim.output_gpu.cpu().numpy())
|
||||
base_hash = arrhash(sim.base.cpu().numpy())
|
||||
masks = {n: pd.read_parquet(exp/'masks'/f'{n}.parquet') for n in cfg['conditions'][1:]}
|
||||
updates = {}
|
||||
for name, mask in masks.items():
|
||||
ids = mask.edge_id.to_numpy(np.int64)
|
||||
values = blocked_weights(frozen[ids], rt.raw_np[ids], mask.blocked_contact_count.to_numpy(np.int64))
|
||||
assert np.all(np.abs(values) <= np.abs(frozen[ids]))
|
||||
updates[name] = (torch.as_tensor(ids, device=rt.device), torch.as_tensor(values, device=rt.device))
|
||||
mask.assign(frozen_effective_weight=frozen[ids], diagnostic_effective_weight=values,
|
||||
diagnostic_delta=(values.astype(float)-frozen[ids])).to_parquet(exp/'audit'/f'{name}-effective-overlay.parquet', index=False)
|
||||
condition = 'frozen_reference'
|
||||
def install(E):
|
||||
assert E is None, 'No learning allowed'
|
||||
original_install(None)
|
||||
if condition != 'frozen_reference':
|
||||
ix, values = updates[condition]; rt.weights[ix] = values
|
||||
sim.install = install
|
||||
vc = yaml.safe_load((root/cfg['input_model']/'phase4v.yaml').read_text())
|
||||
assert vc['short_duration_ms'] == 120 and vc['observation_duration_ms'] == 1000
|
||||
oldwatch = json.loads((root/cfg['event_prior']/'protocol/watched.json').read_text())
|
||||
watch = np.asarray(oldwatch['all_watched'], np.int32)
|
||||
targetkc = np.unique(sim.identity.loc[sim.identity.compartment.eq(vc['target_compartment']), 'source_neuron_idx'].to_numpy(np.int32))
|
||||
oc, panel, vectors, selection, mapping = door_inputs(); panel = panel[panel.split == 'calibration'].reset_index(drop=True)
|
||||
assert len(panel) == 6
|
||||
save(exp/'protocol/calibration-panel.json', panel.to_dict('records'))
|
||||
codes = phase4z.role_codes(sim.meta)
|
||||
trials, regress, request_summary, populations, watched, hashes = [], [], [], [], [], []
|
||||
baseline0 = None
|
||||
try:
|
||||
for oi in cfg['odor_indices']:
|
||||
odor = panel.iloc[oi]
|
||||
targets, rates = make_drive(oc, panel, vectors, selection, mapping, odor.odor_id, 'physiological')
|
||||
forcing_seed = vc['seed_namespace'] + oi*vc['odor_seed_offset'] + cfg['seed']
|
||||
force = phase4v.forcing(sim, targets, rates, forcing_seed, vc, 'offset'); ff = force.cpu().numpy()
|
||||
hashes.append({'odor_index': oi, 'seed': forcing_seed, 'forcing': arrhash(ff), 'targets_rates': arrhash(targets, rates)})
|
||||
np.savez_compressed(exp/'requests'/f'{oi}-scheduled.npz', forcing=ff, targets=targets, rates=rates)
|
||||
requests0 = None
|
||||
for condition in cfg['conditions']:
|
||||
queues(); name = f'{oi}-{cfg["seed"]}-{condition}'
|
||||
obs = phase4z.Observer(rt, watch, ff, targets); obs.attach()
|
||||
try: result = phase4v.observed(sim, targets, force, targetkc, vc)
|
||||
finally: obs.detach()
|
||||
assert result['finite'] and np.isfinite(result['final_v']).all() and np.isfinite(result['final_g']).all()
|
||||
ev = obs.events(); states = np.asarray(obs.states); req = pd.DataFrame(obs.request_rows)
|
||||
assert len(req) == int(ff.sum())
|
||||
assert np.array_equal(np.bincount(ev.neuron_idx, minlength=sim.n), result['counts'])
|
||||
expected_weights = frozen.copy()
|
||||
if condition != 'frozen_reference':
|
||||
ids = masks[condition].edge_id.to_numpy(np.int64)
|
||||
expected_weights[ids] = updates[condition][1].cpu().numpy()
|
||||
assert np.array_equal(rt.weights.cpu().numpy(), expected_weights), 'Undeclared weight change'
|
||||
row = {'trial': name, 'declared_weight_scope_exact': True}
|
||||
if condition == 'frozen_reference':
|
||||
old_ev = pd.read_parquet(root/cfg['event_prior']/'events'/f'{name}-whole-brain.parquet')
|
||||
old_state = np.load(root/cfg['event_prior']/'states'/f'{name}-watched.npz')
|
||||
old_final = np.load(root/cfg['state_prior']/'states'/f'{name}.npz')
|
||||
row['prior_exact'] = {'events': ev.equals(old_ev), 'watched_states': np.array_equal(states, old_state['state']),
|
||||
**{key: bool(np.array_equal(result[key], old_final[key])) for key in ('counts', 'final_v', 'final_g', 'trace')}}
|
||||
assert all(row['prior_exact'].values()), row
|
||||
requests0 = req
|
||||
if oi == cfg['repeat_odor_index']: baseline0 = ev, states, req, result
|
||||
paired = requests0.merge(req, on=['step', 'neuron_idx'], suffixes=('_ref', '_diagnostic'), validate='one_to_one', how='outer', indicator=True)
|
||||
assert paired['_merge'].eq('both').all()
|
||||
request_summary.append({'trial': name, 'scheduled': len(req), 'identical_request_keys': True,
|
||||
'accepted': int(req.status.eq('accepted').sum()), 'blocked': int(req.status.eq('blocked').sum()), 'preexisting': int(req.status.eq('preexisting').sum()),
|
||||
'status_differences_vs_reference': int((paired.status_ref != paired.status_diagnostic).sum()),
|
||||
'actual_output_differences_vs_reference': int((paired.actual_output_spike_ref != paired.actual_output_spike_diagnostic).sum())})
|
||||
late = ev[ev.step >= round(cfg['late_start_ms']/rt.dt)]
|
||||
period_count = np.bincount(late.neuron_idx, minlength=sim.n)
|
||||
for code in np.unique(codes):
|
||||
nodes = np.flatnonzero(codes == code)
|
||||
populations.append({'trial': name, 'population': str(code), 'neurons': len(nodes), 'active': int((result['counts'][nodes] > 0).sum()),
|
||||
'spikes': int(result['counts'][nodes].sum()), 'late_spikes': int(period_count[nodes].sum())})
|
||||
for node in watch:
|
||||
watched.append({'trial': name, 'neuron_idx': int(node), 'body_id': int(sim.meta.iloc[node].body_id),
|
||||
'type': str(sim.meta.iloc[node].annotation_type), 'population': str(codes[node]),
|
||||
'spikes': int(result['counts'][node]), 'late_spikes': int(period_count[node]),
|
||||
'late_rate_hz': float(period_count[node]/((1000-cfg['late_start_ms'])/1000))})
|
||||
trials.append({'trial': name, 'odor_index': oi, 'odor_name': str(odor.odor_name), 'seed': cfg['seed'], 'condition': condition,
|
||||
'whole_spikes': len(ev), 'KC_active': int(np.count_nonzero(result['counts'][sim.kc])),
|
||||
'KC_active_fraction': float(np.count_nonzero(result['counts'][sim.kc])/len(sim.kc)),
|
||||
'KC_spikes': int(result['counts'][sim.kc].sum()), 'KC_late_spikes': int(period_count[sim.kc].sum()),
|
||||
'KC82312_late_spikes': int(period_count[cfg['target_KC']]), 'VA6_late_spikes': int(period_count[cfg['watched_VA6']]),
|
||||
'finite': True, 'wall_seconds': result['wall_seconds'], 'biological_ms': 1000})
|
||||
ev.to_parquet(exp/'events'/f'{name}.parquet', index=False)
|
||||
req.to_parquet(exp/'requests'/f'{name}.parquet', index=False)
|
||||
pd.DataFrame(result['samples']).to_parquet(exp/'observations'/f'{name}.parquet', index=False)
|
||||
np.savez_compressed(exp/'states'/f'{name}.npz', counts=result['counts'], final_v=result['final_v'], final_g=result['final_g'], trace=result['trace'])
|
||||
np.savez_compressed(exp/'states'/f'{name}-watched.npz', state=states, watch=watch)
|
||||
if oi == cfg['repeat_odor_index']:
|
||||
queues(); again = phase4z.Observer(rt, watch, ff, targets); again.attach()
|
||||
try: repeat = phase4v.observed(sim, targets, force, targetkc, vc)
|
||||
finally: again.detach()
|
||||
row['repeat_exact'] = {'events': ev.equals(again.events()), 'states': np.array_equal(states, again.states),
|
||||
'requests': req.equals(pd.DataFrame(again.request_rows)),
|
||||
**{key: bool(np.array_equal(result[key], repeat[key])) for key in ('counts', 'final_v', 'final_g', 'trace')}}
|
||||
assert all(row['repeat_exact'].values()), row
|
||||
regress.append(row); save(exp/'regression'/f'{name}.json', row)
|
||||
print(json.dumps(trials[-1]), flush=True)
|
||||
# Fixed-event replay is an independent direct-operator check, not the dynamic whole-brain result.
|
||||
replay_check(cfg, exp, sim, masks, frozen, baseline0, watch, phase4z, save)
|
||||
finally:
|
||||
sim.install = original_install; original_install(None)
|
||||
assert np.array_equal(rt.weights.cpu().numpy(), frozen)
|
||||
assert arrhash(rt.raw_np) == anatomical_hash and arrhash(sim.base.cpu().numpy()) == base_hash
|
||||
assert arrhash(sim.input_gpu.cpu().numpy(), sim.output_gpu.cpu().numpy()) == apl_input_hash
|
||||
save(exp/'regression/restored.json', {'base': True, 'raw': True, 'APL_matrices': True, 'frozen_effective_weights': True})
|
||||
for name, rows in [('trials', trials), ('request-comparison', request_summary), ('population-counts', populations), ('watched-counts', watched)]:
|
||||
pd.DataFrame(rows).to_csv(exp/'audit'/f'{name}.csv', index=False)
|
||||
save(exp/'protocol/forcing-hashes.json', hashes)
|
||||
# Separate reference trials do not include the graded APL overlay, exactly as archived.
|
||||
rt.weights.copy_(sim.base)
|
||||
reference = []
|
||||
for spec in pc['reference_trials']:
|
||||
queues(); ref = root/'experiments/phase4p-deterministic-runtime/phase4c-v2/reference'/spec['name']
|
||||
ev = pd.read_parquet(ref/'spike-events.parquet')
|
||||
expected = {'step': ev.step.to_numpy(), 'neuron_idx': ev.neuron_idx.to_numpy(),
|
||||
'counts': np.load(ref/'per-neuron-spike-counts.int32.npy'), 'v': np.load(ref/'final-voltage.float32.npy'), 'g': np.load(ref/'final-g.float32.npy')}
|
||||
r = rt.run(np.flatnonzero(rt.pops[spec['population']]), spec['rate_hz'], spec['seed'], spec['duration_ms'])
|
||||
eq = {key: bool(np.array_equal(r[key], value)) for key, value in expected.items()}
|
||||
assert all(eq.values()), eq
|
||||
reference.append({'trial': spec['name'], 'exact': eq, 'wall_seconds': r['metrics']['wall_seconds']})
|
||||
original_install(None)
|
||||
save(exp/'regression/five-frozen-trials.json', {'pass': True, 'trials': reference, 'original_v1_preserved': True, 'oracle': 'frozen deterministic-v2'})
|
||||
save(exp/'performance/summary.json', {'wall_seconds_including_build': time.perf_counter()-start,
|
||||
'peak_VRAM_bytes': int(torch.cuda.max_memory_allocated()), 'device': torch.cuda.get_device_name(),
|
||||
'diagnostic_overhead_included': True, 'not_runtime_benchmark': True, 'ComfyUI_models_unloaded': False})
|
||||
save(exp/'audit/run-complete.json', {'trials': len(trials), 'exact_repeat_trials': 4, 'old_trial_reproductions': 6,
|
||||
'frozen_reference_trials': 5, 'all_finite': True, 'candidate': False, 'training': False})
|
||||
|
||||
|
||||
def replay_check(cfg, exp, sim, masks, frozen, baseline, watch, phase4z, save):
|
||||
ev, state, req, result = baseline; rt = sim.rt; T = len(state)
|
||||
ids = np.flatnonzero(np.isin(rt.indices_np, watch))
|
||||
edges = pd.DataFrame({'edge_id': ids, 'source_neuron_idx': sim.sources[ids], 'target_neuron_idx': rt.indices_np[ids], 'weight': frozen[ids]})
|
||||
arrivals = ev.merge(edges, left_on='neuron_idx', right_on='source_neuron_idx')
|
||||
arrivals['arrival_step'] = arrivals.step + rt.delay_steps
|
||||
arrivals = arrivals[arrivals.arrival_step < T].sort_values(['target_neuron_idx', 'arrival_step', 'edge_id'], kind='stable')
|
||||
rows = []
|
||||
for wi, node in enumerate(watch):
|
||||
subset = arrivals[arrivals.target_neuron_idx == node].copy()
|
||||
after = np.r_[state[1:, 1, wi], result['final_g'][node]]
|
||||
overlay = after.astype(float)-state[:, 5, wi].astype(float)
|
||||
request = np.zeros(T, bool); rq = req[req.neuron_idx == node]; request[rq.step.to_numpy(np.int32)] = True
|
||||
base_arrival = state[:, 3, wi]
|
||||
for name in cfg['conditions']:
|
||||
weights = subset.weight.to_numpy(np.float32).copy()
|
||||
if name != 'frozen_reference':
|
||||
frame = masks[name]; eids = frame.edge_id.to_numpy(np.int64)
|
||||
values = blocked_weights(frozen[eids], rt.raw_np[eids], frame.blocked_contact_count.to_numpy(np.int64))
|
||||
lookup = pd.Series(values, index=eids)
|
||||
mapped = subset.edge_id.map(lookup); keep = mapped.notna().to_numpy()
|
||||
weights[keep] = mapped[keep].to_numpy(np.float32)
|
||||
subset['diagnostic_weight'] = weights
|
||||
a = np.zeros(T, np.float32)
|
||||
for step, group in subset.groupby('arrival_step', sort=False):
|
||||
a[int(step)] = np.cumsum(group.diagnostic_weight.to_numpy(np.float32), dtype=np.float32)[-1]
|
||||
rr = phase4z.replay(a, overlay, request, rt)
|
||||
if name == 'frozen_reference':
|
||||
exact = np.array_equal(a, base_arrival) and np.array_equal(rr[0], ev.loc[ev.neuron_idx == node, 'step']) and np.array_equal(rr[1], state[:, 4:7, wi])
|
||||
assert exact and rr[2] == result['final_v'][node] and rr[3] == result['final_g'][node]
|
||||
unchanged = np.array_equal(a, base_arrival)
|
||||
if name == 'calyx_PN_KC_contact_block' and node not in sim.kc: assert unchanged
|
||||
rows.append({'condition': name, 'neuron_idx': int(node), 'arrival_unchanged': bool(unchanged),
|
||||
'arrival_L1_difference': float(np.abs(a.astype(float)-base_arrival).sum()),
|
||||
'conditional_late_spikes': int((rr[0] >= round(cfg['late_start_ms']/rt.dt)).sum()),
|
||||
'all_other_recorded_events_and_APL_fixed': True, 'whole_brain_causality_claim': False})
|
||||
pd.DataFrame(rows).to_csv(exp/'replay/fixed-event-scope.csv', index=False)
|
||||
save(exp/'regression/fixed-event-replay.json', {'baseline_exact_neurons': len(watch), 'terminal_nonKC_arrivals_exact': True})
|
||||
|
||||
|
||||
def finalize():
|
||||
cfg, root, model, exp, save, digest, immutable_check, seal = load_context()
|
||||
assert (exp/'audit/run-complete.json').exists()
|
||||
pre = json.loads((exp/'protocol/preregistered.json').read_text())
|
||||
assert all(digest(model/n) == h for n, h in pre['source_hashes'].items())
|
||||
post = integrity(cfg, root, digest, immutable_check)
|
||||
assert post == json.loads((exp/'regression/preflight.json').read_text())
|
||||
save(exp/'regression/final-integrity.json', post)
|
||||
q = queues(); save(exp/'audit/final-service-health.json', {'ComfyUI_queues': q, 'services_stopped': False})
|
||||
trials = pd.read_csv(exp/'audit/trials.csv'); scope = json.loads((exp/'audit/scope-summary.json').read_text())
|
||||
requests = pd.read_csv(exp/'audit/request-comparison.csv')
|
||||
lines = ['# Phase 4AB — calyx contact-scope diagnostic', '',
|
||||
'Status: COMPLETE-SCOPE-COUNTERFACTUAL / NO-DYNAMICS-CANDIDATE.',
|
||||
'Diagnostic interventions only; no calibrated biological inhibition, no training, no parameter winner.', '',
|
||||
f"Eligible PNs: {scope['eligible_PNs']}/259. CA mask: {scope['calyx_mask_edges']} edges / {scope['calyx_blocked_contacts']} contacts.",
|
||||
f"Affected KCs: {scope['calyx_affected_KCs']}. Mixed-ROI affected edges: {scope['mixed_CA_nonCA_edges']}; retained other contacts: {scope['calyx_other_contacts_retained_on_affected_edges']}.",
|
||||
f"Broader PN→KC control: {scope['all_PN_KC_edges']} edges / {scope['all_PN_KC_contacts']} contacts. All-fast-PN-output control: {scope['whole_PN_edges']} edges / {scope['whole_PN_contacts']} contacts.", '',
|
||||
'| Odor | Condition | Active KC | KC late spikes | KC82312 late | VA6 late |',
|
||||
'|---|---|---:|---:|---:|---:|']
|
||||
for r in trials.itertuples():
|
||||
lines.append(f'| {r.odor_name} | {r.condition} | {r.KC_active} | {r.KC_late_spikes} | {r.KC82312_late_spikes} | {r.VA6_late_spikes} |')
|
||||
lines += ['', 'Late means 500–1,000 ms; 120 ms old calibration odor input; one seed, six old odors. No held-out generalization claim.',
|
||||
f"Identical scheduled requests across conditions. Delivered request-associated output differences versus reference: {int(requests.actual_output_differences_vs_reference.sum())} (summed over all comparisons). See per-trial table; this is state/refractory feedback, not a changed RNG.", '',
|
||||
'## Interpretation limits', '',
|
||||
'CA selection uses exact primary_post labels plus matched APL→PN/ROI metadata, not bouton identities or inferred receptors. Unknown ROIs are not reclassified.',
|
||||
'Whole-PN control removes fast output, not somatic spikes or the frozen graded APL input. It is not a complete PN knockout.',
|
||||
'Reduction/silencing is not restored information or associative memory. No condition becomes a formal model. Fixed-event replay freezes other recorded events/APL; dynamic trials allow natural recurrent feedback.',
|
||||
'Local-inhibition motivation: [Prisco et al.](https://doi.org/10.7554/eLife.74172), [Amin et al.](https://doi.org/10.7554/eLife.56954). These sources do not establish the complete-block operator as physiological.', '',
|
||||
'## Verification', '',
|
||||
'Six original trials reproduced exactly (whole spike events, watched state and final counts/V/g). Four first-odor repeats exact. Five deterministic-v2 frozen reference trials exact. Finite state throughout.',
|
||||
'Original 230,239 files and both Phase 4AA manifests rechecked before/after. Base CSR/count/sign, frozen LIF/APL, eligibility-v2 and 14,232 plastic mask preserved.',
|
||||
'Timing includes diagnostic CPU/GPU observation overhead and is not a runtime speed benchmark. ComfyUI was not stopped or unloaded.',
|
||||
'Top-level acceptance gates remain 5/10 (50%); anatomical diagnosis is not evidence of learning success.']
|
||||
(exp/'REPORT.md').write_text('\n'.join(lines)+'\n', encoding='utf-8')
|
||||
save(exp/'audit/final-summary.json', {'status': 'COMPLETE-SCOPE-COUNTERFACTUAL / NO-DYNAMICS-CANDIDATE',
|
||||
'integrity_pass': True, 'reference_pass': True, 'scope': scope, 'candidate': False, 'training': False,
|
||||
'preserved_acceptance_gates': '5/10', 'frozen_baseline_modified': False})
|
||||
for folder in (model, exp):
|
||||
count = seal(folder); assert manifest_check(folder, digest)['pass']; print('SEALED', folder, count, flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser(); parser.add_argument('task', choices=['prepare', 'run', 'finalize'])
|
||||
globals()[parser.parse_args().task]()
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Record executable/package/hardware versions without installing anything."""
|
||||
import importlib.metadata
|
||||
import platform
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
MODEL = Path('/data/digital-drosophila/models/male-cns-lif/terminal-scope-diagnostic-v1')
|
||||
EXP = Path('/data/digital-drosophila/experiments/phase4ab-terminal-scope')
|
||||
sys.path.insert(0, str(MODEL))
|
||||
import diagnostic
|
||||
_, _, _, _, save, digest, _, _ = diagnostic.load_context()
|
||||
packages = {}
|
||||
for name in ('torch', 'numpy', 'pandas', 'pyarrow', 'scipy', 'PyYAML', 'ninja'):
|
||||
try: packages[name] = importlib.metadata.version(name)
|
||||
except importlib.metadata.PackageNotFoundError: packages[name] = None
|
||||
gpu = subprocess.check_output(['nvidia-smi', '--query-gpu=index,name,uuid,driver_version,memory.total', '--format=csv,noheader'], text=True)
|
||||
nvcc = subprocess.check_output(['/usr/local/cuda/bin/nvcc', '--version'], text=True)
|
||||
save(EXP/'protocol/environment.json', {'python': sys.version, 'executable': sys.executable,
|
||||
'platform': platform.platform(), 'packages': packages, 'GPU_inventory': gpu, 'nvcc': nvcc,
|
||||
'installed_or_upgraded_packages': False, 'source_sha256': digest(Path(__file__))})
|
||||
print(packages)
|
||||
@@ -0,0 +1,32 @@
|
||||
"""Explicit frozen module paths; repairs launch only, never scientific source."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
ROOT = Path('/data/digital-drosophila')
|
||||
MODEL = ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'
|
||||
EXP = ROOT/'experiments/phase4ab-terminal-scope'
|
||||
PARENT = ROOT/'models/male-cns-lif'
|
||||
names = ['deterministic-runtime-v2', 'credit-v3', 'timing-audit-v1', 'timing-credit-v1',
|
||||
'timing-stability-v1', 'odor-timing-generalization-v1', 'csus-audit-v1',
|
||||
'late-kc-audit-v1', 'dpm-hypothesis-v1', 'pn-feedback-audit-v1', 'transmitter-audit-v1']
|
||||
sys.path[:0] = [str(MODEL), str(PARENT)] + [str(PARENT/n) for n in names]
|
||||
os.environ['PYTHONDONTWRITEBYTECODE'] = '1'
|
||||
import diagnostic
|
||||
_, _, _, _, save, digest, _, _ = diagnostic.load_context()
|
||||
save(EXP/'protocol/launch-addendum.json', {
|
||||
'previous_attempt': 'ModuleNotFoundError: phase4v before run-start and before simulation',
|
||||
'only_fix': 'Add explicit existing frozen dependency directories to module lookup',
|
||||
'model_source_edited': False, 'protocol_parameters_changed': False,
|
||||
'launcher_sha256': digest(Path(__file__)), 'module_paths': [str(PARENT/n) for n in names]})
|
||||
try:
|
||||
diagnostic.run()
|
||||
finally:
|
||||
loaded = {}
|
||||
for name, mod in list(sys.modules.items()):
|
||||
filename = getattr(mod, '__file__', None)
|
||||
if filename:
|
||||
p = Path(filename)
|
||||
if p.is_file() and p.is_relative_to(PARENT) and '.venv' not in p.parts:
|
||||
loaded[name] = {'path': str(p.relative_to(ROOT)), 'sha256': digest(p)}
|
||||
save(EXP/'protocol/imported-code-hashes.json', loaded)
|
||||
@@ -0,0 +1,60 @@
|
||||
{
|
||||
"created_utc": "2026-09-29T16:26:06.650270+00:00",
|
||||
"files": [
|
||||
{
|
||||
"bytes": 4956,
|
||||
"path": "PROTOCOL.md",
|
||||
"sha256": "451b53b128eb0ad67acb81f060f3676794a68c98637dd2c0eddd137da509fccf"
|
||||
},
|
||||
{
|
||||
"bytes": 2083,
|
||||
"path": "README.md",
|
||||
"sha256": "2842b5dd94ed30fc01d1a8b8d95e235de78695d3d1c1076402499276fb903261"
|
||||
},
|
||||
{
|
||||
"bytes": 6922,
|
||||
"path": "analyze.py",
|
||||
"sha256": "33ba5f323e9ed3343f43a8d64fb7e862c8a4f1118b9b441f4745e26cc14303a8"
|
||||
},
|
||||
{
|
||||
"bytes": 835,
|
||||
"path": "config.yaml",
|
||||
"sha256": "e22088d9847504a50c0e684d2cee9420f484ab1ebef9d6057921062d8f785e73"
|
||||
},
|
||||
{
|
||||
"bytes": 28229,
|
||||
"path": "diagnostic.py",
|
||||
"sha256": "c8ec2ae6e72d35fedbd42acc6e6b5e7af2634d5b9805dc20e777b172c55d5aa4"
|
||||
},
|
||||
{
|
||||
"bytes": 1193,
|
||||
"path": "environment.py",
|
||||
"sha256": "be92ce24e63910f67da0238b19e3abb882e1afd317561fccf01734f1b4a56f25"
|
||||
},
|
||||
{
|
||||
"bytes": 1608,
|
||||
"path": "launch.py",
|
||||
"sha256": "0b2f71365267886933ba5becb52f4123d3d4e95e9c5bc0cf19c9bf0563831625"
|
||||
},
|
||||
{
|
||||
"bytes": 1644,
|
||||
"path": "publish.py",
|
||||
"sha256": "bc194bb857336908b1c3fe71cc0e7b7fe4152794745bd3443533d47da886f5ec"
|
||||
},
|
||||
{
|
||||
"bytes": 2797,
|
||||
"path": "scope.py",
|
||||
"sha256": "8f5774496dcceaee1c2793a5308754cbd0864c4a38d507481071d988f9de424d"
|
||||
},
|
||||
{
|
||||
"bytes": 2615,
|
||||
"path": "test_scope.py",
|
||||
"sha256": "38a1c6e653ae97e77e5feda3ee8f2f8a3c366755bfbbd57ca1373b9b8f971bbb"
|
||||
},
|
||||
{
|
||||
"bytes": 2032,
|
||||
"path": "validate.py",
|
||||
"sha256": "971136a667367a84cfc50ebc3fa73a749eddb8b8e1ca6931708ba67bfc12f0c1"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
"""Copy only new small sealed evidence into Git; never copy bulk datasets."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import sys
|
||||
|
||||
ROOT = Path('/data/digital-drosophila'); MODEL = ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'
|
||||
EXP = ROOT/'experiments/phase4ab-terminal-scope'
|
||||
sys.path.insert(0, str(MODEL))
|
||||
import diagnostic
|
||||
_, _, _, _, save, digest, _, _ = diagnostic.load_context()
|
||||
for folder in (MODEL, EXP):
|
||||
assert diagnostic.manifest_check(folder, digest)['pass']
|
||||
registry = ROOT/'registry/phase4ab-terminal-scope'; registry.mkdir(exist_ok=False)
|
||||
records = []
|
||||
for p in sorted(EXP.rglob('*')):
|
||||
if not p.is_file() or p.suffix not in ('.json', '.csv', '.md', '.sha256') or p.stat().st_size > 4*1024**2:
|
||||
continue
|
||||
dst = registry/'evidence'/p.relative_to(EXP); dst.parent.mkdir(parents=True, exist_ok=True)
|
||||
with p.open('rb') as src, dst.open('xb') as out: shutil.copyfileobj(src, out)
|
||||
assert digest(p) == digest(dst)
|
||||
records.append({'source': str(p.relative_to(ROOT)), 'copy': str(dst.relative_to(registry)),
|
||||
'sha256': digest(p), 'bytes': p.stat().st_size})
|
||||
save(registry/'evidence-index.json', records)
|
||||
save(registry/'seal-verification.json', {str(p.relative_to(ROOT)): diagnostic.manifest_check(p, digest) for p in (MODEL, EXP)})
|
||||
for source, dest in [('INTERPRETATION.zh.md', 'phase4ab-terminal-scope.md'), ('REPORT.md', 'phase4ab-technical-results.md')]:
|
||||
target = ROOT/'docs/progress'/dest
|
||||
with (EXP/source).open('rb') as src, target.open('xb') as out: shutil.copyfileobj(src, out)
|
||||
print('Published', len(records), 'small evidence copies; no data copied to Git')
|
||||
@@ -0,0 +1,53 @@
|
||||
"""Pure CPU contact accounting and immutable diagnostic weight operations."""
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
|
||||
|
||||
def contact_scope(apl_pn, pn_kc, rois):
|
||||
"""Only exact, observed PN+postsynaptic-ROI pairs; no side/name inference."""
|
||||
required = {'source_neuron_idx', 'target_neuron_idx', 'primary_post'}
|
||||
if not required.issubset(apl_pn) or not required.issubset(pn_kc):
|
||||
raise ValueError('Actual schema lacks required identifiers/primary_post')
|
||||
permitted = apl_pn.loc[apl_pn.primary_post.isin(rois),
|
||||
['target_neuron_idx', 'primary_post']].drop_duplicates()
|
||||
permitted = permitted.rename(columns={'target_neuron_idx': 'source_neuron_idx'})
|
||||
keys = pd.MultiIndex.from_frame(permitted.astype({'primary_post': 'string'}))
|
||||
observed = pd.MultiIndex.from_frame(pn_kc[['source_neuron_idx', 'primary_post']]
|
||||
.astype({'primary_post': 'string'}))
|
||||
keep = observed.isin(keys)
|
||||
pairs = ['source_neuron_idx', 'target_neuron_idx']
|
||||
total = pn_kc.groupby(pairs, observed=True).size().rename('raw_partner_count')
|
||||
ca = pn_kc.loc[keep].groupby(pairs, observed=True).size().rename('blocked_contact_count')
|
||||
frame = total.to_frame().join(ca).fillna(0).reset_index()
|
||||
for key in pairs + ['raw_partner_count', 'blocked_contact_count']:
|
||||
frame[key] = frame[key].astype(np.int64)
|
||||
frame['retained_contact_count'] = frame.raw_partner_count - frame.blocked_contact_count
|
||||
assert (frame.retained_contact_count >= 0).all()
|
||||
return frame, permitted, keep
|
||||
|
||||
|
||||
def edge_ids_for_pairs(frame, ptr, dst):
|
||||
out = []
|
||||
for s, t in frame[['source_neuron_idx', 'target_neuron_idx']].itertuples(index=False, name=None):
|
||||
ids = np.arange(ptr[s], ptr[s + 1], dtype=np.int64)
|
||||
found = ids[dst[ids] == t]
|
||||
if len(found) != 1:
|
||||
raise ValueError(f'Expected exactly one existing CSR edge for {s}->{t}')
|
||||
out.append(found[0])
|
||||
return np.asarray(out, np.int64)
|
||||
|
||||
|
||||
def blocked_weights(base, raw, contacts):
|
||||
base = np.asarray(base)
|
||||
raw = np.asarray(raw)
|
||||
contacts = np.asarray(contacts)
|
||||
if base.dtype != np.float32 or raw.dtype != np.int64 or contacts.dtype != np.int64:
|
||||
raise TypeError('Expected float32 effective weight and int64 anatomical counts')
|
||||
if base.shape != raw.shape or raw.shape != contacts.shape:
|
||||
raise ValueError('Shape mismatch')
|
||||
if not np.isfinite(base).all() or np.any(raw <= 0) or np.any(contacts < 0) or np.any(contacts > raw):
|
||||
raise ValueError('Invalid contact partition or weights')
|
||||
out = base.copy()
|
||||
m = contacts > 0 # No-op copies original bits; no rounding at strength zero.
|
||||
out[m] = (base[m].astype(np.float64) * (raw[m] - contacts[m]) / raw[m]).astype(np.float32)
|
||||
return out
|
||||
@@ -0,0 +1,52 @@
|
||||
import unittest
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
from scope import contact_scope, edge_ids_for_pairs, blocked_weights
|
||||
|
||||
|
||||
class ScopeTests(unittest.TestCase):
|
||||
def test_roi_and_side_not_guessed(self):
|
||||
apl = pd.DataFrame({'source_neuron_idx': [0, 0, 1], 'target_neuron_idx': [3, 3, 4],
|
||||
'primary_post': ['CA(R)', 'CentralBrain-unspecified', 'CA(L)']})
|
||||
pn = pd.DataFrame({'source_neuron_idx': [3, 3, 3, 4, 4], 'target_neuron_idx': [5]*5,
|
||||
'primary_post': ['CA(R)', 'CA(L)', None, 'CA(R)', 'CA(L)']})
|
||||
pairs, allowed, keep = contact_scope(apl, pn, ['CA(L)', 'CA(R)'])
|
||||
self.assertEqual(keep.tolist(), [True, False, False, False, True])
|
||||
self.assertEqual(pairs.blocked_contact_count.tolist(), [1, 1])
|
||||
self.assertEqual(pairs.retained_contact_count.tolist(), [2, 1])
|
||||
self.assertEqual(len(allowed), 2)
|
||||
|
||||
def test_zero_identity_and_nonmutation(self):
|
||||
w = np.array([.275, -3.14159, 0, -0.0], np.float32)
|
||||
raw = np.array([1, 43, 10, 1], np.int64)
|
||||
old = w.tobytes(), raw.tobytes()
|
||||
result = blocked_weights(w, raw, np.zeros(4, np.int64))
|
||||
self.assertEqual(w.tobytes(), result.tobytes())
|
||||
self.assertEqual((w.tobytes(), raw.tobytes()), old)
|
||||
|
||||
def test_partial_and_full(self):
|
||||
w = np.array([2., -4., 0., 1.], np.float32)
|
||||
raw = np.array([4, 4, 3, 7], np.int64)
|
||||
result = blocked_weights(w, raw, np.array([1, 4, 2, 0], np.int64))
|
||||
np.testing.assert_array_equal(result, np.array([1.5, -0., 0., 1.], np.float32))
|
||||
np.testing.assert_array_equal(w, np.array([2., -4., 0., 1.], np.float32))
|
||||
|
||||
def test_invalid_partition_rejected(self):
|
||||
for bad in [-1, 3]:
|
||||
with self.assertRaises(ValueError):
|
||||
blocked_weights(np.array([1.], np.float32), np.array([2], np.int64), np.array([bad], np.int64))
|
||||
|
||||
def test_existing_edge_only(self):
|
||||
p = pd.DataFrame({'source_neuron_idx': [0, 1], 'target_neuron_idx': [2, 0]})
|
||||
ptr = np.array([0, 2, 3, 3], np.int64); dst = np.array([1, 2, 0], np.int32)
|
||||
np.testing.assert_array_equal(edge_ids_for_pairs(p, ptr, dst), [1, 2])
|
||||
p.loc[1, 'target_neuron_idx'] = 2
|
||||
with self.assertRaises(ValueError): edge_ids_for_pairs(p, ptr, dst)
|
||||
|
||||
def test_duplicates_rejected(self):
|
||||
p = pd.DataFrame({'source_neuron_idx': [0], 'target_neuron_idx': [1]})
|
||||
with self.assertRaises(ValueError):
|
||||
edge_ids_for_pairs(p, np.array([0, 2, 2], np.int64), np.array([1, 1], np.int32))
|
||||
|
||||
|
||||
if __name__ == '__main__': unittest.main()
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Additional adversarial operator tests and durable unit-test record."""
|
||||
import io
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import unittest
|
||||
import numpy as np
|
||||
|
||||
MODEL = Path('/data/digital-drosophila/models/male-cns-lif/terminal-scope-diagnostic-v1')
|
||||
EXP = Path('/data/digital-drosophila/experiments/phase4ab-terminal-scope')
|
||||
sys.path.insert(0, str(MODEL))
|
||||
from scope import blocked_weights
|
||||
import diagnostic
|
||||
|
||||
_, _, _, _, save, digest, _, _ = diagnostic.load_context()
|
||||
stream = io.StringIO()
|
||||
suite = unittest.defaultTestLoader.discover(str(MODEL), pattern='test_scope.py')
|
||||
result = unittest.TextTestRunner(stream=stream, verbosity=2).run(suite)
|
||||
assert result.wasSuccessful(), stream.getvalue()
|
||||
rng = np.random.default_rng(41337)
|
||||
raw = rng.integers(1, 100000, size=10000, dtype=np.int64)
|
||||
contacts = np.asarray([rng.integers(0, n+1) for n in raw], dtype=np.int64)
|
||||
base = (raw.astype(np.float32)*np.float32(.275)*rng.choice([-1, 0, 1], size=len(raw))).astype(np.float32)
|
||||
raw_bytes, base_bytes = raw.tobytes(), base.tobytes()
|
||||
out = blocked_weights(base, raw, contacts)
|
||||
assert np.isfinite(out).all() and np.all(np.abs(out) <= np.abs(base))
|
||||
assert np.all(out[contacts == raw] == 0)
|
||||
assert out[contacts == 0].tobytes() == base[contacts == 0].tobytes()
|
||||
assert np.all(np.sign(out[out != 0]) == np.sign(base[out != 0]))
|
||||
assert raw_bytes == raw.tobytes() and base_bytes == base.tobytes()
|
||||
for bad in [np.array([np.inf], np.float32), np.array([np.nan], np.float32)]:
|
||||
try: blocked_weights(bad, np.array([2], np.int64), np.array([1], np.int64))
|
||||
except ValueError: pass
|
||||
else: raise AssertionError('nonfinite accepted')
|
||||
save(EXP/'regression/unit-tests.json', {'pass': True, 'unit_tests': result.testsRun, 'fuzz_cases': len(raw),
|
||||
'fuzz_seed': 41337, 'finite': True, 'never_sign_flip_or_amplify': True, 'input_nonmutation': True,
|
||||
'zero_identity_bitwise': True, 'unittest_log': stream.getvalue(), 'source_sha256': digest(Path(__file__))})
|
||||
print('PASS', result.testsRun, 'unit tests +', len(raw), 'randomized operator cases')
|
||||
Reference in New Issue
Block a user