Seal Phase 4AG author-APL audit and fixed CAL failure
This commit is contained in:
Digital Drosophila Archive
@@ -0,0 +1,76 @@
|
||||
# Phase4AG: author LIF + frozen legacy graded APL
|
||||
|
||||
Authorized independent hypothesis branch. All old source, raw connectome, signs,
|
||||
W_syn, APL calibration, eligibility-v2 and 14,232-edge mask remain read-only.
|
||||
No learning, gain/threshold changes, new edges, top-K, heldout activity, game or RL.
|
||||
|
||||
Author code provenance is the already archived MIT-licensed
|
||||
https://github.com/philshiu/Drosophila_brain_model at
|
||||
91bdd1e7dcf193f3e7ca5a8933497fcef63b7960; original model.py SHA-256
|
||||
fc45837d7122c6ce2a7f3f2f23c515992e4b232aadb919efabb72337fac88e4e.
|
||||
Associated paper: https://www.nature.com/articles/s41586-024-07763-9.
|
||||
This new MaleCNS/APL combination is not a strict reproduction of that model.
|
||||
|
||||
The declared combination is author float64 LIF plus the exact old float32 graded
|
||||
APL operator after synapses/resets at end. All 5,845 original APL fast out-edges
|
||||
are zeroed only in the effective in-memory model, as in the legacy overlay.
|
||||
Only KC receives graded replacement; known 1,212 non-KC edges remain unreplaced.
|
||||
This existing scope limitation is not silently repaired in a compatibility test.
|
||||
|
||||
Use the original full normalized I[2,166700]/O[2,4064] float32 bytes and original
|
||||
normalizers, even for induced real subgraphs. APL reads actual binary spikes,
|
||||
never requests, and keeps independent float32 state. Preserve unfused operations:
|
||||
state.mul_(float(f32(exp(-.1/20)))).add_(torch.mv(I, spikes_f32));
|
||||
h32=torch.matmul(state,O)*f32(.01). Promote h32 to float64, multiply by float64
|
||||
.001, then separately subtract from SI g. Do not alter TF32/default CuBLAS flags;
|
||||
record them. No added APL delay, drive dt/tau scaling, truncation or clipping.
|
||||
|
||||
APL writes KC g unconditionally after author reset, including refractory KC.
|
||||
Author g integration is frozen during refractory; therefore negative overlay g
|
||||
may accumulate and differs from old dynamics. This is an explicit hypothesis,
|
||||
not an assertion of original author or measured physiology. Guarded writes or
|
||||
reset-before/after changes would be different hypotheses, not fixes to this one.
|
||||
|
||||
Eight fixed metadata/connectivity-selected fixtures cover zero/reset, real PN/KC
|
||||
drive, current actual event feedback, duplicate natural/request event, 18-step
|
||||
arrival vs22-step refractory boundary, full fast APL mask, non-KC scope and mixed
|
||||
precision/high fan-in. No choice is made from runtime responses. Each GPU case
|
||||
has3 independent fresh runs. A Brian2 author oracle starts from its own state,
|
||||
computes its own actual spikes, and uses an isolated standard-I/O legacy-APL
|
||||
worker with independent state. No candidate inhibition trace is fed into Brian.
|
||||
All tick states, events, requests, queue order/guard, APL drive/state/h must agree
|
||||
exactly. The APL component is shared, so this is NOT a fully independent CPU
|
||||
oracle. Independently reconstruct old matrices using frozen initializer AST,
|
||||
replay actual events through original update AST, require exact component bytes;
|
||||
also quantify an independent float64 positive-sum arithmetic error bound. It is
|
||||
not a replacement for exact legacy equivalence. No arithmetic bound is adjusted.
|
||||
|
||||
Only after all fixture gates pass, run two predeclared full-graph probes: first
|
||||
old calibration odor, first original seed, both encoder modes. Compare each with
|
||||
the independently running Brian + isolated APL worker. Save state hashes and sparse
|
||||
events, not a time-by-all-neurons dense spike matrix. If a numerical gate fails,
|
||||
stop CAL; keep diagnostic evidence and do not widen tolerance.
|
||||
|
||||
Then run the complete original6 CAL odors ×5 original seeds ×2 original modes,
|
||||
100ms per run. Retain the ORIGINAL12-odor median for matched-energy encoding,
|
||||
original pn_drive target/RNG column order, original evaluation seeds51101..51105.
|
||||
Save Poisson requests before any trajectory. Only calibration dynamic cases are
|
||||
used; formal A/B and heldout remain unrun. Repeat first seed for every odor/mode.
|
||||
No per-odor pulse, amplitude, timing or threshold search. No learning.
|
||||
|
||||
Recompute the full original4L representation criteria on CAL only: mean KC active
|
||||
fraction .03..09, min>0, max<.5, same-odor Jaccard>different, between Euclidean>
|
||||
within, >=3 pairs whose mean between exceeds THAT PAIR's pooled within distances,
|
||||
both modes. Add finiteness and exact repeat requirements. CAL PASS does not imply
|
||||
heldout generalization, timing/credit specificity, memory or learning usability.
|
||||
If CAL fails stop promotion and report it, not a least-bad candidate.
|
||||
|
||||
Before/after verify SHA-256 of all original230239 files and AA--AF seals; rerun original five
|
||||
deterministic reference trials to new output. Archive code/env/source SHA,
|
||||
matrices/norms/masks/identities, requests and all outputs. No published candidate.
|
||||
|
||||
Technical references (queried2026-09-30): Brian2 2.5.1 refractory documentation
|
||||
https://brian2.readthedocs.io/en/2.5.1/user/refractoriness.html (actual generated code
|
||||
remains authoritative); PyTorch numerical accuracy notes
|
||||
https://docs.pytorch.org/docs/2.14/notes/numerical_accuracy.html (cross-backend
|
||||
floating operations are not assumed bitwise equivalent).
|
||||
@@ -0,0 +1,396 @@
|
||||
"""Complete original Phase 4L representation metrics on fixed CAL inputs only.
|
||||
|
||||
No simulation, fitting, heldout trajectory access, or candidate generation is
|
||||
performed here. The original 4L pair-specific within-odor baseline is retained;
|
||||
the weaker Phase 4P r4 aggregate gate is deliberately not used.
|
||||
"""
|
||||
import csv
|
||||
import hashlib
|
||||
import itertools
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
STATE_DTYPES = {"step": np.int32, "neuron_idx": np.int32, "counts": np.int32,
|
||||
"v": np.float64, "g": np.float64, "ref": np.int32}
|
||||
REQUEST_FIELDS = ("tick", "neuron_idx", "attempted", "accepted", "rejected")
|
||||
APL_FILES = {"apl-drive.float32.npy": 2, "apl-state.float32.npy": 2,
|
||||
"apl-inhibition.float32.npy": 4064}
|
||||
ORIGINAL_CAL_IDS = {
|
||||
"OMSUIQOIVADKIM-RXMQYKEDSA-N", "LABTWGUMFABVFG-ARJAWSKDSA-N",
|
||||
"QNVRIHYSUZMSGM-ZCFIWIBFSA-N", "VLSVVMPLPMNWBH-UHFFFAOYSA-N",
|
||||
"XHIUFYZDQBSEMF-UHFFFAOYSA-N", "OMSUIQOIVADKIM-YFKPBYRVSA-N"}
|
||||
ORIGINAL_MODES = ("physiological", "matched_energy")
|
||||
ORIGINAL_SEEDS = (51101, 51102, 51103, 51104, 51105)
|
||||
HEX_SHA = re.compile(r"[0-9a-f]{64}\Z")
|
||||
_DEFAULT_OUTPUT = object()
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def read_json(path):
|
||||
with Path(path).open(encoding="utf-8") as stream:
|
||||
return json.load(stream)
|
||||
|
||||
|
||||
def file_hash(path):
|
||||
digest = hashlib.sha256()
|
||||
with Path(path).open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(8 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def raw_hash(value):
|
||||
return hashlib.sha256(np.ascontiguousarray(value).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def arrays_bitwise_equal(first, second):
|
||||
"""Unlike array_equal, also reject different dtypes and signed-zero bits."""
|
||||
return bool(first.dtype == second.dtype and first.shape == second.shape and
|
||||
np.array_equal(np.ascontiguousarray(first).view(np.uint8),
|
||||
np.ascontiguousarray(second).view(np.uint8)))
|
||||
|
||||
|
||||
def pair_metrics(a, b):
|
||||
"""Exact definitions copied from phase4i.pair_metrics/safe_corr/cosine."""
|
||||
x, y = a.astype(np.float64), b.astype(np.float64)
|
||||
aa, bb = a > 0, b > 0
|
||||
union = int(np.count_nonzero(aa | bb))
|
||||
intersection = int(np.count_nonzero(aa & bb))
|
||||
denominator = float(np.linalg.norm(x) * np.linalg.norm(y))
|
||||
cosine = 1.0 if denominator == 0 and np.array_equal(x, y) else (
|
||||
0.0 if denominator == 0 else float(np.dot(x, y) / denominator))
|
||||
correlation = (1.0 if np.array_equal(x, y) else 0.0) if np.std(x) == 0 or np.std(y) == 0 else float(np.corrcoef(x, y)[0, 1])
|
||||
return {"active_intersection": intersection, "active_union": union,
|
||||
"active_jaccard": 1.0 if union == 0 else float(intersection / union),
|
||||
"cosine_similarity": cosine, "correlation": correlation,
|
||||
"euclidean_distance": float(np.linalg.norm(x - y))}
|
||||
|
||||
|
||||
def evaluate_gates(summary, gates):
|
||||
"""Each conjunct in original phase4l.py:239, kept visible for audit."""
|
||||
return {
|
||||
"mean_sparsity_in_closed_interval": bool(float(gates["sparse_min_fraction"]) <=
|
||||
summary["KC_active_fraction_mean"] <= float(gates["sparse_max_fraction"])),
|
||||
"maximum_strictly_below_half": bool(summary["KC_active_fraction_max"] < 0.5),
|
||||
"minimum_strictly_positive": bool(summary["KC_active_fraction_min"] > 0),
|
||||
"same_jaccard_gt_different": bool(summary["same_odor_jaccard_mean"] > summary["different_odor_jaccard_mean"]),
|
||||
"different_euclidean_gt_same": bool(summary["different_odor_euclidean_mean"] > summary["same_odor_euclidean_mean"]),
|
||||
"at_least_original_pair_count": bool(summary["pairwise_separation_pass_count"] >= int(gates["minimum_separable_odor_pairs"]))}
|
||||
|
||||
|
||||
def sample_cv(values):
|
||||
"""Original pandas Series.std uses ddof=1. Undefined diagnostic -> null."""
|
||||
values = np.asarray(values, dtype=np.float64)
|
||||
mean = float(values.mean())
|
||||
return None if mean == 0 else float(values.std(ddof=1) / mean)
|
||||
|
||||
|
||||
def summarize_mode(mode, odor_ids, seeds, runs, gates):
|
||||
within, within_j, between, between_j, between_cos, between_corr = [], [], [], [], [], []
|
||||
pairs = []
|
||||
for odor in odor_ids:
|
||||
arrays = [runs[(mode, odor, seed)]["kc_counts"] for seed in seeds]
|
||||
for first, second in itertools.combinations(arrays, 2):
|
||||
metric = pair_metrics(first, second)
|
||||
within.append(metric["euclidean_distance"])
|
||||
within_j.append(metric["active_jaccard"])
|
||||
for first_odor, second_odor in itertools.combinations(odor_ids, 2):
|
||||
distances = []
|
||||
for seed in seeds:
|
||||
metric = pair_metrics(runs[(mode, first_odor, seed)]["kc_counts"],
|
||||
runs[(mode, second_odor, seed)]["kc_counts"])
|
||||
distances.append(metric["euclidean_distance"])
|
||||
between.append(metric["euclidean_distance"])
|
||||
between_j.append(metric["active_jaccard"])
|
||||
between_cos.append(metric["cosine_similarity"])
|
||||
between_corr.append(metric["correlation"])
|
||||
pair_within = []
|
||||
for odor in (first_odor, second_odor):
|
||||
arrays = [runs[(mode, odor, seed)]["kc_counts"] for seed in seeds]
|
||||
pair_within.extend(pair_metrics(first, second)["euclidean_distance"]
|
||||
for first, second in itertools.combinations(arrays, 2))
|
||||
cross_mean, within_mean = float(np.mean(distances)), float(np.mean(pair_within))
|
||||
pairs.append({"odor_id_a": first_odor, "odor_id_b": second_odor,
|
||||
"between_same_seed_mean": cross_mean, "within_this_pair_mean": within_mean,
|
||||
"between_distance_count": len(distances), "within_distance_count": len(pair_within),
|
||||
"gate": bool(cross_mean > within_mean)})
|
||||
selected = [runs[(mode, odor, seed)] for odor in odor_ids for seed in seeds]
|
||||
fractions = [int(np.count_nonzero(run["kc_counts"])) / len(run["kc_counts"]) for run in selected]
|
||||
summary = {"KC_active_fraction_mean": float(np.mean(fractions)),
|
||||
"KC_active_fraction_min": float(np.min(fractions)), "KC_active_fraction_max": float(np.max(fractions)),
|
||||
"same_odor_jaccard_mean": float(np.mean(within_j)),
|
||||
"different_odor_jaccard_mean": float(np.mean(between_j)),
|
||||
"same_odor_euclidean_mean": float(np.mean(within)),
|
||||
"different_odor_euclidean_mean": float(np.mean(between)),
|
||||
"different_odor_cosine_mean": float(np.mean(between_cos)),
|
||||
"different_odor_correlation_mean": float(np.mean(between_corr)),
|
||||
"within_variance": float(np.var(within)), "between_distance_mean": float(np.mean(between)),
|
||||
"pairwise_separation_pass_count": sum(row["gate"] for row in pairs), "pair_count": len(pairs),
|
||||
"expected_event_count_cv": sample_cv([run["expected_external_events"] for run in selected]),
|
||||
"actual_event_count_cv": sample_cv([run["external_events"] for run in selected]),
|
||||
"within_distance_count": len(within), "between_distance_count": len(between),
|
||||
"pairwise_baseline": "Each pair's two odors: 10+10 within-seed distances; never pooled across all odors",
|
||||
"pairs": pairs}
|
||||
summary["conditions"] = evaluate_gates(summary, gates)
|
||||
summary["gate"] = bool(all(summary["conditions"].values()))
|
||||
return summary
|
||||
|
||||
|
||||
def validate_state(state, neurons, steps):
|
||||
require(set(state) == set(STATE_DTYPES), "state.npz must have exactly the six declared arrays")
|
||||
for name, dtype in STATE_DTYPES.items():
|
||||
value = state[name]
|
||||
require(value.dtype == dtype and value.ndim == 1, "Invalid state dtype/rank: " + name)
|
||||
require(bool(np.isfinite(value).all()), "Nonfinite state: " + name)
|
||||
if name not in ("step", "neuron_idx"):
|
||||
require(value.shape == (neurons,), "Invalid state length: " + name)
|
||||
ticks, nodes = state["step"], state["neuron_idx"]
|
||||
require(ticks.shape == nodes.shape and bool(np.all((ticks >= 0) & (ticks < steps))) and
|
||||
bool(np.all((nodes >= 0) & (nodes < neurons))), "Invalid sparse event indices")
|
||||
keys = ticks.astype(np.int64) * neurons + nodes
|
||||
require(bool(np.all(np.diff(keys) > 0)), "Sparse events not sorted unique by tick/global neuron")
|
||||
require(np.array_equal(np.bincount(nodes, minlength=neurons), state["counts"]), "Counts disagree with sparse events")
|
||||
require(bool(np.all((state["ref"] >= 0) & (state["ref"] <= 22))), "Final refractory outside frozen range")
|
||||
return keys
|
||||
|
||||
|
||||
def read_jsonl(path, steps, kind):
|
||||
with Path(path).open(encoding="utf-8") as source:
|
||||
rows = [json.loads(line) for line in source if line.strip()]
|
||||
require(len(rows) == steps and [row.get("tick") for row in rows] == list(range(steps)),
|
||||
"Missing/duplicate/reordered ticks in " + kind)
|
||||
for row in rows:
|
||||
names = ("v_sha256", "g_sha256", "ref_sha256") if kind == "state hashes" else (
|
||||
"ordered_edge_sha256", "accepted_mask_sha256", "joint_sha256")
|
||||
for name in names:
|
||||
require(isinstance(row.get(name), str) and HEX_SHA.fullmatch(row[name]), "Invalid SHA256 in " + kind)
|
||||
if kind == "queue":
|
||||
require(type(row.get("count")) is int and type(row.get("accepted_count")) is int and
|
||||
0 <= row["accepted_count"] <= row["count"], "Invalid queue counts")
|
||||
return rows
|
||||
|
||||
|
||||
def read_trial(output, case, spec, kc_globals, neurons, edges):
|
||||
summary = read_json(output / "summary.json")
|
||||
steps = spec["steps"]
|
||||
require(summary.get("complete") is True and summary.get("pass_all") is True and
|
||||
summary.get("all_finite") is True and summary.get("steps") == steps and
|
||||
summary.get("completed_ticks") == steps and summary.get("nodes") == neurons and
|
||||
summary.get("edges") == edges and summary.get("precision") == "float64" and
|
||||
summary.get("apl_precision") == "float32" and summary.get("case") == spec["name"] and
|
||||
summary.get("record_history") is False and summary.get("learning") is False,
|
||||
"Incomplete/failed/wrong-scope GPU summary")
|
||||
with np.load(output / "state.npz", allow_pickle=False) as source:
|
||||
state = {name: source[name].copy() for name in source.files}
|
||||
event_keys = validate_state(state, neurons, steps)
|
||||
hashes = read_jsonl(output / "tick-state-hashes.jsonl", steps, "state hashes")
|
||||
for name in ("v", "g", "ref"):
|
||||
require(hashes[-1][name + "_sha256"] == raw_hash(state[name]), "Final state does not match last tick hash: " + name)
|
||||
queues = read_jsonl(output / "queue.jsonl", steps, "queue")
|
||||
apl = {}
|
||||
for name, width in APL_FILES.items():
|
||||
value = np.load(output / name, mmap_mode="r", allow_pickle=False)
|
||||
require(value.dtype == np.float32 and value.shape == (steps, width) and
|
||||
bool(np.isfinite(value).all()) and bool(np.all(value >= 0)), "Invalid full APL trace: " + name)
|
||||
apl[name] = value
|
||||
with (output / "requests.csv").open(encoding="utf-8", newline="") as source:
|
||||
reader = csv.DictReader(source)
|
||||
require(tuple(reader.fieldnames or ()) == REQUEST_FIELDS, "Request CSV schema mismatch")
|
||||
request_rows = [{name: int(row[name]) for name in REQUEST_FIELDS} for row in reader]
|
||||
with np.load(case / "requests.npz", allow_pickle=False) as source:
|
||||
require(set(source.files) == {"tick", "neuron_idx"}, "Input request schema mismatch")
|
||||
expected_ticks, expected_nodes = source["tick"], source["neuron_idx"]
|
||||
require(expected_ticks.dtype == np.int32 and expected_nodes.dtype == np.int32 and
|
||||
expected_ticks.ndim == 1 and expected_ticks.shape == expected_nodes.shape,
|
||||
"Input request dtype/shape mismatch")
|
||||
require(len(request_rows) == len(expected_ticks), "Requested event coverage mismatch")
|
||||
require([row["tick"] for row in request_rows] == expected_ticks.tolist() and
|
||||
[row["neuron_idx"] for row in request_rows] == expected_nodes.tolist(), "Requests differ from frozen input")
|
||||
require(all(row["attempted"] == 1 and row["accepted"] in (0, 1) and
|
||||
row["rejected"] == 1 - row["accepted"] for row in request_rows), "Invalid request acceptance bookkeeping")
|
||||
requested_keys = expected_ticks.astype(np.int64) * neurons + expected_nodes
|
||||
require(bool(np.all(np.diff(requested_keys) > 0)) and bool(np.all((expected_ticks >= 0) & (expected_ticks < steps))) and
|
||||
bool(np.all((expected_nodes >= 0) & (expected_nodes < neurons))), "Input requests not sorted unique/in range")
|
||||
actual_accepted = np.array([row["accepted"] for row in request_rows], dtype=np.uint8)
|
||||
require(np.array_equal(np.isin(requested_keys, event_keys, assume_unique=True), actual_accepted.astype(bool)),
|
||||
"Request accepted/rejected values disagree with actual spikes")
|
||||
accepted = int(actual_accepted.sum())
|
||||
require(summary.get("requested") == len(request_rows) and summary.get("accepted_requests") == accepted and
|
||||
summary.get("rejected_requests") == len(request_rows) - accepted and
|
||||
summary.get("total_spikes") == len(state["step"]), "Summary counts disagree with raw output")
|
||||
component = read_json(output / "apl-component.json")
|
||||
require(component.get("ok") is True and component.get("n_neurons") == neurons and
|
||||
component.get("n_kc") == len(kc_globals) and component.get("n_apl") == 2 and
|
||||
component.get("dt_ms") == .1 and summary.get("apl_component") == component,
|
||||
"APL component provenance mismatch")
|
||||
bundle = Path(spec["apl_bundle_dir"])
|
||||
require(component.get("files_sha256") == {name: file_hash(bundle / name) for name in
|
||||
("input_matrix.npy", "output_matrix.npy", "kc_globals.npy", "apl_globals.npy")}, "APL bundle hashes changed")
|
||||
inputs = summary.get("input_sha256", {})
|
||||
require(inputs.get("spec.json") == file_hash(case / "spec.json") and
|
||||
inputs.get("requests.npz") == file_hash(case / "requests.npz"), "Case input provenance mismatch")
|
||||
kc_counts = state["counts"][kc_globals].copy()
|
||||
kc_event_mask = np.isin(state["neuron_idx"], kc_globals)
|
||||
record = {"case": spec["name"], "mode": spec["mode"], "odor_id": spec["odor_id"],
|
||||
"odor_name": spec["odor_name"], "seed": spec["seed"], "split": "calibration",
|
||||
"KC_active": int(np.count_nonzero(kc_counts)), "KC_active_fraction": float(np.count_nonzero(kc_counts) / len(kc_globals)),
|
||||
"KC_spikes": int(kc_counts.sum()), "KC_first_latency_ms":
|
||||
float(state["step"][kc_event_mask][0] * .1) if bool(np.any(kc_event_mask)) else None,
|
||||
"external_events": accepted, "expected_external_events": float(spec["expected_external_events"]),
|
||||
"total_PN_rate_hz": float(spec["rate_sum_hz"]), "finite": True,
|
||||
"total_spikes": len(state["step"]), "requested": len(request_rows)}
|
||||
require(np.isfinite(record["expected_external_events"]) and record["expected_external_events"] > 0,
|
||||
"CAL expected external events must be finite and positive")
|
||||
return {"state": state, "hashes": hashes, "queues": queues, "apl": apl,
|
||||
"requests": request_rows, "component": component, "kc_counts": kc_counts,
|
||||
"expected_external_events": record["expected_external_events"], "external_events": accepted,
|
||||
"record": record, "output": output}
|
||||
|
||||
|
||||
def exact_repeat(first, repeat):
|
||||
checks = {"state/" + name: arrays_bitwise_equal(first["state"][name], repeat["state"][name]) for name in STATE_DTYPES}
|
||||
checks.update({name: arrays_bitwise_equal(first["apl"][name], repeat["apl"][name]) for name in APL_FILES})
|
||||
checks.update(tick_state_hashes=first["hashes"] == repeat["hashes"],
|
||||
queues=first["queues"] == repeat["queues"], requests=first["requests"] == repeat["requests"],
|
||||
apl_component=first["component"] == repeat["component"])
|
||||
return {"pass_all": bool(all(checks.values())), "checks": checks,
|
||||
"scope": "All six state arrays, every tick all-neuron hashes, APL raw float32 traces, queues, requests and APL environment"}
|
||||
|
||||
|
||||
def validate_inventory(records, cfg):
|
||||
require(tuple(cfg["calibration"]["modes"]) == ORIGINAL_MODES and
|
||||
tuple(cfg["calibration"]["seeds"]) == ORIGINAL_SEEDS and
|
||||
cfg["calibration"]["duration_ms"] == 100.0 and cfg["calibration"]["odor_count"] == 6 and
|
||||
cfg["calibration"]["split"] == "calibration" and
|
||||
cfg["calibration"]["repeat_first_seed_each_odor"] is True, "Frozen CAL design changed")
|
||||
gates = cfg["gates"]
|
||||
require(gates["sparse_min_fraction"] == .03 and gates["sparse_max_fraction"] == .09 and
|
||||
gates["minimum_separable_odor_pairs"] == 3 and gates["max_fraction_strict"] == .5 and
|
||||
gates["min_fraction_strict"] == 0 and all(gates[key] is True for key in
|
||||
("same_jaccard_gt_between", "between_euclidean_gt_within", "both_modes_required", "finite_required", "repeat_exact_required")),
|
||||
"Original representation gates or additional integrity gates changed")
|
||||
expected = {(mode, index, seed) for mode in ORIGINAL_MODES for index in range(6) for seed in ORIGINAL_SEEDS}
|
||||
require(len(records) == 60 and {(row["mode"], row["odor_index"], row["seed"]) for row in records} == expected,
|
||||
"CAL inventory must cover exactly 2 modes x 6 odors x 5 seeds")
|
||||
odors = []
|
||||
for index in range(6):
|
||||
ids = {row["odor_id"] for row in records if row["odor_index"] == index}
|
||||
require(len(ids) == 1, "Odor identity changes across seeds/modes")
|
||||
odors.append(next(iter(ids)))
|
||||
require(set(odors) == ORIGINAL_CAL_IDS, "CAL identities differ from the original frozen six")
|
||||
for row in records:
|
||||
require(row["name"] == f"cal-{row['mode']}-{row['odor_index']:02d}-s{row['seed']}", "Unexpected CAL case name")
|
||||
return odors
|
||||
|
||||
|
||||
def summarize(out_json=_DEFAULT_OUTPUT):
|
||||
"""Recompute; default writes exclusively, out_json=None is strictly read-only."""
|
||||
from context import EXP, ROOT, CFG, check
|
||||
target = EXP / "calibration/summary.json" if out_json is _DEFAULT_OUTPUT else (
|
||||
None if out_json is None else Path(out_json))
|
||||
require(target is None or not target.exists(), "Refusing existing calibration summary")
|
||||
failures, runs, run_records, repeats, provenance = [], {}, [], [], []
|
||||
modes, odors = {}, []
|
||||
prerequisites_ok = False
|
||||
encoder = None
|
||||
records = []
|
||||
try:
|
||||
check()
|
||||
records = read_json(EXP / "protocol/calibration-cases.json")
|
||||
odors = validate_inventory(records, CFG)
|
||||
encoder = read_json(EXP / "protocol/encoder-audit.json")
|
||||
require(encoder["panel_count"] == 12 and encoder["cal_count"] == 6 and
|
||||
encoder["seeds"] == list(ORIGINAL_SEEDS) and encoder["requested_before_any_dynamics"] is True and
|
||||
encoder["formal_AB_run"] is False and encoder["heldout_run"] is False, "Invalid frozen encoder scope")
|
||||
# Only static original-panel metadata is read, never heldout trajectories.
|
||||
import pandas as pd
|
||||
original = pd.read_parquet(ROOT / "experiments/phase4l-olfactory-grounding/odor-encoder/frozen-odor-panel.parquet")
|
||||
require(len(original) == 12 and set(original[original.split == "calibration"].odor_id) == ORIGINAL_CAL_IDS,
|
||||
"Original static panel identity mismatch")
|
||||
require(float(np.median(original.mapped_response_sum)) == encoder["median_mapped_response_sum"],
|
||||
"Matched-energy reference is not the original frozen 12-panel median")
|
||||
kc_globals = np.load(EXP / "apl-bundle/kc_globals.npy", allow_pickle=False)
|
||||
require(kc_globals.dtype == np.int32 and kc_globals.shape == (4064,) and
|
||||
bool(np.all(np.diff(kc_globals) > 0)) and bool(np.all((kc_globals >= 0) & (kc_globals < CFG["neurons"]))),
|
||||
"Invalid frozen KC identities")
|
||||
prerequisites_ok = True
|
||||
except Exception as error:
|
||||
failures.append({"stage": "prerequisites", "type": type(error).__name__, "message": str(error)})
|
||||
if prerequisites_ok:
|
||||
for row in records:
|
||||
name = row["name"]
|
||||
first = None
|
||||
try:
|
||||
case = EXP / "cases" / name
|
||||
spec = read_json(case / "spec.json")
|
||||
require(spec["name"] == name and spec["kind"] == "calibration" and spec["steps"] == 1000 and
|
||||
spec["duration_ms"] == 100.0 and spec["dt_ms"] == .1 and spec["training"] is False and
|
||||
all(spec[key] == row[key] for key in ("mode", "odor_id", "odor_index", "seed")),
|
||||
"Case spec differs from fixed CAL inventory")
|
||||
first = read_trial(EXP / "gpu" / (name + "-r0"), case, spec, kc_globals, CFG["neurons"], CFG["edges"])
|
||||
require(first["record"]["requested"] == row["requests"], "Frozen inventory request count mismatch")
|
||||
runs[(row["mode"], row["odor_id"], row["seed"])] = {key: first[key] for key in
|
||||
("kc_counts", "expected_external_events", "external_events")}
|
||||
run_records.append(first["record"])
|
||||
provenance.append({"case": name, "repeat": 0, "state_npz_sha256": file_hash(first["output"] / "state.npz"),
|
||||
"summary_sha256": file_hash(first["output"] / "summary.json")})
|
||||
except Exception as error:
|
||||
failures.append({"stage": "raw_trial", "case": name, "repeat": 0,
|
||||
"type": type(error).__name__, "message": str(error)})
|
||||
if row["seed"] == ORIGINAL_SEEDS[0]:
|
||||
try:
|
||||
require(first is not None, "Baseline unavailable for exact repeat")
|
||||
repeat = read_trial(EXP / "gpu" / (name + "-r1"), case, spec, kc_globals, CFG["neurons"], CFG["edges"])
|
||||
repeat_report = {"case": name, **exact_repeat(first, repeat)}
|
||||
repeats.append(repeat_report)
|
||||
provenance.append({"case": name, "repeat": 1, "state_npz_sha256": file_hash(repeat["output"] / "state.npz"),
|
||||
"summary_sha256": file_hash(repeat["output"] / "summary.json")})
|
||||
require(repeat_report["pass_all"], "Exact repeat differs: " + ", ".join(
|
||||
key for key, value in repeat_report["checks"].items() if not value))
|
||||
except Exception as error:
|
||||
failures.append({"stage": "exact_repeat", "case": name, "repeat": 1,
|
||||
"type": type(error).__name__, "message": str(error)})
|
||||
if len(runs) == 60:
|
||||
for mode in ORIGINAL_MODES:
|
||||
try:
|
||||
modes[mode] = summarize_mode(mode, odors, ORIGINAL_SEEDS, runs, CFG["gates"])
|
||||
except Exception as error:
|
||||
failures.append({"stage": "representation", "mode": mode,
|
||||
"type": type(error).__name__, "message": str(error)})
|
||||
integrity_pass = bool(prerequisites_ok and len(runs) == 60 and not any(row["stage"] in
|
||||
("prerequisites", "raw_trial") for row in failures))
|
||||
repeat_pass = bool(len(repeats) == 12 and all(row["pass_all"] for row in repeats) and
|
||||
not any(row["stage"] == "exact_repeat" for row in failures))
|
||||
representation_pass = bool(len(modes) == 2 and all(row["gate"] for row in modes.values()))
|
||||
result = {"phase": "4AG", "scope": "Original fixed six CAL odors only; not heldout generalization or learning",
|
||||
"pass_all": bool(integrity_pass and repeat_pass and representation_pass),
|
||||
"numerical_integrity_pass": integrity_pass, "exact_repeat_pass": repeat_pass,
|
||||
"original_representation_gate_pass": representation_pass,
|
||||
"expected_base_trials": 60, "valid_base_trials": len(runs), "expected_repeats": 12,
|
||||
"checked_repeats": len(repeats), "seeds": list(ORIGINAL_SEEDS), "duration_ms": 100.0,
|
||||
"odor_ids": odors, "modes": modes, "runs": run_records, "repeats": repeats,
|
||||
"failures": failures, "input_output_provenance": provenance, "encoder_audit": encoder,
|
||||
"metric_source": "phase4l.representation (full original gate), phase4i.pair_metrics",
|
||||
"finite_gate_note": "Added integrity prerequisite; original 4L recorded finite but omitted it from its gate",
|
||||
"repeat_gate_note": "Additional exact repeat requirement; original metrics and numerical thresholds unchanged",
|
||||
"shared_apl_component_oracle_limit": True, "independent_oracle_equivalence_claim": False,
|
||||
"candidate_generated": False, "training": False, "heldout_dynamic_access": False}
|
||||
if target is not None:
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
with target.open("x", encoding="utf-8") as output:
|
||||
json.dump(result, output, ensure_ascii=False, indent=2, allow_nan=False)
|
||||
output.write("\n")
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(json.dumps({key: value for key, value in summarize().items() if key in
|
||||
("pass_all", "numerical_integrity_pass", "exact_repeat_pass", "original_representation_gate_pass")}), flush=True)
|
||||
@@ -0,0 +1,17 @@
|
||||
c14c2d3c35e2b404c516befa66370723fba075516540214ae46e4104ff87e813 PROTOCOL.md
|
||||
edb9cb2d890471fd07afd88c959ef3540b0cd40af490fac3f26396b5e7f5bf9c calibration.py
|
||||
6e1616b086f667fa4e4b5087f500f8080281977ca9922e7b055acfddd2003bd5 compare.py
|
||||
2aa6b4d777ff698018e1a6eaf571215208b8c874a4c848a6d632e2bd778c90e3 config.json
|
||||
b8b1231d958a3ec935ec1d5a36e08cbd9bcd1127f4d4bb243a9051884a0e7a16 context.py
|
||||
b6e3c67e668a971b544df32d76435ab5e537061b9fc2e8d9dedb5d2a726f5782 finalize.py
|
||||
69f5d893f47dafff4ef7c3186c3df0b9014666b32c8f4e943b57016bec92b18a gpu_apl.py
|
||||
21ceec65fe42581dc55872010e0025d660e80358905f1cf4df1dadf341a3b291 legacy_apl_worker.py
|
||||
9b7f3e0b59f2117b94b41a6f542f37e6f7950138d324c905e8e0ac6cb7d17151 oracle_apl.py
|
||||
197ccc6aae70566c4db4d4de17fa032e853b9bd9a04262905d6c71e12a930b6f prepare.py
|
||||
afbe6d5a37c29bda66d38c470093584130d1852e67881c681662aec3c76b33c7 run_case.py
|
||||
84c681030b0032119b5392d0c3f3c136a9c6b0d133eb548c40791a10bbbb8b63 run_suite.py
|
||||
bab3136eecbc5bb09a985d715844ee845cb72b07930ce376bfb39be33bd1845a test_calibration.py
|
||||
1a233f47943e01bbd845ac1dacdb1eec3f88c270c65a5f75bce1cf679b5de097 test_protocol.py
|
||||
5cfc9a4505da6b8fb4e7d2617cdaedaa8475ed46a11ec9bdf0d4d25b1d22c2d3 validate_components.py
|
||||
0d0e72b2f1348f6648124c8c34f75094f76fc5e1a281fbcca130a5fad2f35293 verify_frozen.py
|
||||
6aa01a937f1f3c6504ee92e847cf7b3a884e88aa4ef37a68b0b7d4e7deed4bf2 manifest.json
|
||||
@@ -0,0 +1,508 @@
|
||||
"""Fail-closed byte-exact audit of the shared-APL closed-loop fixtures.
|
||||
|
||||
Comparison is independent of either backend's stored PASS. No tolerance is
|
||||
used, timing fields are never compared, and reports contain no wall timestamp.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import re
|
||||
import struct
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
N_GLOBAL = 166700
|
||||
STATE_KEYS = ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref')
|
||||
REQUEST_FIELDS = ('tick', 'neuron_idx', 'attempted', 'accepted', 'rejected')
|
||||
HASH_FIELDS = {'tick', 'v_sha256', 'g_sha256', 'ref_sha256'}
|
||||
QUEUE_FIELDS = {'tick', 'count', 'accepted_count', 'ordered_edge_sha256',
|
||||
'accepted_mask_sha256', 'joint_sha256'}
|
||||
BUNDLE_FILES = ('input_matrix.npy', 'output_matrix.npy', 'kc_globals.npy', 'apl_globals.npy')
|
||||
COEFFICIENTS = ('gd', 'c0', 'cg', 'cv', 'threshold', 'reset', 'seed_volt')
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def file_hash(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open('rb') as source:
|
||||
for block in iter(lambda: source.read(8 << 20), b''):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def raw_hash(array, dtype=None):
|
||||
return hashlib.sha256(np.ascontiguousarray(array, dtype=dtype).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def save(path, report):
|
||||
if path is None:
|
||||
return
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open('x', encoding='utf-8') as target:
|
||||
json.dump(report, target, indent=2, sort_keys=True, allow_nan=False)
|
||||
target.write('\n')
|
||||
|
||||
|
||||
def _read_json(path, used):
|
||||
used.add(Path(path))
|
||||
return json.loads(Path(path).read_text(encoding='utf-8'))
|
||||
|
||||
|
||||
def _read_jsonl(path, used):
|
||||
used.add(Path(path))
|
||||
with Path(path).open(encoding='utf-8') as source:
|
||||
return [json.loads(line) for line in source if line.strip()]
|
||||
|
||||
|
||||
def _read_array(path, used):
|
||||
used.add(Path(path))
|
||||
return np.load(path, mmap_mode='r', allow_pickle=False)
|
||||
|
||||
|
||||
def _check(checks, name, condition):
|
||||
checks[name] = bool(condition)
|
||||
|
||||
|
||||
def _bit_mask(a, b):
|
||||
if a.dtype.kind == 'f':
|
||||
unsigned = np.dtype('uint'+str(a.dtype.itemsize*8))
|
||||
return np.ascontiguousarray(a).view(unsigned) != np.ascontiguousarray(b).view(unsigned)
|
||||
return a != b
|
||||
|
||||
|
||||
def _compare_array(name, a, b, checks, differences, dtype=None, shape=None):
|
||||
a, b = np.asarray(a), np.asarray(b)
|
||||
schema = a.shape == b.shape and a.dtype == b.dtype and \
|
||||
(dtype is None or a.dtype == np.dtype(dtype)) and (shape is None or a.shape == shape)
|
||||
_check(checks, name+'.schema', schema)
|
||||
if not schema:
|
||||
differences[name] = {'left_shape': list(a.shape), 'right_shape': list(b.shape),
|
||||
'left_dtype': str(a.dtype), 'right_dtype': str(b.dtype), 'expected_shape': None if shape is None else list(shape),
|
||||
'expected_dtype': None if dtype is None else str(np.dtype(dtype))}
|
||||
_check(checks, name+'.bitwise_exact', False)
|
||||
return False
|
||||
finite = bool(np.isfinite(a).all() and np.isfinite(b).all())
|
||||
_check(checks, name+'.finite', finite)
|
||||
mask = _bit_mask(a, b)
|
||||
exact = not bool(np.any(mask))
|
||||
_check(checks, name+'.bitwise_exact', exact)
|
||||
if not exact or not finite:
|
||||
flat = np.flatnonzero(mask.ravel())
|
||||
detail = {'mismatching_elements': int(len(flat)),
|
||||
'first_positions': [list(map(int, np.unravel_index(int(index), a.shape))) for index in flat[:5]],
|
||||
'left_nonfinite': int(np.count_nonzero(~np.isfinite(a))),
|
||||
'right_nonfinite': int(np.count_nonzero(~np.isfinite(b)))}
|
||||
if finite and a.size:
|
||||
error = np.abs(a.astype(np.float64)-b.astype(np.float64))
|
||||
detail.update(max_abs_error=float(error.max()), mean_abs_error=float(error.mean()))
|
||||
if a.dtype.kind == 'f' and len(flat):
|
||||
unsigned = np.dtype('uint'+str(a.dtype.itemsize*8))
|
||||
av = np.ascontiguousarray(a).view(unsigned).ravel()
|
||||
bv = np.ascontiguousarray(b).view(unsigned).ravel()
|
||||
detail['first_bit_patterns'] = [{'left': hex(int(av[index])), 'right': hex(int(bv[index]))} for index in flat[:5]]
|
||||
differences[name] = detail
|
||||
return exact and finite
|
||||
|
||||
|
||||
def _sha(value):
|
||||
return isinstance(value, str) and re.fullmatch('[0-9a-f]{64}', value) is not None
|
||||
|
||||
|
||||
def _read_requests(path, steps, used):
|
||||
used.add(Path(path))
|
||||
with Path(path).open(encoding='utf-8', newline='') as source:
|
||||
reader = csv.DictReader(source)
|
||||
require(tuple(reader.fieldnames or ()) == REQUEST_FIELDS, 'Request CSV schema mismatch: '+str(path))
|
||||
rows = [{key: int(row[key]) for key in REQUEST_FIELDS} for row in reader]
|
||||
previous = (-1, -1)
|
||||
for row in rows:
|
||||
key = row['tick'], row['neuron_idx']
|
||||
require(previous < key and 0 <= key[0] < steps and 0 <= key[1] < N_GLOBAL,
|
||||
'Request rows must be sorted, unique, and in range: '+str(path))
|
||||
require(row['attempted'] == 1 and row['accepted'] in (0, 1) and row['rejected'] == 1-row['accepted'],
|
||||
'Request accounting is incomplete: '+str(path))
|
||||
previous = key
|
||||
return rows
|
||||
|
||||
|
||||
def _load_run(directory, role, n, edges, steps, record_history, name, used, checks):
|
||||
directory = Path(directory)
|
||||
summary = _read_json(directory/'summary.json', used)
|
||||
_check(checks, role+'.summary_complete', summary.get('complete') is True and summary.get('pass_all') is True and
|
||||
summary.get('steps') == steps and summary.get('completed_ticks') == steps and
|
||||
summary.get('nodes') == n and summary.get('edges') == edges and summary.get('precision') == 'float64')
|
||||
is_cpu = role == 'cpu'
|
||||
_check(checks, role+'.case_name', summary.get('name' if is_cpu else 'case') == name)
|
||||
_check(checks, role+'.history_mode', summary.get('history_recorded' if is_cpu else 'record_history') is record_history)
|
||||
_check(checks, role+'.apl_precision', summary.get('APL_precision' if is_cpu else 'apl_precision') == 'float32')
|
||||
_check(checks, role+'.no_error_marker', not (directory/'error.json').exists())
|
||||
used.add(directory/'state.npz')
|
||||
with np.load(directory/'state.npz', allow_pickle=False) as source:
|
||||
require(set(source.files) == set(STATE_KEYS), 'State NPZ keys changed: '+str(directory))
|
||||
state = {key: source[key].copy() for key in STATE_KEYS}
|
||||
for key, dtype in (('v', 'float64'), ('g', 'float64'), ('ref', 'int32'), ('counts', 'int32')):
|
||||
require(state[key].shape == (n,) and state[key].dtype == np.dtype(dtype), 'Invalid final '+key+': '+str(directory))
|
||||
require(state['step'].dtype == state['neuron_idx'].dtype == np.int32 and state['step'].ndim == 1 and
|
||||
state['step'].shape == state['neuron_idx'].shape, 'Invalid sparse event schema: '+str(directory))
|
||||
require(bool(np.all((state['step'] >= 0) & (state['step'] < steps))) and
|
||||
bool(np.all((state['neuron_idx'] >= 0) & (state['neuron_idx'] < N_GLOBAL))), 'Sparse event range error')
|
||||
event_keys = state['step'].astype(np.int64)*N_GLOBAL+state['neuron_idx']
|
||||
_check(checks, role+'.events_sorted_unique', len(event_keys) < 2 or np.all(np.diff(event_keys) > 0))
|
||||
_check(checks, role+'.count_nonnegative_total', np.all(state['counts'] >= 0) and int(state['counts'].sum()) == len(event_keys))
|
||||
_check(checks, role+'.ref_range', np.all((state['ref'] >= 0) & (state['ref'] <= 22)))
|
||||
_check(checks, role+'.reported_events', summary.get('spikes' if is_cpu else 'total_spikes') == len(event_keys))
|
||||
hashes = _read_jsonl(directory/'tick-state-hashes.jsonl', used)
|
||||
require(len(hashes) == steps and [row.get('tick') for row in hashes] == list(range(steps)), 'Incomplete tick hash coverage')
|
||||
require(all(set(row) == HASH_FIELDS and all(_sha(row[key]) for key in HASH_FIELDS-{'tick'}) for row in hashes),
|
||||
'Invalid tick state hash schema')
|
||||
queues = _read_jsonl(directory/'queue.jsonl', used)
|
||||
require(len(queues) == steps and [row.get('tick') for row in queues] == list(range(steps)), 'Incomplete queue coverage')
|
||||
require(all(set(row) == QUEUE_FIELDS and type(row['count']) is int and type(row['accepted_count']) is int and
|
||||
0 <= row['accepted_count'] <= row['count'] <= edges and
|
||||
all(_sha(row[key]) for key in QUEUE_FIELDS-{'tick', 'count', 'accepted_count'}) for row in queues),
|
||||
'Invalid queue row schema/count/hash')
|
||||
for key in ('v', 'g', 'ref'):
|
||||
_check(checks, role+'.last_hash_'+key, hashes[-1][key+'_sha256'] == raw_hash(state[key], '<i4' if key == 'ref' else '<f8'))
|
||||
requests = _read_requests(directory/'requests.csv', steps, used)
|
||||
requested, accepted = len(requests), sum(row['accepted'] for row in requests)
|
||||
_check(checks, role+'.request_summary', summary.get('requests' if is_cpu else 'requested') == requested and
|
||||
summary.get('applicable_requests' if is_cpu else 'accepted_requests') == accepted and
|
||||
summary.get('ref_rejected_requests' if is_cpu else 'rejected_requests') == requested-accepted)
|
||||
apl = {field: _read_array(directory/('apl-'+field+'.float32.npy'), used) for field in ('drive', 'state', 'inhibition')}
|
||||
for field, width in (('drive', 2), ('state', 2), ('inhibition', 4064)):
|
||||
require(apl[field].dtype == np.float32 and apl[field].shape == (steps, width), 'Invalid APL '+field+' history schema')
|
||||
_check(checks, role+'.apl_positive_'+field, np.all(apl[field] >= 0))
|
||||
histories, pre = {}, {}
|
||||
for field, dtype in (('v', 'float64'), ('g', 'float64'), ('ref', 'int32')):
|
||||
files = (directory/(field+'-history.'+dtype+'.npy'), directory/('preAPL-'+field+'.'+dtype+'.npy'))
|
||||
if record_history:
|
||||
histories[field], pre[field] = (_read_array(path, used) for path in files)
|
||||
require(all(array.dtype == np.dtype(dtype) and array.shape == (steps, n) for array in (histories[field], pre[field])),
|
||||
'Small pre/post history schema mismatch')
|
||||
_check(checks, role+'.history_hashes_'+field, all(row[field+'_sha256'] == raw_hash(histories[field][tick])
|
||||
for tick, row in enumerate(hashes)))
|
||||
_check(checks, role+'.history_final_'+field, not np.any(_bit_mask(histories[field][-1], state[field])))
|
||||
else:
|
||||
_check(checks, role+'.no_dense_'+field, not any(path.exists() for path in files))
|
||||
if is_cpu:
|
||||
component = _read_json(directory/'apl-worker-handshake.json', used)['handshake']
|
||||
else:
|
||||
component = _read_json(directory/'apl-component.json', used)
|
||||
_check(checks, role+'.component_summary_consistent', summary.get('apl_component') == component)
|
||||
return dict(directory=directory, summary=summary, state=state, hashes=hashes, queues=queues,
|
||||
requests=requests, apl=apl, histories=histories, pre=pre, component=component)
|
||||
|
||||
|
||||
def _pair_outputs(left, right, checks, differences, n, steps, record_history):
|
||||
for field in STATE_KEYS:
|
||||
shape = None if field in ('step', 'neuron_idx') else (n,)
|
||||
dtype = 'float64' if field in ('v', 'g') else 'int32'
|
||||
_compare_array('state.'+field, left['state'][field], right['state'][field], checks, differences, dtype, shape)
|
||||
for field, width in (('drive', 2), ('state', 2), ('inhibition', 4064)):
|
||||
_compare_array('apl.'+field, left['apl'][field], right['apl'][field], checks, differences, 'float32', (steps, width))
|
||||
for field in ('hashes', 'queues', 'requests', 'component'):
|
||||
same = left[field] == right[field]
|
||||
_check(checks, field+'.parsed_exact', same)
|
||||
if not same:
|
||||
if isinstance(left[field], list):
|
||||
first = next((i for i, (a, b) in enumerate(zip(left[field], right[field])) if a != b), None)
|
||||
differences[field] = {'left_count': len(left[field]), 'right_count': len(right[field]),
|
||||
'first_differing_row': first,
|
||||
'left_row': None if first is None else left[field][first],
|
||||
'right_row': None if first is None else right[field][first]}
|
||||
else:
|
||||
differences[field] = {'left': left[field], 'right': right[field]}
|
||||
if record_history:
|
||||
for stage in ('histories', 'pre'):
|
||||
for field, dtype in (('v', 'float64'), ('g', 'float64'), ('ref', 'int32')):
|
||||
_compare_array(stage+'.'+field, left[stage][field], right[stage][field], checks, differences, dtype, (steps, n))
|
||||
|
||||
|
||||
def _queue_digest(tick, ids, accepted):
|
||||
ids = np.ascontiguousarray(ids, dtype='<i8').tobytes()
|
||||
mask = np.ascontiguousarray(accepted, dtype=np.uint8).tobytes()
|
||||
return {'tick': tick, 'count': len(mask), 'accepted_count': int(np.count_nonzero(accepted)),
|
||||
'ordered_edge_sha256': hashlib.sha256(ids).hexdigest(),
|
||||
'accepted_mask_sha256': hashlib.sha256(mask).hexdigest(),
|
||||
'joint_sha256': hashlib.sha256(b'phase4af-queue-v1\0'+struct.pack('<qq', tick, len(mask))+ids+mask).hexdigest()}
|
||||
|
||||
|
||||
def _independent_event_accounting(run, role, graph, requested, steps, checks, differences):
|
||||
nodes, ptr = graph['nodes'], graph['ptr']
|
||||
n = len(nodes)
|
||||
state = run['state']
|
||||
local = np.searchsorted(nodes, state['neuron_idx'])
|
||||
require(np.all(local < n) and np.array_equal(nodes[local], state['neuron_idx']), 'Actual spike outside graph identity')
|
||||
_check(checks, role+'.events_per_neuron_counts', np.array_equal(np.bincount(local, minlength=n), state['counts']))
|
||||
actual_request_pairs = [(row['tick'], row['neuron_idx']) for row in run['requests']]
|
||||
_check(checks, role+'.requests_match_frozen_input', actual_request_pairs == requested)
|
||||
require(actual_request_pairs == requested, 'Actual request population/time differs from frozen input')
|
||||
bounds = np.searchsorted(state['step'], np.arange(steps+1))
|
||||
request_ticks = np.asarray([row['tick'] for row in run['requests']], np.int32)
|
||||
rbounds = np.searchsorted(request_ticks, np.arange(steps+1))
|
||||
request_nodes = np.asarray([row['neuron_idx'] for row in run['requests']], np.int32)
|
||||
request_local = np.searchsorted(nodes, request_nodes)
|
||||
require(np.all(request_local < n) and np.array_equal(nodes[request_local], request_nodes), 'Request outside graph')
|
||||
last = np.full(n, -1_000_000_000, np.int64)
|
||||
degrees = np.diff(ptr)
|
||||
expected_counts = np.zeros(steps, np.int64)
|
||||
arrival_tick = state['step'].astype(np.int64)+18
|
||||
valid = arrival_tick < steps
|
||||
np.add.at(expected_counts, arrival_tick[valid], degrees[local[valid]])
|
||||
_check(checks, role+'.delayed_all_edges_count', np.array_equal(expected_counts, [row['count'] for row in run['queues']]))
|
||||
request_ok, delivered_ok, ref_ok, queue_ok, reset_ok = True, True, True, True, True
|
||||
first = {}
|
||||
for tick in range(steps):
|
||||
fired = local[bounds[tick]:bounds[tick+1]]
|
||||
lo, hi = rbounds[tick:tick+2]
|
||||
targets = request_local[lo:hi]
|
||||
accepted = (tick-last[targets]) >= 22
|
||||
for index, ok in zip(range(lo, hi), accepted):
|
||||
row = run['requests'][index]
|
||||
if row['accepted'] != int(ok) or row['rejected'] != int(not ok):
|
||||
request_ok = False
|
||||
first.setdefault('request_ref_mismatch', {'tick': tick, 'neuron_idx': row['neuron_idx']})
|
||||
if not np.isin(targets[accepted], fired).all() or np.isin(targets[~accepted], fired).any():
|
||||
delivered_ok = False
|
||||
first.setdefault('request_delivery_mismatch', tick)
|
||||
last[fired] = tick
|
||||
ref = np.maximum(22-(tick-last), 0).astype(np.int32)
|
||||
if raw_hash(ref, '<i4') != run['hashes'][tick]['ref_sha256']:
|
||||
ref_ok = False
|
||||
first.setdefault('ref_hash_mismatch', tick)
|
||||
if run['pre']:
|
||||
if len(fired) and not np.array_equal(np.ascontiguousarray(run['pre']['g'][tick, fired]).view(np.uint64), np.zeros(len(fired), np.uint64)):
|
||||
reset_ok = False
|
||||
first.setdefault('pre_apl_spike_reset_mismatch', tick)
|
||||
due = local[bounds[tick-18]:bounds[tick-17]] if tick >= 18 else np.empty(0, np.int64)
|
||||
chunks = [np.arange(ptr[source], ptr[source+1], dtype=np.int64) for source in due]
|
||||
indices = np.concatenate(chunks) if chunks else np.empty(0, np.int64)
|
||||
expected_queue = _queue_digest(tick, graph['edge_id'][indices], (ref[graph['dst'][indices]] == 0).astype(np.uint8))
|
||||
if expected_queue != run['queues'][tick]:
|
||||
queue_ok = False
|
||||
first.setdefault('queue_reconstruction_mismatch', tick)
|
||||
_check(checks, role+'.request_refractory_gate', request_ok)
|
||||
_check(checks, role+'.accepted_spikes_rejected_absent', delivered_ok)
|
||||
_check(checks, role+'.all_ref_hashes_from_actual_events', ref_ok)
|
||||
if run['pre']:
|
||||
_check(checks, role+'.small_queue_independent_reconstruction', queue_ok)
|
||||
_check(checks, role+'.spike_g_reset_before_apl', reset_ok)
|
||||
if first:
|
||||
differences[role+'.independent_accounting'] = first
|
||||
|
||||
|
||||
def _check_apl_application(run, role, nodes, kc, checks, differences):
|
||||
if not run['pre']:
|
||||
return
|
||||
common, local_kc, bundle_kc = np.intersect1d(nodes, kc, assume_unique=True, return_indices=True)
|
||||
_compare_array(role+'.apl_keeps_voltage', run['pre']['v'], run['histories']['v'], checks, differences, 'float64')
|
||||
_compare_array(role+'.apl_keeps_ref', run['pre']['ref'], run['histories']['ref'], checks, differences, 'int32')
|
||||
expected = np.array(run['pre']['g'], copy=True)
|
||||
delta = run['apl']['inhibition'][:, bundle_kc].astype(np.float64)*np.float64(.001)
|
||||
expected[:, local_kc] = expected[:, local_kc]-delta
|
||||
_compare_array(role+'.apl_unconditional_g_write', expected, run['histories']['g'], checks, differences, 'float64')
|
||||
|
||||
|
||||
def _finish(report, checks, differences, used, out_json):
|
||||
report['checks'] = {name: bool(value) for name, value in checks.items()}
|
||||
report['differences'] = differences
|
||||
report['pass_all'] = bool(checks) and all(checks.values()) and 'error' not in report
|
||||
report['failed_checks'] = sorted(name for name, value in checks.items() if not value)
|
||||
report['evidence_sha256'] = {str(path): file_hash(path) for path in sorted(used) if path.is_file()}
|
||||
save(out_json, report)
|
||||
return report
|
||||
|
||||
|
||||
def compare(case_dir, gpu_dir, cpu_dir, out_json=None):
|
||||
"""Strict independent closed-loop comparison; out_json=None is read-only."""
|
||||
case_dir, gpu_dir, cpu_dir = Path(case_dir), Path(gpu_dir), Path(cpu_dir)
|
||||
if out_json is not None:
|
||||
require(not Path(out_json).exists(), 'Refusing existing comparison report')
|
||||
checks, differences, used = {}, {}, set()
|
||||
report = {'kind': 'shared-APL-component closed-loop byte-exact comparison', 'pass_all': False,
|
||||
'case_dir': str(case_dir), 'gpu_dir': str(gpu_dir), 'cpu_dir': str(cpu_dir),
|
||||
'tolerance': None, 'signed_zero_included': True, 'timing_fields_compared': False,
|
||||
'independent_CPU_all_components_claim': False}
|
||||
try:
|
||||
spec = _read_json(case_dir/'spec.json', used)
|
||||
steps, history = spec['steps'], spec['record_history']
|
||||
require(type(steps) is int and steps > 0 and type(history) is bool and spec['dt_ms'] == .1,
|
||||
'Invalid fixed case clock/history schema')
|
||||
graph_dir, bundle_dir = Path(spec['graph_dir']), Path(spec['apl_bundle_dir'])
|
||||
graph = {field: _read_array(graph_dir/(field+'.npy'), used) for field in
|
||||
('nodes', 'edge_id', 'src', 'dst', 'ptr', 'signed_raw', 'weights')}
|
||||
nodes = graph['nodes']
|
||||
n, edges = len(nodes), len(graph['edge_id'])
|
||||
require(n > 0 and edges > 0 and nodes.dtype == np.int32 and nodes.ndim == 1 and np.all((nodes >= 0) & (nodes < N_GLOBAL)) and
|
||||
np.all(np.diff(nodes) > 0), 'Invalid canonical node identities')
|
||||
require(graph['ptr'].dtype == np.int64 and graph['ptr'].shape == (n+1,) and graph['ptr'][0] == 0 and
|
||||
graph['ptr'][-1] == edges and np.all(np.diff(graph['ptr']) >= 0), 'Invalid CSR')
|
||||
for key, dtype in (('edge_id', 'int64'), ('src', 'int32'), ('dst', 'int32'), ('signed_raw', 'int64'), ('weights', 'float64')):
|
||||
require(graph[key].dtype == np.dtype(dtype) and graph[key].shape == (edges,), 'Invalid graph '+key)
|
||||
require(np.array_equal(graph['src'], np.repeat(np.arange(n, dtype=np.int32), np.diff(graph['ptr']))) and
|
||||
np.all((graph['dst'] >= 0) & (graph['dst'] < n)), 'Invalid source/destination row mapping')
|
||||
require(history == (n < 512) and (n < 512 or n == N_GLOBAL), 'Invalid small/full history declaration')
|
||||
require(np.all(graph['edge_id'] >= 0) and np.all(np.diff(graph['edge_id']) > 0),
|
||||
'Original canonical edge IDs must be unique and increasing')
|
||||
if n == N_GLOBAL:
|
||||
require(edges == 25582938 and np.array_equal(nodes, np.arange(n, dtype=np.int32)) and
|
||||
np.array_equal(graph['edge_id'], np.arange(edges, dtype=np.int64)), 'Full canonical graph identity changed')
|
||||
bundle = {name: _read_array(bundle_dir/name, used) for name in BUNDLE_FILES}
|
||||
kc, apl_ids = bundle['kc_globals.npy'], bundle['apl_globals.npy']
|
||||
require(kc.dtype == apl_ids.dtype == np.int32 and kc.shape == (4064,) and apl_ids.shape == (2,), 'Invalid APL/KC identities')
|
||||
require(np.all((kc >= 0) & (kc < N_GLOBAL)) and np.all(np.diff(kc) > 0) and
|
||||
np.all((apl_ids >= 0) & (apl_ids < N_GLOBAL)) and np.all(np.diff(apl_ids) > 0) and
|
||||
not np.intersect1d(kc, apl_ids).size, 'APL/KC identities overlap or are invalid')
|
||||
for key, shape in (('input_matrix.npy', (2, N_GLOBAL)), ('output_matrix.npy', (2, 4064))):
|
||||
require(bundle[key].dtype == np.float32 and bundle[key].shape == shape and
|
||||
np.isfinite(bundle[key]).all() and np.all(bundle[key] >= 0), 'Invalid normalized APL matrix')
|
||||
gpu = _load_run(gpu_dir, 'gpu', n, edges, steps, history, spec['name'], used, checks)
|
||||
cpu = _load_run(cpu_dir, 'cpu', n, edges, steps, history, spec['name'], used, checks)
|
||||
_pair_outputs(gpu, cpu, checks, differences, n, steps, history)
|
||||
used.add(case_dir/'requests.npz')
|
||||
with np.load(case_dir/'requests.npz', allow_pickle=False) as source:
|
||||
require(set(source.files) == {'tick', 'neuron_idx'} and source['tick'].dtype == source['neuron_idx'].dtype == np.int32,
|
||||
'Fixed request NPZ schema changed')
|
||||
require(source['tick'].shape == source['neuron_idx'].shape and source['tick'].ndim == 1, 'Invalid sparse request shape')
|
||||
requested = list(zip(map(int, source['tick']), map(int, source['neuron_idx'])))
|
||||
for role, run in (('gpu', gpu), ('cpu', cpu)):
|
||||
_independent_event_accounting(run, role, graph, requested, steps, checks, differences)
|
||||
_check_apl_application(run, role, nodes, kc, checks, differences)
|
||||
|
||||
# Independently derive the effective clone. Original graph weights stay read-only.
|
||||
cpu_weights = _read_array(cpu_dir/'weights.npy', used)
|
||||
require(cpu_weights.dtype == np.float64 and cpu_weights.shape == (edges,), 'CPU effective weight schema mismatch')
|
||||
expected_hash, base_hash, mask_hash = hashlib.sha256(), hashlib.sha256(), hashlib.sha256()
|
||||
weight_ok, mask_count = True, 0
|
||||
for start in range(0, edges, 1 << 20):
|
||||
stop = min(edges, start+(1 << 20))
|
||||
original = np.asarray(graph['weights'][start:stop])
|
||||
require(np.isfinite(original).all(), 'Nonfinite original weights')
|
||||
raw_expected = graph['signed_raw'][start:stop]*(.275*.001)
|
||||
require(not np.any(_bit_mask(original, raw_expected)), 'Original SI weights differ from unaltered signed contacts')
|
||||
mask = np.isin(nodes[graph['src'][start:stop]], apl_ids)
|
||||
expected = original.copy()
|
||||
expected[mask] = np.float64(0)
|
||||
mask_count += int(mask.sum())
|
||||
expected_hash.update(np.ascontiguousarray(expected, dtype='<f8').tobytes())
|
||||
base_hash.update(np.ascontiguousarray(original, dtype='<f8').tobytes())
|
||||
mask_hash.update(np.ascontiguousarray(graph['edge_id'][start:stop][mask], dtype='<i8').tobytes())
|
||||
weight_ok &= not np.any(_bit_mask(expected, cpu_weights[start:stop]))
|
||||
expected_sha = expected_hash.hexdigest()
|
||||
_check(checks, 'cpu.effective_weights_independent_bytes', weight_ok)
|
||||
for role, run in (('gpu', gpu), ('cpu', cpu)):
|
||||
_check(checks, role+'.effective_weights_raw_sha', run['summary'].get('effective_weights_raw_sha256') == expected_sha)
|
||||
_check(checks, role+'.apl_fast_edge_count', run['summary'].get('apl_fast_edges_zeroed') == mask_count)
|
||||
_check(checks, role+'.kc_target_count', run['summary'].get('kc_targets_present') == len(np.intersect1d(nodes, kc)))
|
||||
mask_audit = _read_json(cpu_dir/'apl-mask-audit.json', used)
|
||||
_check(checks, 'cpu.mask_provenance', mask_audit['count'] == mask_count and
|
||||
mask_audit['ordered_original_edge_id_sha256'] == mask_hash.hexdigest() and
|
||||
mask_audit['base_weight_raw_bytes_sha256'] == base_hash.hexdigest() and
|
||||
mask_audit['effective_weights_raw_sha256'] == expected_sha and mask_audit['base_graph_files_modified'] is False)
|
||||
|
||||
case_hashes = {name: file_hash(case_dir/name) for name in ('spec.json', 'requests.npz')}
|
||||
if (case_dir/'arrays.npz').exists():
|
||||
case_hashes['arrays.npz'] = file_hash(case_dir/'arrays.npz')
|
||||
used.add(case_dir/'arrays.npz')
|
||||
baseline_coeff = _read_json(Path(spec['coefficients_path']), used)
|
||||
actual_coeff = _read_json(cpu_dir/'coefficients.json', used)
|
||||
_check(checks, 'cpu.actual_author_coefficients_exact', all(float(actual_coeff[key]).hex() == float(baseline_coeff[key]).hex()
|
||||
for key in COEFFICIENTS))
|
||||
audit = _read_json(cpu_dir/'input-audit.json', used)
|
||||
expected_graph_hashes = {name: file_hash(graph_dir/name) for name in
|
||||
tuple(field+'.npy' for field in graph)+('completeness.csv', 'connectivity.parquet')}
|
||||
bundle_hashes = {name: file_hash(bundle_dir/name) for name in BUNDLE_FILES}
|
||||
_check(checks, 'cpu.read_actual_case', audit['spec'] == spec and audit['case_hashes'] == case_hashes)
|
||||
_check(checks, 'cpu.graph_hashes', audit['graph_hashes'] == expected_graph_hashes)
|
||||
_check(checks, 'cpu.bundle_hashes', audit['bundle_hashes'] == bundle_hashes)
|
||||
_check(checks, 'cpu.no_candidate_trace_input', audit.get('candidate_trace_used_as_input') is False and
|
||||
cpu['summary'].get('candidate_trace_used_as_input') is False)
|
||||
_check(checks, 'cpu.worker_source_hash', audit['legacy_apl_worker_sha256'] == file_hash(spec['legacy_apl_worker']))
|
||||
_check(checks, 'cpu.coefficient_input_hash', audit['coefficients_sha256'] == file_hash(spec['coefficients_path']))
|
||||
gpu_hashes = dict(case_hashes, **{'base_weights.npy': file_hash(graph_dir/'weights.npy'),
|
||||
'coefficients.json': file_hash(spec['coefficients_path'])})
|
||||
_check(checks, 'gpu.input_hashes', gpu['summary']['input_sha256'] == gpu_hashes)
|
||||
for role, run in (('gpu', gpu), ('cpu', cpu)):
|
||||
component = run['component']
|
||||
_check(checks, role+'.actual_component_bundle_bytes', component['files_sha256'] == bundle_hashes and
|
||||
component['array_sha256'] == {name: raw_hash(value) for name, value in bundle.items()})
|
||||
_check(checks, role+'.component_dimensions_clock', component.get('ok') is True and
|
||||
component['protocol'] == 'phase4ag-apl-v1' and component['n_neurons'] == N_GLOBAL and
|
||||
component['n_apl'] == 2 and component['n_kc'] == 4064 and component['dt_ms'] == .1)
|
||||
decay = np.float32(math.exp(-.1/20.0))
|
||||
gain = np.float32(.01)
|
||||
_check(checks, role+'.frozen_apl_constants', component['decay_f32_bits'] == f'{int(decay.view(np.uint32)):08x}' and
|
||||
component['gain_f32_bits'] == f'{int(gain.view(np.uint32)):08x}')
|
||||
env = component['environment']
|
||||
_check(checks, role+'.component_precision_environment', env['cuda_matmul_allow_tf32'] is False and
|
||||
env['float32_matmul_precision'] == 'highest' and env['precision_configuration_changed_by_component'] is False and
|
||||
'4090' in env['device_name'])
|
||||
environment = _read_json(cpu_dir/'environment.json', used)
|
||||
_check(checks, 'cpu.pinned_environment', environment['packages']['brian2'] == '2.5.1' and
|
||||
environment['packages']['numpy'] == '1.22.3' and environment['packages']['sympy'] == '1.12' and
|
||||
environment['Torch_imported_into_Brian'] is False and environment['author_precision'] == 'float64')
|
||||
queue_audit = _read_json(cpu_dir/'queue-audit.json', used)
|
||||
_check(checks, 'cpu.queue_observer_coverage', queue_audit['peek_calls'] == queue_audit['advance_calls'] == steps and
|
||||
queue_audit['same_peek_object_returned'] is True and queue_audit['zero_weight_and_refractory_rejected_edges_retained'] is True)
|
||||
_check(checks, 'cpu.sidecar_complete', cpu['summary']['APL_worker_exit_status'] == 0 and cpu['summary']['APL_worker_ticks'] == steps)
|
||||
report.update(case=spec['name'], steps=steps, nodes=n, edges=edges, record_history=history,
|
||||
total_spikes=len(gpu['state']['step']), active_neurons=int(np.count_nonzero(gpu['state']['counts'])),
|
||||
requested=len(requested), accepted_requests=sum(row['accepted'] for row in gpu['requests']),
|
||||
rejected_requests=sum(row['rejected'] for row in gpu['requests']),
|
||||
arriving_edges=sum(row['count'] for row in gpu['queues']),
|
||||
accepted_arriving_edges=sum(row['accepted_count'] for row in gpu['queues']),
|
||||
apl_fast_edges_zeroed=mask_count, effective_weights_raw_sha256=expected_sha,
|
||||
component_environment=gpu['component']['environment'],
|
||||
limits=['APL Torch primitives are shared; original AST component validation is a separate required gate.',
|
||||
'Small cases preserve and independently hash all pre/post states; full cases compare all-tick instrumented state hashes without saving dense whole-brain histories.',
|
||||
'For full cases intermediate finite checks are performed by the instrumented backends; this file independently inspects final states and all APL traces.',
|
||||
'This is exact agreement for the declared trajectory, not physiological validity or learning.'])
|
||||
except Exception as error:
|
||||
report['error'] = {'type': type(error).__name__, 'message': str(error)}
|
||||
return _finish(report, checks, differences, used, out_json)
|
||||
|
||||
|
||||
def compare_gpu_repeats(first, second, out_json=None):
|
||||
"""Repeat equality only; the first run's independent oracle gate is separate."""
|
||||
first, second = Path(first), Path(second)
|
||||
if out_json is not None:
|
||||
require(not Path(out_json).exists(), 'Refusing existing repeat report')
|
||||
checks, differences, used = {}, {}, set()
|
||||
report = {'kind': 'fresh-GPU-run exact repeat comparison', 'first': str(first), 'second': str(second),
|
||||
'pass_all': False, 'tolerance': None, 'signed_zero_included': True, 'timing_fields_compared': False,
|
||||
'scope': 'Scientific outputs, recording integrity and input hashes; independent case/graph audit belongs to compare()'}
|
||||
try:
|
||||
summary = _read_json(first/'summary.json', used)
|
||||
n, edges, steps, name, history = (summary[key] for key in ('nodes', 'edges', 'steps', 'case', 'record_history'))
|
||||
require(type(n) is int and type(edges) is int and type(steps) is int and n > 0 and edges > 0 and steps > 0 and
|
||||
type(history) is bool and history == (n < 512), 'Invalid repeat dimensions')
|
||||
left = _load_run(first, 'first_gpu', n, edges, steps, history, name, used, checks)
|
||||
right = _load_run(second, 'second_gpu', n, edges, steps, history, name, used, checks)
|
||||
_pair_outputs(left, right, checks, differences, n, steps, history)
|
||||
stable = ('case', 'complete', 'pass_all', 'steps', 'completed_ticks', 'nodes', 'edges', 'precision',
|
||||
'apl_precision', 'state_units', 'total_spikes', 'requested', 'accepted_requests', 'rejected_requests',
|
||||
'arriving_edges', 'accepted_arriving_edges', 'counts_match_sparse_events', 'all_finite', 'record_history',
|
||||
'apl_fast_edges_zeroed', 'kc_targets_present', 'effective_weights_raw_sha256', 'apl_schedule',
|
||||
'apl_si_conversion', 'build', 'sealed_ae_backend_sha256', 'learning', 'input_sha256')
|
||||
for key in stable:
|
||||
_check(checks, 'summary.'+key, key in left['summary'] and key in right['summary'] and
|
||||
left['summary'][key] == right['summary'][key])
|
||||
if not checks['summary.'+key]:
|
||||
differences['summary.'+key] = {'first': left['summary'].get(key), 'second': right['summary'].get(key)}
|
||||
report.update(case=name, steps=steps, nodes=n, edges=edges, record_history=history,
|
||||
total_spikes=len(left['state']['step']),
|
||||
explicit_exclusions=['wall_seconds', 'diagnostic_simulation_seconds', 'build_or_cache_seconds',
|
||||
'peak_allocated_vram_bytes', 'peak_reserved_vram_bytes'])
|
||||
except Exception as error:
|
||||
report['error'] = {'type': type(error).__name__, 'message': str(error)}
|
||||
return _finish(report, checks, differences, used, out_json)
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"phase": "4AG",
|
||||
"name": "author-apl-lab-v1",
|
||||
"human_authorization": "允许独立新模型分支测试动力学假设",
|
||||
"gpu_uuid": "GPU-79f3a85d-adce-7833-7003-6488140ca596",
|
||||
"neurons": 166700,
|
||||
"edges": 25582938,
|
||||
"raw_weight_sum": 124177617,
|
||||
"precision": "float64",
|
||||
"apl_precision": "float32",
|
||||
"apl": {"tau_ms": 20.0, "gain": 0.01, "normalization": 1.0},
|
||||
"hypothesis": "Author LIF plus unchanged legacy graded APL operator at end, after reset, unconditional KC g write. Mixed precision: h32 promoted to float64, then multiplied by float64 .001 SI, then subtracted. Not original author's APL model.",
|
||||
"fixtures": ["zero", "pn_pulse", "kc_ref_boundary", "two_sources", "arrival_collision", "natural_request", "apl_only", "high_fanin"],
|
||||
"fixture_repeats": 3,
|
||||
"calibration": {"split": "calibration", "odor_count": 6, "modes": ["physiological", "matched_energy"], "seeds": [51101,51102,51103,51104,51105], "duration_ms": 100.0, "median_scope": "original frozen 12-odor panel", "full_oracle_probes": ["cal-physiological-00-s51101", "cal-matched_energy-00-s51101"], "repeat_first_seed_each_odor": true},
|
||||
"gates": {"sparse_min_fraction": 0.03, "sparse_max_fraction": 0.09, "max_fraction_strict": 0.5, "min_fraction_strict": 0.0, "minimum_separable_odor_pairs": 3, "same_jaccard_gt_between": true, "between_euclidean_gt_within": true, "both_modes_required": true, "finite_required": true, "repeat_exact_required": true},
|
||||
"training": false,
|
||||
"heldout_dynamic_access": false,
|
||||
"candidate_generation": false,
|
||||
"output_budget_bytes": 16106127360,
|
||||
"scope": "Semantic compatibility and fixed old CAL only. Shared-APL-component closed-loop Brian oracle plus independent arithmetic audit; never claim all-component independent CPU equivalence or learning success. No parameter/input/site/window selection from results."
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
ROOT=Path('/data/digital-drosophila')
|
||||
MODEL=Path(__file__).resolve().parent
|
||||
EXP=ROOT/'experiments/phase4ag-author-apl'
|
||||
AF_MODEL=ROOT/'models/male-cns-lif/author-fullgraph-lab-v1'
|
||||
AF_EXP=ROOT/'experiments/phase4af-author-fullgraph'
|
||||
AD_EXP=ROOT/'experiments/phase4ad-author-semantics'
|
||||
CFG=json.loads((MODEL/'config.json').read_text(encoding='utf-8'))
|
||||
|
||||
def digest(path):
|
||||
h=hashlib.sha256()
|
||||
with Path(path).open('rb') as f:
|
||||
for b in iter(lambda:f.read(8<<20),b''):h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
def save(path,value):
|
||||
path=Path(path);path.parent.mkdir(parents=True,exist_ok=True)
|
||||
with path.open('x',encoding='utf-8') as f:json.dump(value,f,indent=2,ensure_ascii=False,allow_nan=False);f.write('\n')
|
||||
|
||||
def sources():
|
||||
return {p.name:digest(p) for p in sorted(MODEL.iterdir()) if p.is_file() and p.name not in ('manifest.json','checksums.sha256')}
|
||||
|
||||
def integrity():
|
||||
spec=importlib.util.spec_from_file_location('frozen_af_context',AF_MODEL/'context.py')
|
||||
mod=importlib.util.module_from_spec(spec);spec.loader.exec_module(mod)
|
||||
result=mod.integrity()
|
||||
import sys
|
||||
sys.path.insert(0,str(ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'))
|
||||
from diagnostic import manifest_check
|
||||
for p in (AF_MODEL,AF_EXP):result[str(p.relative_to(ROOT))]=manifest_check(p,digest)
|
||||
assert all(r['pass'] for r in result.values())
|
||||
return result
|
||||
|
||||
def check():
|
||||
p=json.loads((EXP/'protocol/preregistered.json').read_text())
|
||||
assert p['sources']==sources(),'Source changed after preregistration'
|
||||
for row in p['inputs']:assert digest(ROOT/row['path'])==row['sha256'],row['path']
|
||||
for row in json.loads((EXP/'protocol/prepared-hashes.json').read_text()):
|
||||
assert digest(EXP/row['path'])==row['sha256'],row['path']
|
||||
return p
|
||||
|
||||
def budget():
|
||||
size=sum(p.stat().st_size for p in EXP.rglob('*') if p.is_file())
|
||||
assert size<CFG['output_budget_bytes'],'New output resource budget exceeded'
|
||||
return size
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Seal a completed gate or preserved failure; never claim learning."""
|
||||
import json
|
||||
from context import ROOT,MODEL,EXP,CFG,check,save,digest,integrity,budget
|
||||
|
||||
|
||||
def seal(root):
|
||||
assert not (root/'manifest.json').exists() and not (root/'checksums.sha256').exists()
|
||||
rows=[dict(path=str(p.relative_to(root)),bytes=p.stat().st_size,sha256=digest(p))
|
||||
for p in sorted(root.rglob('*')) if p.is_file() and '__pycache__' not in p.parts]
|
||||
save(root/'manifest.json',dict(root=str(root),files=rows))
|
||||
with (root/'checksums.sha256').open('x') as f:
|
||||
for row in rows:f.write(row['sha256']+' '+row['path']+'\n')
|
||||
f.write(digest(root/'manifest.json')+' manifest.json\n')
|
||||
|
||||
|
||||
def main():
|
||||
from compare import compare,compare_gpu_repeats
|
||||
check();budget();comparisons=[];repeats={};components={}
|
||||
for p in sorted((EXP/'audit').glob('compare-*.json')):
|
||||
name=p.stem[len('compare-'):]
|
||||
actual=compare(EXP/'cases'/name,EXP/'gpu'/(name+'-r0'),EXP/'oracle'/name,None)
|
||||
assert actual==json.loads(p.read_text()),'Changed raw closed-loop evidence: '+name
|
||||
comparisons.append(dict(case=name,pass_all=actual['pass_all']))
|
||||
for p in sorted((EXP/'audit').glob('repeat-*.json')):
|
||||
name,repeat=p.stem[len('repeat-'):].rsplit('-r',1)
|
||||
actual=compare_gpu_repeats(EXP/'gpu'/(name+'-r0'),EXP/'gpu'/f'{name}-r{repeat}',None)
|
||||
assert actual==json.loads(p.read_text()),'Changed raw repeat evidence: '+name
|
||||
repeats[f'{name}-r{repeat}']=actual['pass_all']
|
||||
for p in (EXP/'component-audit').glob('*.json'):
|
||||
result=json.loads(p.read_text())
|
||||
components[p.stem]=result['pass_all']
|
||||
for filename,sha in result.get('input_sha256',{}).items():
|
||||
assert digest(EXP/'gpu'/(p.stem+'-r0')/filename)==sha
|
||||
gates={}
|
||||
for name in ('fixtures','probes','calibration'):
|
||||
path=EXP/'audit'/(name+'-gate.json')
|
||||
gates[name]=json.loads(path.read_text()) if path.exists() else None
|
||||
calpath=EXP/'calibration/summary.json'
|
||||
cal=json.loads(calpath.read_text()) if calpath.exists() else None
|
||||
if cal is not None:
|
||||
from calibration import summarize
|
||||
assert summarize(out_json=None)==cal,'CAL raw evidence changed'
|
||||
frozen=json.loads((EXP/'regression/frozen-five.json').read_text())
|
||||
after=integrity();save(EXP/'regression/old-integrity-after.json',after)
|
||||
stops=[dict(path=str(p.relative_to(EXP)),report=json.loads(p.read_text())) for p in sorted((EXP/'audit').glob('stopped-*.json'))]
|
||||
expected_cases=CFG['fixtures']+CFG['calibration']['full_oracle_probes']
|
||||
expected_repeats=[f'{name}-r{r}' for name in CFG['fixtures'] for r in range(1,CFG['fixture_repeats'])]
|
||||
compared={x['case']:x['pass_all'] for x in comparisons}
|
||||
missing=dict(comparisons=sorted(set(expected_cases)-set(compared)),
|
||||
components=sorted(set(expected_cases)-set(components)),repeats=sorted(set(expected_repeats)-set(repeats)))
|
||||
evidence_ok=(set(compared)==set(components)==set(expected_cases) and set(repeats)==set(expected_repeats) and
|
||||
all(compared.values()) and all(components.values()) and all(repeats.values()))
|
||||
numerical=evidence_ok and all(gates[x] is not None and gates[x]['pass_all'] for x in ('fixtures','probes'))
|
||||
representation=bool(cal is not None and cal['pass_all'])
|
||||
if numerical:status='PASS-FIXED-CAL-REPRESENTATION' if representation else 'FAIL-FIXED-CAL-REPRESENTATION' if cal is not None else 'INCOMPLETE-CAL-EXECUTION'
|
||||
else:status='FAIL-NUMERICAL-OR-AUDIT-GATE' if any(not p for group in (compared,components,repeats) for p in group.values()) else 'INCOMPLETE-NUMERICAL-EXECUTION'
|
||||
oldpass=bool(frozen['pass_all'] and all(r['pass'] for r in after.values()))
|
||||
if not oldpass:status='FAIL-FROZEN-INTEGRITY-OR-REGRESSION'
|
||||
measurements=[]
|
||||
for p in sorted((EXP/'gpu').glob('*/summary.json')):
|
||||
s=json.loads(p.read_text())
|
||||
measurements.append({k:s[k] for k in ('case','total_spikes','accepted_requests','nodes','edges','diagnostic_simulation_seconds','peak_allocated_vram_bytes')})
|
||||
result=dict(status=status,numerical_gate_pass=bool(numerical),fixed_CAL_representation_pass=representation,
|
||||
gates=gates,closed_loop_comparisons=comparisons,component_results=components,repeat_results=repeats,
|
||||
missing_numerical_evidence=missing,calibration=cal,stopped=stops,
|
||||
old_frozen_integrity_and_five_reference_pass=oldpass,measurements=measurements,
|
||||
unchanged_APL_parameters=CFG['apl'],author_LIF_precision='float64',legacy_APL_precision='float32',
|
||||
explicit_hypothesis=CFG['hypothesis'],APL_nonKC_fast_edges_unreplaced=1212,
|
||||
shared_APL_component_oracle_limitation=True,whole_brain_universal_equivalence_claim=False,
|
||||
heldout_generalization_tested=False,formal_AB_tested=False,learning_achieved=False,candidate_generated=False,
|
||||
payload_bytes_before_manifest=budget(),
|
||||
next_gate='Independent seeds/novel-odor representation, then credit specificity; no learning promotion from CAL' if numerical and representation else
|
||||
'Audit first failure in this explicitly declared hypothesis; no gain search, no heldout tuning, no conditioning')
|
||||
save(EXP/'audit/final-status.json',result)
|
||||
lines=['# Phase 4AG: author LIF + unchanged legacy graded APL','',f'Status: `{status}`.','',
|
||||
f'Numerical closed-loop gate: {numerical}; original complete fixed six-CAL gate: {representation}.',
|
||||
'The independent Brian LIF and candidate use independent state but share the APL arithmetic component. Original frozen APL AST replay and a float64 error audit are separately recorded.',
|
||||
'Author LIF uses float64; APL retains float32, tau20ms/gain0.01/normalization1, full original matrices and order. No learning/parameter search.',
|
||||
'The explicitly hypothesized unconditional end-APL write includes refractory KC. Original fast APL mask leaves1,212 non-KC edges unreplaced; this scope limitation was not silently fixed.',
|
||||
f'Old checksums and five frozen reference trials: {oldpass}.',
|
||||
f'Successfully compared cases: {comparisons}.',
|
||||
'Detailed CAL, exact comparisons, component errors, event/request traces and execution exceptions are linked from audit/final-status.json.',
|
||||
'Wall/VRAM here include per-tick CPU synchronization, IPC and hashes. They are diagnostic costs, not runtime-performance benchmarks.',
|
||||
'No heldout dynamic outputs, formal A/B, learning, game, reward, decoder or new candidate.',
|
||||
'Next gate: '+result['next_gate'],'']
|
||||
with (EXP/'REPORT.md').open('x',encoding='utf-8') as f:f.write('\n'.join(lines))
|
||||
check();seal(MODEL);seal(EXP);print(status,flush=True)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,346 @@
|
||||
"""Diagnostic author-float64 LIF plus frozen legacy-float32 APL overlay.
|
||||
|
||||
The sealed Phase 4AE CUDA operators are reused without modification. APL is
|
||||
an explicitly mixed-precision engineering overlay, not an author-model claim.
|
||||
All CUDA/CPU synchronization here is diagnostic; no performance claim is made.
|
||||
"""
|
||||
from functools import lru_cache
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import time
|
||||
import traceback
|
||||
import types
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from legacy_apl_worker import LegacyAPL, N_NEURONS, file_sha256, array_sha256
|
||||
|
||||
|
||||
SOURCE_ROOT = Path(__file__).resolve().parent
|
||||
AE_MODEL = SOURCE_ROOT.parent / "author-runtime-lab-v1"
|
||||
AF_MODEL = SOURCE_ROOT.parent / "author-fullgraph-lab-v1"
|
||||
COEFFICIENT_NAMES = ("gd", "c0", "cg", "cv", "threshold", "reset", "seed_volt")
|
||||
REQUEST_FIELDS = ("tick", "neuron_idx", "attempted", "accepted", "rejected")
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def write_json(path, value):
|
||||
with Path(path).open("x", encoding="utf-8") as output:
|
||||
json.dump(value, output, indent=2, allow_nan=False)
|
||||
output.write("\n")
|
||||
|
||||
|
||||
def raw_hash(value, dtype):
|
||||
return hashlib.sha256(np.ascontiguousarray(value, dtype=dtype).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def load_source(path, name):
|
||||
module = types.ModuleType(name)
|
||||
module.__file__ = str(path)
|
||||
exec(compile(Path(path).read_bytes(), str(path), "exec"), module.__dict__)
|
||||
return module
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def sealed_backend():
|
||||
return load_source(AE_MODEL / "gpu_backend.py", "phase4ag_sealed_ae_backend")
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def queue_function():
|
||||
return load_source(AF_MODEL / "oracle_full.py", "phase4ag_sealed_af_queue").queue_summary
|
||||
|
||||
|
||||
@lru_cache(maxsize=4)
|
||||
def load_graph(graph_directory):
|
||||
root = Path(graph_directory)
|
||||
require(root.is_absolute(), "graph_dir must be absolute")
|
||||
graph = {}
|
||||
for name, dtype in (("nodes", np.int32), ("edge_id", np.int64), ("src", np.int32),
|
||||
("dst", np.int32), ("signed_raw", np.int64), ("ptr", np.int64),
|
||||
("weights", np.float64)):
|
||||
value = np.load(root / (name + ".npy"), mmap_mode="r", allow_pickle=False)
|
||||
require(value.dtype == dtype and value.ndim == 1, "Graph dtype/rank mismatch: " + name)
|
||||
graph[name] = value
|
||||
nodes, edges = graph["nodes"], graph["edge_id"]
|
||||
n, e = len(nodes), len(edges)
|
||||
require(0 < n <= N_NEURONS and bool(np.all((nodes >= 0) & (nodes < N_NEURONS))) and
|
||||
bool(np.all(np.diff(nodes) > 0)), "Global node IDs must be sorted unique")
|
||||
require(n < 512 or (n == N_NEURONS and np.array_equal(nodes, np.arange(N_NEURONS, dtype=np.int32))),
|
||||
"Only small fixtures or the intact frozen full graph are permitted")
|
||||
require(n != N_NEURONS or e == 25582938, "Full graph edge count changed")
|
||||
for name in ("src", "dst", "signed_raw", "weights"):
|
||||
require(graph[name].shape == (e,), "Graph edge array length mismatch: " + name)
|
||||
ptr = graph["ptr"]
|
||||
require(ptr.shape == (n + 1,) and ptr[0] == 0 and ptr[-1] == e and
|
||||
bool(np.all(np.diff(ptr) >= 0)), "Invalid graph CSR")
|
||||
require(np.array_equal(graph["src"], np.repeat(np.arange(n, dtype=np.int32), np.diff(ptr))),
|
||||
"Source order does not match CSR")
|
||||
# Global edge IDs need not equal local row indices in a fixture.
|
||||
for start in range(0, e, 1 << 20):
|
||||
stop = min(start + (1 << 20), e)
|
||||
require(bool(np.all((graph["dst"][start:stop] >= 0) & (graph["dst"][start:stop] < n))),
|
||||
"Local destination out of range")
|
||||
require(bool(np.isfinite(graph["weights"][start:stop]).all()), "Nonfinite base SI weights")
|
||||
require(np.array_equal(np.sign(graph["weights"][start:stop]), np.sign(graph["signed_raw"][start:stop])),
|
||||
"Base weight sign/zero pattern differs from frozen raw contacts")
|
||||
lo = max(0, start - 1)
|
||||
require(bool(np.all(edges[start:stop] >= 0)) and bool(np.all(np.diff(edges[lo:stop]) > 0)),
|
||||
"Global edge IDs must retain canonical increasing source/edge order")
|
||||
return graph
|
||||
|
||||
|
||||
def load_inputs(case_dir):
|
||||
spec = json.loads((case_dir / "spec.json").read_text(encoding="utf-8"))
|
||||
for name in ("name", "steps", "dt_ms", "duration_ms", "model", "graph_dir", "apl",
|
||||
"apl_bundle_dir", "coefficients_path", "record_history"):
|
||||
require(name in spec, "Missing case field: " + name)
|
||||
steps = spec["steps"]
|
||||
require(type(steps) is int and 0 < steps <= 1000 and float(spec["dt_ms"]) == .1 and
|
||||
float(spec["duration_ms"]) == steps * .1, "Unexpected steps/dt/duration")
|
||||
require(type(spec["record_history"]) is bool, "record_history must be Boolean")
|
||||
require(spec["apl"] == {"tau_ms": 20, "gain": .01, "normalization": 1}, "Frozen APL parameters changed")
|
||||
for name in ("graph_dir", "apl_bundle_dir", "coefficients_path"):
|
||||
require(Path(spec[name]).is_absolute(), "Input path must be absolute: " + name)
|
||||
model = spec["model"]
|
||||
for name in ("resting_potential_mv", "threshold_mv", "reset_potential_mv", "synaptic_delay_ms",
|
||||
"refractory_ms", "w_syn_mv_per_anatomical_synapse"):
|
||||
require(name in model and math.isfinite(float(model[name])), "Invalid model field: " + name)
|
||||
require(float(model["synaptic_delay_ms"]) == 1.8 and float(model["refractory_ms"]) == 2.2 and
|
||||
float(model["w_syn_mv_per_anatomical_synapse"]) == .275, "Frozen delay/ref/W_syn changed")
|
||||
coeff = json.loads(Path(spec["coefficients_path"]).read_text(encoding="utf-8"))
|
||||
for name in COEFFICIENT_NAMES:
|
||||
require(name in coeff and math.isfinite(float(coeff[name])), "Invalid author coefficient: " + name)
|
||||
require(coeff["threshold"] == float(model["threshold_mv"]) * .001 and
|
||||
coeff["reset"] == float(model["reset_potential_mv"]) * .001, "Coefficient threshold/reset mismatch")
|
||||
graph = load_graph(str(Path(spec["graph_dir"]).resolve()))
|
||||
n = len(graph["nodes"])
|
||||
require(spec["record_history"] == (n < 512), "Fixtures need dense history; full graph must not duplicate dense histories")
|
||||
with np.load(case_dir / "requests.npz", allow_pickle=False) as source:
|
||||
require(set(source.files) == {"tick", "neuron_idx"}, "Request sparse schema changed")
|
||||
ticks, requested = source["tick"].copy(), source["neuron_idx"].copy()
|
||||
require(ticks.dtype == np.int32 and requested.dtype == np.int32 and ticks.ndim == 1 and
|
||||
ticks.shape == requested.shape, "Invalid sparse request dtype/shape")
|
||||
require(bool(np.all((ticks >= 0) & (ticks < steps))), "Request tick outside trial")
|
||||
if len(ticks) > 1:
|
||||
require(bool(np.all((ticks[1:] > ticks[:-1]) |
|
||||
((ticks[1:] == ticks[:-1]) & (requested[1:] > requested[:-1])))),
|
||||
"Requests must be sorted unique by tick/global neuron")
|
||||
local = np.searchsorted(graph["nodes"], requested)
|
||||
require(bool(np.all(local < n)) and np.array_equal(graph["nodes"][local], requested),
|
||||
"Requested neuron absent from graph")
|
||||
initial = {"initial_v_mv": np.full(n, float(model["resting_potential_mv"]), dtype=np.float64),
|
||||
"initial_g_mv": np.zeros(n, dtype=np.float64)}
|
||||
initial_path = case_dir / "arrays.npz"
|
||||
if initial_path.exists():
|
||||
with np.load(initial_path, allow_pickle=False) as source:
|
||||
require(set(source.files).issubset(initial), "Unexpected initial-state array")
|
||||
for name in source.files:
|
||||
value = source[name]
|
||||
require(value.dtype == np.float64 and value.shape == (n,) and bool(np.isfinite(value).all()),
|
||||
"Invalid initial state: " + name)
|
||||
initial[name] = value.copy()
|
||||
return spec, graph, coeff, ticks, requested, local, initial
|
||||
|
||||
|
||||
def new_array(path, dtype, shape):
|
||||
require(not Path(path).exists(), "Refusing existing output: " + str(path))
|
||||
return np.lib.format.open_memmap(path, mode="w+", dtype=dtype, shape=shape)
|
||||
|
||||
|
||||
def run_gpu(case_dir, out_dir, build_root):
|
||||
"""Run fresh candidate state; never consume oracle/candidate APL traces.
|
||||
|
||||
pass_all denotes internal recording/finite integrity only. Independent
|
||||
comparison and the original complete representation gate are separate.
|
||||
"""
|
||||
case_dir, out_dir = Path(case_dir), Path(out_dir)
|
||||
require(not out_dir.exists(), "Refusing existing output directory: " + str(out_dir))
|
||||
out_dir.mkdir(parents=True, exist_ok=False)
|
||||
start, tick, completed = time.perf_counter(), -1, 0
|
||||
maps = {}
|
||||
try:
|
||||
spec, graph, coeff, request_ticks, request_nodes, request_local, initial = load_inputs(case_dir)
|
||||
require(torch.cuda.is_available() and torch.cuda.device_count() == 1 and
|
||||
"4090" in torch.cuda.get_device_name(), "Only the declared single visible RTX4090 is permitted")
|
||||
ext, build = sealed_backend().build_extension(build_root)
|
||||
build_done = time.perf_counter()
|
||||
apl = LegacyAPL(spec["apl_bundle_dir"], spec["dt_ms"])
|
||||
write_json(out_dir / "apl-component.json", apl.metadata)
|
||||
device = apl.device
|
||||
torch.cuda.reset_peak_memory_stats(device)
|
||||
n, e, steps = len(graph["nodes"]), len(graph["edge_id"]), spec["steps"]
|
||||
|
||||
def upload(value, dtype):
|
||||
return torch.from_numpy(np.array(value, copy=True, order="C")).to(device=device, dtype=dtype).contiguous()
|
||||
|
||||
indptr, dst = upload(graph["ptr"], torch.int64), upload(graph["dst"], torch.int32)
|
||||
global_nodes_gpu = upload(graph["nodes"], torch.int64)
|
||||
base_weights = upload(graph["weights"], torch.float64)
|
||||
weights = base_weights.clone()
|
||||
del base_weights
|
||||
local_apl = np.flatnonzero(np.isin(graph["nodes"], apl.apl_globals)).astype(np.int64)
|
||||
apl_out_mask = np.isin(graph["src"], local_apl)
|
||||
if bool(np.any(apl_out_mask)):
|
||||
weights[upload(apl_out_mask, torch.bool)] = 0.0
|
||||
effective_hash = hashlib.sha256()
|
||||
for offset in range(0, e, 1 << 20):
|
||||
effective_hash.update(weights[offset:offset + (1 << 20)].cpu().numpy().astype("<f8", copy=False).tobytes())
|
||||
common_kc, local_kc, full_kc = np.intersect1d(graph["nodes"], apl.kc_globals, assume_unique=True, return_indices=True)
|
||||
kc_local_gpu, kc_full_gpu = upload(local_kc, torch.int64), upload(full_kc, torch.int64)
|
||||
v = upload(initial["initial_v_mv"] * .001, torch.float64)
|
||||
g = upload(initial["initial_g_mv"] * .001, torch.float64)
|
||||
ref = torch.zeros(n, device=device, dtype=torch.int32)
|
||||
counts = torch.zeros(n, device=device, dtype=torch.int32)
|
||||
requests = torch.zeros(n, device=device, dtype=torch.uint8)
|
||||
ring = torch.zeros((19, n), device=device, dtype=torch.uint8)
|
||||
for name, width in (("drive", 2), ("state", 2), ("inhibition", 4064)):
|
||||
maps["apl_" + name] = new_array(out_dir / ("apl-" + name + ".float32.npy"), np.float32, (steps, width))
|
||||
if spec["record_history"]:
|
||||
for name, dtype in (("v", np.float64), ("g", np.float64), ("ref", np.int32)):
|
||||
maps[name] = new_array(out_dir / (name + "-history." + np.dtype(dtype).name + ".npy"), dtype, (steps, n))
|
||||
maps["pre_" + name] = new_array(out_dir / ("preAPL-" + name + "." + np.dtype(dtype).name + ".npy"), dtype, (steps, n))
|
||||
event_steps, event_nodes = [], []
|
||||
accepted_requests_total, due_total, accepted_due_total = 0, 0, 0
|
||||
queue_summary = queue_function()
|
||||
simulation_start = time.perf_counter()
|
||||
with (out_dir / "queue.jsonl").open("x", encoding="utf-8", buffering=1) as queue_output, \
|
||||
(out_dir / "tick-state-hashes.jsonl").open("x", encoding="utf-8", buffering=1) as hash_output, \
|
||||
(out_dir / "requests.csv").open("x", encoding="utf-8", newline="", buffering=1) as request_output:
|
||||
writer = csv.DictWriter(request_output, fieldnames=REQUEST_FIELDS)
|
||||
writer.writeheader()
|
||||
for tick in range(steps):
|
||||
requests.zero_()
|
||||
begin, end = np.searchsorted(request_ticks, (tick, tick + 1))
|
||||
if end > begin:
|
||||
requested_local_gpu = upload(request_local[begin:end], torch.int64)
|
||||
requests[requested_local_gpu] = 1
|
||||
accepted = (ref[requested_local_gpu] <= 1).to(torch.uint8).cpu().numpy()
|
||||
accepted_requests_total += int(accepted.sum())
|
||||
writer.writerows({"tick": tick, "neuron_idx": int(node), "attempted": 1,
|
||||
"accepted": int(ok), "rejected": 1 - int(ok)}
|
||||
for node, ok in zip(request_nodes[begin:end], accepted))
|
||||
spikes = ring[tick % 19]
|
||||
ext.state_step(v, g, ref, requests, spikes, counts,
|
||||
*[float(coeff[name]) for name in COEFFICIENT_NAMES], 22)
|
||||
expanded_global = np.empty(0, dtype=np.int64)
|
||||
accepted_cpu = np.empty(0, dtype=np.uint8)
|
||||
if tick >= 18:
|
||||
due = ring[(tick - 18) % 19]
|
||||
sources = torch.nonzero(due, as_tuple=False).flatten().contiguous()
|
||||
if sources.numel():
|
||||
degrees = indptr[sources + 1] - indptr[sources]
|
||||
offsets = (torch.cumsum(degrees, dim=0) - degrees).contiguous()
|
||||
edge_count = int(degrees.sum().item())
|
||||
expanded = torch.empty(edge_count, device=device, dtype=torch.int64)
|
||||
ext.fill_edges(indptr, sources, offsets, expanded)
|
||||
if edge_count:
|
||||
destination = dst[expanded]
|
||||
# Actual active fanout order, BEFORE stable target grouping.
|
||||
expanded_global = graph["edge_id"][expanded.cpu().numpy()].copy()
|
||||
accepted_cpu = (ref[destination.long()] == 0).to(torch.uint8).cpu().numpy().copy()
|
||||
order = torch.argsort(destination, stable=True)
|
||||
ordered_edges, ordered_targets = expanded[order].contiguous(), destination[order]
|
||||
boundary = torch.ones(edge_count, device=device, dtype=torch.bool)
|
||||
boundary[1:] = ordered_targets[1:] != ordered_targets[:-1]
|
||||
starts = torch.nonzero(boundary, as_tuple=False).flatten().contiguous()
|
||||
targets = ordered_targets[starts].contiguous()
|
||||
ext.apply_arrivals(ordered_edges, starts, targets, weights, g, ref)
|
||||
queue = queue_summary(tick, expanded_global, accepted_cpu)
|
||||
queue_output.write(json.dumps(queue, allow_nan=False) + "\n")
|
||||
due_total += queue["count"]
|
||||
accepted_due_total += queue["accepted_count"]
|
||||
if spec["record_history"]:
|
||||
for name, value in (("v", v), ("g", g), ("ref", ref)):
|
||||
maps["pre_" + name][tick] = value.cpu().numpy()
|
||||
if n == N_NEURONS:
|
||||
full_spikes = spikes
|
||||
else:
|
||||
apl.spikes.zero_()
|
||||
apl.spikes[global_nodes_gpu] = spikes
|
||||
full_spikes = apl.spikes
|
||||
drive, apl_state, h = apl.advance(full_spikes)
|
||||
# Mixed precision contract: widen f32 h EXACTLY, THEN multiply
|
||||
# in f64, THEN subtract in f64. Do not guard refractory KCs.
|
||||
delta_si = h[kc_full_gpu].to(torch.float64) * 0.001
|
||||
g[kc_local_gpu] = g[kc_local_gpu] - delta_si
|
||||
apl_cpu = {"drive": drive.cpu().numpy().copy(), "state": apl_state.cpu().numpy().copy(),
|
||||
"inhibition": h.cpu().numpy().copy()}
|
||||
for name, value in apl_cpu.items():
|
||||
maps["apl_" + name][tick] = value
|
||||
actual = {"v": v.cpu().numpy().copy(), "g": g.cpu().numpy().copy(),
|
||||
"ref": ref.cpu().numpy().copy()}
|
||||
finite = all(bool(np.isfinite(value).all()) for value in list(actual.values()) + list(apl_cpu.values()))
|
||||
row = {"tick": tick, **{name + "_sha256": raw_hash(value, "<i4" if name == "ref" else "<f8")
|
||||
for name, value in actual.items()}}
|
||||
hash_output.write(json.dumps(row, allow_nan=False) + "\n")
|
||||
if spec["record_history"]:
|
||||
for name, value in actual.items():
|
||||
maps[name][tick] = value
|
||||
fired_local = torch.nonzero(spikes, as_tuple=False).flatten().cpu().numpy()
|
||||
if len(fired_local):
|
||||
event_steps.append(np.full(len(fired_local), tick, dtype=np.int32))
|
||||
event_nodes.append(graph["nodes"][fired_local].copy())
|
||||
completed += 1
|
||||
if not finite:
|
||||
with (out_dir / "first-failure-state.npz").open("xb") as target:
|
||||
np.savez_compressed(target, tick=np.array(tick, dtype=np.int32), **actual,
|
||||
**{"apl_" + name: value for name, value in apl_cpu.items()})
|
||||
require(False, "Nonfinite state; first failure saved")
|
||||
torch.cuda.synchronize(device)
|
||||
|
||||
def joined(values):
|
||||
return np.concatenate(values).astype(np.int32, copy=False) if values else np.empty(0, dtype=np.int32)
|
||||
|
||||
state = {"step": joined(event_steps), "neuron_idx": joined(event_nodes), "counts": counts.cpu().numpy(),
|
||||
"v": v.cpu().numpy(), "g": g.cpu().numpy(), "ref": ref.cpu().numpy()}
|
||||
with (out_dir / "state.npz").open("xb") as target:
|
||||
np.savez_compressed(target, **state)
|
||||
event_locals = np.searchsorted(graph["nodes"], state["neuron_idx"])
|
||||
counts_exact = np.array_equal(np.bincount(event_locals, minlength=n), state["counts"])
|
||||
require(counts_exact and completed == steps, "Sparse events/counts/completed ticks mismatch")
|
||||
for value in maps.values():
|
||||
value.flush()
|
||||
summary = {"case": spec["name"], "complete": True, "pass_all": True,
|
||||
"pass_all_scope": "internal finite/recording/event-count integrity only; independent equivalence not claimed",
|
||||
"steps": steps, "completed_ticks": completed, "nodes": n, "edges": e, "precision": "float64",
|
||||
"apl_precision": "float32", "state_units": "SI volts", "total_spikes": len(state["step"]),
|
||||
"requested": len(request_nodes), "accepted_requests": accepted_requests_total,
|
||||
"rejected_requests": len(request_nodes) - accepted_requests_total,
|
||||
"arriving_edges": due_total, "accepted_arriving_edges": accepted_due_total,
|
||||
"counts_match_sparse_events": bool(counts_exact), "all_finite": True,
|
||||
"record_history": spec["record_history"], "apl_fast_edges_zeroed": int(apl_out_mask.sum()),
|
||||
"kc_targets_present": len(common_kc), "effective_weights_raw_sha256": effective_hash.hexdigest(),
|
||||
"apl_component": apl.metadata, "apl_schedule": "end, after threshold/reset and guarded arrivals, unconditional all KC",
|
||||
"apl_si_conversion": "h32 -> float64 -> multiply float64 0.001 -> separate float64 subtraction",
|
||||
"build": build, "sealed_ae_backend_sha256": file_sha256(AE_MODEL / "gpu_backend.py"),
|
||||
"shared_apl_component_oracle_limitation": True, "oracle_validation_claim": False,
|
||||
"learning": False, "performance_runtime_claim": False, "cpu_sync_each_tick": True,
|
||||
"build_or_cache_seconds": build_done - start,
|
||||
"diagnostic_simulation_seconds": time.perf_counter() - simulation_start,
|
||||
"wall_seconds": time.perf_counter() - start,
|
||||
"peak_allocated_vram_bytes": int(torch.cuda.max_memory_allocated(device)),
|
||||
"peak_reserved_vram_bytes": int(torch.cuda.max_memory_reserved(device)),
|
||||
"input_sha256": {"spec.json": file_sha256(case_dir / "spec.json"),
|
||||
"requests.npz": file_sha256(case_dir / "requests.npz"),
|
||||
"base_weights.npy": file_sha256(Path(spec["graph_dir"]) / "weights.npy"),
|
||||
"coefficients.json": file_sha256(spec["coefficients_path"])}}
|
||||
if (case_dir / "arrays.npz").exists():
|
||||
summary["input_sha256"]["arrays.npz"] = file_sha256(case_dir / "arrays.npz")
|
||||
write_json(out_dir / "summary.json", summary)
|
||||
return summary
|
||||
except Exception as error:
|
||||
for value in maps.values():
|
||||
value.flush()
|
||||
write_json(out_dir / "error.json", {"complete": False, "pass_all": False,
|
||||
"tick_in_progress": tick, "completed_ticks": completed,
|
||||
"type": type(error).__name__, "message": str(error), "traceback": traceback.format_exc()})
|
||||
raise
|
||||
@@ -0,0 +1,175 @@
|
||||
"""Frozen eager Torch float32 APL component and binary oracle sidecar.
|
||||
|
||||
This is the Phase 4P operator sequence, not an author-neuron implementation.
|
||||
No precision/determinism flags are changed here. The full matrix dimensions
|
||||
are retained even when the caller simulates a small neuronal fixture.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
|
||||
PROTOCOL = "phase4ag-apl-v1"
|
||||
N_NEURONS, N_APL, N_KC = 166700, 2, 4064
|
||||
BUNDLE_FILES = ("input_matrix.npy", "output_matrix.npy", "kc_globals.npy", "apl_globals.npy")
|
||||
FRAME = struct.Struct("<qQ")
|
||||
PAYLOAD_BYTES = (N_APL + N_APL + N_KC) * 4
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def file_sha256(path):
|
||||
digest = hashlib.sha256()
|
||||
with Path(path).open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(8 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def array_sha256(value):
|
||||
return hashlib.sha256(np.ascontiguousarray(value).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def environment_metadata():
|
||||
device = torch.cuda.current_device()
|
||||
return {"torch": str(torch.__version__), "numpy": str(np.__version__),
|
||||
"cuda": torch.version.cuda, "cudnn": torch.backends.cudnn.version(),
|
||||
"device_name": torch.cuda.get_device_name(device), "device_index": device,
|
||||
"device_capability": list(torch.cuda.get_device_capability(device)),
|
||||
"deterministic_algorithms": torch.are_deterministic_algorithms_enabled(),
|
||||
"cudnn_deterministic": torch.backends.cudnn.deterministic,
|
||||
"cudnn_benchmark": torch.backends.cudnn.benchmark,
|
||||
"cuda_matmul_allow_tf32": torch.backends.cuda.matmul.allow_tf32,
|
||||
"cudnn_allow_tf32": torch.backends.cudnn.allow_tf32,
|
||||
"float32_matmul_precision": torch.get_float32_matmul_precision(),
|
||||
"CUBLAS_WORKSPACE_CONFIG": os.environ.get("CUBLAS_WORKSPACE_CONFIG"),
|
||||
"precision_configuration_changed_by_component": False}
|
||||
|
||||
|
||||
class LegacyAPL:
|
||||
"""Independent zero-initialized APL state with original full-width matrices."""
|
||||
|
||||
def __init__(self, bundle_dir, dt_ms):
|
||||
require(float(dt_ms) == 0.1, "Only frozen dt_ms=0.1 is allowed")
|
||||
require(torch.cuda.is_available(), "CUDA is required for the legacy Torch component")
|
||||
self.bundle_dir = Path(bundle_dir)
|
||||
expected = {"input_matrix.npy": (np.float32, (N_APL, N_NEURONS)),
|
||||
"output_matrix.npy": (np.float32, (N_APL, N_KC)),
|
||||
"kc_globals.npy": (np.int32, (N_KC,)),
|
||||
"apl_globals.npy": (np.int32, (N_APL,))}
|
||||
arrays = {}
|
||||
for name, (dtype, shape) in expected.items():
|
||||
value = np.load(self.bundle_dir / name, allow_pickle=False)
|
||||
require(value.dtype == dtype and value.shape == shape and value.flags.c_contiguous,
|
||||
"APL bundle dtype/shape/layout mismatch: " + name)
|
||||
require(bool(np.isfinite(value).all()), "Nonfinite APL bundle: " + name)
|
||||
arrays[name] = value
|
||||
for name in ("kc_globals.npy", "apl_globals.npy"):
|
||||
ids = arrays[name]
|
||||
require(bool(np.all((ids >= 0) & (ids < N_NEURONS))) and
|
||||
bool(np.all(np.diff(ids) > 0)), "APL/KC globals must be sorted unique: " + name)
|
||||
require(not np.intersect1d(arrays["kc_globals.npy"], arrays["apl_globals.npy"]).size,
|
||||
"APL and KC identities overlap")
|
||||
require(bool(np.all(arrays["input_matrix.npy"] >= 0)) and
|
||||
bool(np.all(arrays["output_matrix.npy"] >= 0)), "APL raw-contact matrices must be nonnegative")
|
||||
self.device = torch.device("cuda", torch.cuda.current_device())
|
||||
self.input_gpu = torch.from_numpy(arrays["input_matrix.npy"].copy()).to(self.device)
|
||||
self.output_gpu = torch.from_numpy(arrays["output_matrix.npy"].copy()).to(self.device)
|
||||
self.kc_globals = arrays["kc_globals.npy"].copy()
|
||||
self.apl_globals = arrays["apl_globals.npy"].copy()
|
||||
self.state = torch.zeros(N_APL, dtype=torch.float32, device=self.device)
|
||||
self.spikes = torch.zeros(N_NEURONS, dtype=torch.uint8, device=self.device)
|
||||
self.decay = np.float32(math.exp(-float(dt_ms) / 20.0))
|
||||
self.gain = np.float32(0.01)
|
||||
self.metadata = {"ok": True, "protocol": PROTOCOL, "n_neurons": N_NEURONS,
|
||||
"n_apl": N_APL, "n_kc": N_KC, "dt_ms": float(dt_ms),
|
||||
"files_sha256": {name: file_sha256(self.bundle_dir / name) for name in BUNDLE_FILES},
|
||||
"array_sha256": {name: array_sha256(value) for name, value in arrays.items()},
|
||||
"decay_f32_bits": f"{int(self.decay.view(np.uint32)):08x}",
|
||||
"gain_f32_bits": f"{int(self.gain.view(np.uint32)):08x}",
|
||||
"environment": environment_metadata(),
|
||||
"operator_sequence": "state.mul_(decay32); drive=mv(I32,spikes32); state.add_(drive); h=matmul(state,O32)*gain32",
|
||||
"h_units": "legacy mV", "initial_state": "float32 zeros"}
|
||||
|
||||
@torch.no_grad()
|
||||
def advance(self, spikes_u8):
|
||||
require(spikes_u8.device == self.device and spikes_u8.dtype == torch.uint8 and
|
||||
tuple(spikes_u8.shape) == (N_NEURONS,) and spikes_u8.is_contiguous(),
|
||||
"APL expects full-width contiguous CUDA uint8 spikes")
|
||||
# Preserve separate eager operations, including Python evaluation order
|
||||
# in the frozen state.mul_(...).add_(torch.mv(...)) expression.
|
||||
self.state.mul_(float(self.decay))
|
||||
drive = torch.mv(self.input_gpu, spikes_u8.to(torch.float32))
|
||||
self.state.add_(drive)
|
||||
h = torch.matmul(self.state, self.output_gpu) * self.gain
|
||||
return drive, self.state, h
|
||||
|
||||
def advance_global_fired(self, fired):
|
||||
require(fired.dtype == np.int32 and fired.ndim == 1, "Global spikes must be int32 vector")
|
||||
require(bool(np.all((fired >= 0) & (fired < N_NEURONS))) and
|
||||
bool(np.all(np.diff(fired) > 0)), "Global spikes must be sorted unique and in range")
|
||||
self.spikes.zero_()
|
||||
if fired.size:
|
||||
indices = torch.from_numpy(fired.copy()).to(device=self.device, dtype=torch.int64)
|
||||
self.spikes[indices] = 1
|
||||
return self.advance(self.spikes)
|
||||
|
||||
|
||||
def read_exact(stream, count, boundary_eof=False):
|
||||
result = bytearray()
|
||||
while len(result) < count:
|
||||
block = stream.read(count - len(result))
|
||||
if not block:
|
||||
if boundary_eof and not result:
|
||||
return None
|
||||
raise EOFError("Truncated APL request frame")
|
||||
result.extend(block)
|
||||
return bytes(result)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--bundle-dir", required=True)
|
||||
parser.add_argument("--dt-ms", required=True, type=float)
|
||||
args = parser.parse_args()
|
||||
# stdout is exclusively the handshake line and binary response frames.
|
||||
output, source = sys.stdout.buffer, sys.stdin.buffer
|
||||
component = LegacyAPL(args.bundle_dir, args.dt_ms)
|
||||
output.write((json.dumps(component.metadata, allow_nan=False) + "\n").encode("utf-8"))
|
||||
output.flush()
|
||||
expected_tick = 0
|
||||
while True:
|
||||
header = read_exact(source, FRAME.size, boundary_eof=True)
|
||||
if header is None:
|
||||
return
|
||||
tick, count = FRAME.unpack(header)
|
||||
require(tick == expected_tick and count <= N_NEURONS, "Invalid APL request tick/count")
|
||||
fired = np.frombuffer(read_exact(source, count * 4), dtype="<i4").copy()
|
||||
drive, state, h = component.advance_global_fired(fired)
|
||||
arrays = [value.detach().cpu().numpy().astype("<f4", copy=False) for value in (drive, state, h)]
|
||||
require(all(bool(np.isfinite(value).all()) for value in arrays), "Nonfinite APL output")
|
||||
output.write(FRAME.pack(tick, PAYLOAD_BYTES))
|
||||
for value in arrays:
|
||||
output.write(value.tobytes())
|
||||
output.flush()
|
||||
expected_tick += 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception:
|
||||
traceback.print_exc(file=sys.stderr)
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,85 @@
|
||||
{
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/author-apl-lab-v1",
|
||||
"files": [
|
||||
{
|
||||
"path": "PROTOCOL.md",
|
||||
"bytes": 5033,
|
||||
"sha256": "c14c2d3c35e2b404c516befa66370723fba075516540214ae46e4104ff87e813"
|
||||
},
|
||||
{
|
||||
"path": "calibration.py",
|
||||
"bytes": 25414,
|
||||
"sha256": "edb9cb2d890471fd07afd88c959ef3540b0cd40af490fac3f26396b5e7f5bf9c"
|
||||
},
|
||||
{
|
||||
"path": "compare.py",
|
||||
"bytes": 33632,
|
||||
"sha256": "6e1616b086f667fa4e4b5087f500f8080281977ca9922e7b055acfddd2003bd5"
|
||||
},
|
||||
{
|
||||
"path": "config.json",
|
||||
"bytes": 1845,
|
||||
"sha256": "2aa6b4d777ff698018e1a6eaf571215208b8c874a4c848a6d632e2bd778c90e3"
|
||||
},
|
||||
{
|
||||
"path": "context.py",
|
||||
"bytes": 2038,
|
||||
"sha256": "b8b1231d958a3ec935ec1d5a36e08cbd9bcd1127f4d4bb243a9051884a0e7a16"
|
||||
},
|
||||
{
|
||||
"path": "finalize.py",
|
||||
"bytes": 6747,
|
||||
"sha256": "b6e3c67e668a971b544df32d76435ab5e537061b9fc2e8d9dedb5d2a726f5782"
|
||||
},
|
||||
{
|
||||
"path": "gpu_apl.py",
|
||||
"bytes": 21206,
|
||||
"sha256": "69f5d893f47dafff4ef7c3186c3df0b9014666b32c8f4e943b57016bec92b18a"
|
||||
},
|
||||
{
|
||||
"path": "legacy_apl_worker.py",
|
||||
"bytes": 8158,
|
||||
"sha256": "21ceec65fe42581dc55872010e0025d660e80358905f1cf4df1dadf341a3b291"
|
||||
},
|
||||
{
|
||||
"path": "oracle_apl.py",
|
||||
"bytes": 46017,
|
||||
"sha256": "9b7f3e0b59f2117b94b41a6f542f37e6f7950138d324c905e8e0ac6cb7d17151"
|
||||
},
|
||||
{
|
||||
"path": "prepare.py",
|
||||
"bytes": 13633,
|
||||
"sha256": "197ccc6aae70566c4db4d4de17fa032e853b9bd9a04262905d6c71e12a930b6f"
|
||||
},
|
||||
{
|
||||
"path": "run_case.py",
|
||||
"bytes": 2116,
|
||||
"sha256": "afbe6d5a37c29bda66d38c470093584130d1852e67881c681662aec3c76b33c7"
|
||||
},
|
||||
{
|
||||
"path": "run_suite.py",
|
||||
"bytes": 3700,
|
||||
"sha256": "84c681030b0032119b5392d0c3f3c136a9c6b0d133eb548c40791a10bbbb8b63"
|
||||
},
|
||||
{
|
||||
"path": "test_calibration.py",
|
||||
"bytes": 8127,
|
||||
"sha256": "bab3136eecbc5bb09a985d715844ee845cb72b07930ce376bfb39be33bd1845a"
|
||||
},
|
||||
{
|
||||
"path": "test_protocol.py",
|
||||
"bytes": 11764,
|
||||
"sha256": "1a233f47943e01bbd845ac1dacdb1eec3f88c270c65a5f75bce1cf679b5de097"
|
||||
},
|
||||
{
|
||||
"path": "validate_components.py",
|
||||
"bytes": 24824,
|
||||
"sha256": "5cfc9a4505da6b8fb4e7d2617cdaedaa8475ed46a11ec9bdf0d4d25b1d22c2d3"
|
||||
},
|
||||
{
|
||||
"path": "verify_frozen.py",
|
||||
"bytes": 2160,
|
||||
"sha256": "0d0e72b2f1348f6648124c8c34f75094f76fc5e1a281fbcca130a5fad2f35293"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,740 @@
|
||||
"""Phase 4AG shared-APL-component closed-loop Brian oracle.
|
||||
|
||||
Brian has no Torch dependency. A separate, pinned science-Python process
|
||||
executes the unchanged legacy float32 APL operations from Brian's own spikes.
|
||||
This is independent LIF/scheduling validation with a shared APL numerical
|
||||
component, NOT an all-components-independent CPU oracle.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import queue
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import types
|
||||
from contextlib import ExitStack
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
BLOCK_BYTES = 8 << 20
|
||||
PROTOCOL = "phase4ag-apl-v1"
|
||||
AUTHOR_PATH = Path("/data/digital-drosophila/experiments/phase4ad-author-semantics/external-resources/model.py")
|
||||
AUTHOR_SHA256 = "fc45837d7122c6ce2a7f3f2f23c515992e4b232aadb919efabb72337fac88e4e"
|
||||
AE_PATH = Path("/data/digital-drosophila/models/male-cns-lif/author-runtime-lab-v1/oracle_stress.py")
|
||||
AE_SHA256 = "32eb1c42383068ce52701d2139a376ce155ad7072cde019949f8ca2d55492409"
|
||||
AF_PATH = Path("/data/digital-drosophila/models/male-cns-lif/author-fullgraph-lab-v1/oracle_full.py")
|
||||
AF_SHA256 = "d72003d98f0f1624b12126eff50bd361c43d888f4a64d90ba9247a58976f1961"
|
||||
EVENT_DTYPE = np.dtype([("step", "<i4"), ("neuron_idx", "<i4")])
|
||||
BUNDLE_FILES = ("input_matrix.npy", "output_matrix.npy", "kc_globals.npy", "apl_globals.npy")
|
||||
_PROCESS_USED = False
|
||||
|
||||
|
||||
def _check(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def _save(path, value):
|
||||
with Path(path).open("x", encoding="utf-8") as handle:
|
||||
json.dump(value, handle, indent=2, sort_keys=True, allow_nan=False)
|
||||
handle.write("\n")
|
||||
|
||||
|
||||
def _digest(path):
|
||||
result = hashlib.sha256()
|
||||
with Path(path).open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(BLOCK_BYTES), b""):
|
||||
result.update(block)
|
||||
return result.hexdigest()
|
||||
|
||||
|
||||
def _array_hash(array, dtype):
|
||||
return hashlib.sha256(np.ascontiguousarray(array, dtype=dtype).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def _load_checked(path, expected, name):
|
||||
data = Path(path).read_bytes()
|
||||
_check(hashlib.sha256(data).hexdigest() == expected, f"Pinned source hash mismatch: {path}")
|
||||
module = types.ModuleType(name)
|
||||
module.__file__ = str(path)
|
||||
exec(compile(data, str(path), "exec"), module.__dict__)
|
||||
return module
|
||||
|
||||
|
||||
def _new_memmap(path, dtype, shape):
|
||||
_check(not Path(path).exists(), f"Refusing to overwrite output: {path}")
|
||||
return np.lib.format.open_memmap(path, mode="w+", dtype=dtype, shape=shape)
|
||||
|
||||
|
||||
def _budget(root, limit, reserve=0):
|
||||
total = 0
|
||||
for directory, _, filenames in os.walk(root, followlinks=False):
|
||||
for filename in filenames:
|
||||
total += (Path(directory) / filename).stat(follow_symlinks=False).st_size
|
||||
_check(total + int(reserve) <= limit,
|
||||
f"Output budget exceeded: {total}+{reserve}>{limit}; inputs and duration were not adjusted")
|
||||
return total
|
||||
|
||||
|
||||
class _PipeReader:
|
||||
"""One bounded reader thread, with timeout/EOF checks for every frame."""
|
||||
|
||||
def __init__(self, pipe, timeout):
|
||||
self.pipe, self.timeout = pipe, float(timeout)
|
||||
self.chunks = queue.Queue(maxsize=8)
|
||||
self.buffer = bytearray()
|
||||
self.thread = threading.Thread(target=self._pump, daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
def _pump(self):
|
||||
try:
|
||||
while True:
|
||||
chunk = os.read(self.pipe.fileno(), 65536)
|
||||
self.chunks.put(chunk)
|
||||
if not chunk:
|
||||
return
|
||||
except BaseException as error:
|
||||
self.chunks.put(error)
|
||||
|
||||
def _fill(self, deadline):
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise TimeoutError("APL sidecar response timed out")
|
||||
try:
|
||||
chunk = self.chunks.get(timeout=remaining)
|
||||
except queue.Empty as error:
|
||||
raise TimeoutError("APL sidecar response timed out") from error
|
||||
if isinstance(chunk, BaseException):
|
||||
raise RuntimeError("APL sidecar stdout reader failed") from chunk
|
||||
if not chunk:
|
||||
raise EOFError("APL sidecar ended before a complete response")
|
||||
self.buffer.extend(chunk)
|
||||
|
||||
def read(self, size):
|
||||
deadline = time.monotonic() + self.timeout
|
||||
while len(self.buffer) < size:
|
||||
self._fill(deadline)
|
||||
result = bytes(self.buffer[:size])
|
||||
del self.buffer[:size]
|
||||
return result
|
||||
|
||||
def line(self):
|
||||
deadline = time.monotonic() + self.timeout
|
||||
while b"\n" not in self.buffer:
|
||||
_check(len(self.buffer) < 1 << 20, "APL sidecar handshake exceeds 1 MiB")
|
||||
self._fill(deadline)
|
||||
end = self.buffer.index(b"\n") + 1
|
||||
result = bytes(self.buffer[:end])
|
||||
del self.buffer[:end]
|
||||
return result
|
||||
|
||||
|
||||
class _APLWorker:
|
||||
def __init__(self, python, script, bundle_dir, stderr, timeout=180):
|
||||
self.command = [str(python), "-B", str(script), "--bundle-dir", str(bundle_dir), "--dt-ms", "0.1"]
|
||||
self.process = subprocess.Popen(self.command, stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE, stderr=stderr, bufsize=0, shell=False)
|
||||
self.reader = _PipeReader(self.process.stdout, timeout)
|
||||
self.tick = 0
|
||||
try:
|
||||
self.hello = json.loads(self.reader.line().decode("utf-8"))
|
||||
_check(self.hello.get("ok") is True, "APL sidecar did not acknowledge initialization")
|
||||
_check(self.hello.get("protocol") == PROTOCOL, "Unexpected APL sidecar protocol")
|
||||
for key, value in (("n_neurons", 166700), ("n_apl", 2), ("n_kc", 4064)):
|
||||
_check(self.hello.get(key) == value, f"APL sidecar changed {key}")
|
||||
_check(self.hello.get("dt_ms") == .1, "APL sidecar changed dt")
|
||||
expected_decay = f"{int(np.float32(math.exp(-.1 / 20.)).view(np.uint32)):08x}"
|
||||
expected_gain = f"{int(np.float32(.01).view(np.uint32)):08x}"
|
||||
_check(self.hello.get("decay_f32_bits") == expected_decay and self.hello.get("gain_f32_bits") == expected_gain,
|
||||
"APL sidecar decay/gain differs from the frozen float32 scalars")
|
||||
_check(self.hello.get("files_sha256") == {name: _digest(bundle_dir / name) for name in BUNDLE_FILES},
|
||||
"APL sidecar bundle hashes differ from the independently read bundle")
|
||||
except BaseException:
|
||||
self.abort()
|
||||
raise
|
||||
|
||||
def step(self, tick, global_spikes):
|
||||
_check(tick == self.tick, "APL sidecar request ticks must be consecutive")
|
||||
nodes = np.asarray(global_spikes)
|
||||
_check(nodes.ndim == 1 and nodes.dtype == np.dtype("int32"), "APL input spikes must be int32 globals")
|
||||
_check(not len(nodes) or (nodes[0] >= 0 and nodes[-1] < 166700 and np.all(np.diff(nodes) > 0)),
|
||||
"APL input spikes must be sorted, unique, and in the frozen global range")
|
||||
packet = struct.pack("<qQ", int(tick), len(nodes)) + np.ascontiguousarray(nodes, dtype="<i4").tobytes()
|
||||
view = memoryview(packet)
|
||||
while view:
|
||||
written = self.process.stdin.write(view)
|
||||
if not written:
|
||||
raise BrokenPipeError("APL sidecar stdin failed")
|
||||
view = view[written:]
|
||||
self.process.stdin.flush()
|
||||
response_tick, byte_count = struct.unpack("<qQ", self.reader.read(16))
|
||||
_check(response_tick == tick and byte_count == (2 + 2 + 4064) * 4,
|
||||
"APL sidecar frame tick/length mismatch")
|
||||
values = np.frombuffer(self.reader.read(byte_count), dtype="<f4")
|
||||
_check(np.isfinite(values).all() and np.all(values >= 0), "APL sidecar returned invalid nonnegative f32 states")
|
||||
self.tick += 1
|
||||
return values[:2], values[2:4], values[4:]
|
||||
|
||||
def finish(self, steps):
|
||||
_check(self.tick == steps, "APL sidecar did not process every tick")
|
||||
self.process.stdin.close()
|
||||
status = self.process.wait(timeout=30)
|
||||
_check(status == 0, f"APL sidecar exited unsuccessfully: {status}")
|
||||
self.reader.thread.join(timeout=5)
|
||||
_check(not self.reader.thread.is_alive(), "APL stdout did not reach EOF")
|
||||
extra = bytes(self.reader.buffer)
|
||||
while not self.reader.chunks.empty():
|
||||
chunk = self.reader.chunks.get_nowait()
|
||||
_check(not isinstance(chunk, BaseException), "APL stdout reader failed after the last frame")
|
||||
extra += chunk
|
||||
_check(not extra, "APL sidecar emitted unrequested trailing stdout bytes")
|
||||
self.process.stdout.close()
|
||||
return status
|
||||
|
||||
def abort(self):
|
||||
if self.process.poll() is None:
|
||||
self.process.terminate()
|
||||
try:
|
||||
self.process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.process.kill()
|
||||
self.process.wait(timeout=5)
|
||||
for handle in (self.process.stdin, self.process.stdout):
|
||||
if handle is not None and not handle.closed:
|
||||
handle.close()
|
||||
|
||||
|
||||
def _graph_inputs(directory):
|
||||
arrays = {}
|
||||
definitions = {"nodes": "int32", "edge_id": "int64", "src": "int32",
|
||||
"dst": "int32", "signed_raw": "int64", "ptr": "int64", "weights": "float64"}
|
||||
for name, dtype in definitions.items():
|
||||
value = np.load(directory / (name + ".npy"), mmap_mode="r", allow_pickle=False)
|
||||
_check(value.ndim == 1 and value.dtype == np.dtype(dtype), f"Invalid graph array {name}")
|
||||
arrays[name] = value
|
||||
nodes, edge_ids = arrays["nodes"], arrays["edge_id"]
|
||||
n, edges = len(nodes), len(edge_ids)
|
||||
_check(0 < n <= 166700 and edges > 0, "A case needs real global nodes and explicit nonempty connectivity")
|
||||
_check(nodes[0] >= 0 and nodes[-1] < 166700 and np.all(np.diff(nodes) > 0), "Invalid global node order")
|
||||
for name in ("src", "dst", "signed_raw", "weights"):
|
||||
_check(arrays[name].shape == (edges,), f"Graph array {name} has the wrong shape")
|
||||
pointer = arrays["ptr"]
|
||||
_check(pointer.shape == (n + 1,) and pointer[0] == 0 and pointer[-1] == edges and
|
||||
np.all(np.diff(pointer) >= 0), "Invalid graph CSR pointer")
|
||||
source_counts = np.zeros(n, np.int64)
|
||||
identity_edges = True
|
||||
previous_source, previous_edge = -1, -1
|
||||
width = BLOCK_BYTES // 8
|
||||
for begin in range(0, edges, width):
|
||||
stop = min(edges, begin + width)
|
||||
source, target, ids = arrays["src"][begin:stop], arrays["dst"][begin:stop], edge_ids[begin:stop]
|
||||
_check(np.all((source >= 0) & (source < n)) and np.all((target >= 0) & (target < n)),
|
||||
"Graph edge endpoint is outside the local node array")
|
||||
_check(np.all(ids >= 0), "Graph edge IDs must be nonnegative")
|
||||
_check(source[0] > previous_source or (source[0] == previous_source and ids[0] > previous_edge),
|
||||
"Graph edge blocks are not in canonical source/edge order")
|
||||
_check(np.all((source[1:] > source[:-1]) |
|
||||
((source[1:] == source[:-1]) & (ids[1:] > ids[:-1]))),
|
||||
"Graph edges are not in canonical source/edge order")
|
||||
previous_source, previous_edge = int(source[-1]), int(ids[-1])
|
||||
identity_edges &= np.array_equal(ids, np.arange(begin, stop, dtype=np.int64))
|
||||
source_counts += np.bincount(source, minlength=n)
|
||||
_check(np.isfinite(arrays["weights"][begin:stop]).all(), "Nonfinite graph weights")
|
||||
_check(np.array_equal(source_counts, np.diff(pointer)), "CSR pointer and source rows disagree")
|
||||
if not identity_edges:
|
||||
_check(len(np.unique(edge_ids)) == edges, "Original edge IDs must be unique")
|
||||
for name in ("completeness.csv", "connectivity.parquet"):
|
||||
_check((directory / name).is_file(), f"Missing unchanged author input: {name}")
|
||||
if n == 166700:
|
||||
_check(edges == 25582938 and identity_edges and np.array_equal(nodes, np.arange(n, dtype=np.int32)),
|
||||
"A full probe must use the unchanged complete canonical graph")
|
||||
return arrays, identity_edges
|
||||
|
||||
|
||||
def _requests(case_dir, nodes, steps):
|
||||
with np.load(case_dir / "requests.npz", allow_pickle=False) as archive:
|
||||
_check(set(archive.files) == {"tick", "neuron_idx"}, "Unexpected sparse request schema")
|
||||
ticks, requested = archive["tick"].copy(), archive["neuron_idx"].copy()
|
||||
_check(ticks.dtype == requested.dtype == np.dtype("int32") and ticks.ndim == requested.ndim == 1 and
|
||||
ticks.shape == requested.shape, "Requests must be equal-length int32 vectors")
|
||||
_check(np.all((ticks >= 0) & (ticks < steps)), "Request time outside the case")
|
||||
_check(not len(ticks) or np.all((ticks[1:] > ticks[:-1]) |
|
||||
((ticks[1:] == ticks[:-1]) & (requested[1:] > requested[:-1]))),
|
||||
"Requests must be sorted and unique by tick/global node")
|
||||
local = np.searchsorted(nodes, requested)
|
||||
_check(np.all(local < len(nodes)) and np.array_equal(nodes[local], requested),
|
||||
"Request references a global neuron absent from this case")
|
||||
counts = np.bincount(ticks, minlength=steps).astype(np.int32)
|
||||
pointer = np.concatenate((np.zeros(1, np.int64), np.cumsum(counts, dtype=np.int64)))
|
||||
return requested, local.astype(np.int32), pointer, counts
|
||||
|
||||
|
||||
class _QueueProxy:
|
||||
def __init__(self, original, get_tick, available, arrays, identity_edges,
|
||||
due_counts, spike_ring, delay_steps, handle, records, summarize):
|
||||
self._original_queue, self._get_tick = original, get_tick
|
||||
self._available, self._arrays, self._identity_edges = available, arrays, identity_edges
|
||||
self._due_counts, self._spike_ring, self._delay_steps = due_counts, spike_ring, delay_steps
|
||||
self._handle, self._records, self._summarize = handle, records, summarize
|
||||
|
||||
def peek(self, *args, **kwargs):
|
||||
original = self._original_queue.peek(*args, **kwargs)
|
||||
local_edges = np.asarray(original)
|
||||
tick = self._get_tick()
|
||||
_check(tick == len(self._records["rows"]), "Queue peek ticks must be consecutive")
|
||||
_check(local_edges.ndim == 1 and np.all((local_edges >= 0) & (local_edges < len(self._arrays["edge_id"]))),
|
||||
"Original queue contains an invalid local edge index")
|
||||
_check(len(local_edges) == self._due_counts[tick % len(self._due_counts)],
|
||||
"Queue arrivals differ from the fixed-delay outgoing degree of actual spikes")
|
||||
fired_then = np.zeros(len(self._arrays["nodes"]), dtype=bool)
|
||||
if tick >= self._delay_steps:
|
||||
fired_then[self._spike_ring[(tick - self._delay_steps) % len(self._spike_ring)]] = True
|
||||
|
||||
def accepted(begin, stop):
|
||||
part = local_edges[begin:stop]
|
||||
_check(np.all(fired_then[self._arrays["src"][part]]), "Arrival lacks a presynaptic spike at author delay")
|
||||
return self._available[self._arrays["dst"][part]]
|
||||
|
||||
# Only the observed ID view changes; the queue is returned verbatim.
|
||||
edge_ids = local_edges if self._identity_edges else self._arrays["edge_id"][local_edges]
|
||||
summary = self._summarize(tick, edge_ids, accepted)
|
||||
self._handle.write(json.dumps(summary, sort_keys=True) + "\n")
|
||||
self._records["rows"].append(summary)
|
||||
self._records["peek_calls"] += 1
|
||||
return original
|
||||
|
||||
def advance(self, *args, **kwargs):
|
||||
self._records["advance_calls"] += 1
|
||||
return self._original_queue.advance(*args, **kwargs)
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self._original_queue, name)
|
||||
|
||||
|
||||
def run_case(case_dir: Path, out_dir: Path) -> dict:
|
||||
"""Run one continuous case, with exclusive outputs and no candidate trace input.
|
||||
|
||||
Graph inputs are the AF array/author-table schema plus original weights.npy.
|
||||
requests.npz contains sparse tick/neuron_idx int32 vectors. Optional
|
||||
arrays.npz contains initial_v_mv/initial_g_mv float64 vectors only.
|
||||
Full probes write state hashes, sparse events and APL traces, not dense
|
||||
whole-brain histories. Each case requires a fresh process.
|
||||
"""
|
||||
global _PROCESS_USED
|
||||
_check(not _PROCESS_USED, "Every oracle case requires a fresh process")
|
||||
_PROCESS_USED = True
|
||||
case_dir, out_dir = Path(case_dir).resolve(strict=True), Path(out_dir).resolve()
|
||||
spec = json.loads((case_dir / "spec.json").read_text(encoding="utf-8"))
|
||||
graph_dir = Path(spec["graph_dir"]).resolve(strict=True)
|
||||
bundle_dir = Path(spec["apl_bundle_dir"]).resolve(strict=True)
|
||||
worker_path = Path(spec["legacy_apl_worker"]).resolve(strict=True)
|
||||
science_python = Path(spec["science_python"]).resolve(strict=True)
|
||||
author_path = Path(spec.get("author_path", AUTHOR_PATH)).resolve(strict=True)
|
||||
ae_path = Path(spec.get("ae_oracle_path", AE_PATH)).resolve(strict=True)
|
||||
af_path = Path(spec.get("af_oracle_path", AF_PATH)).resolve(strict=True)
|
||||
coefficients_path = Path(spec["coefficients_path"]).resolve(strict=True)
|
||||
steps = int(spec["steps"])
|
||||
_check(steps == spec["steps"] and 0 < steps <= np.iinfo(np.int32).max, "Invalid number of steps")
|
||||
_check(float(spec["dt_ms"]) == .1 and np.isclose(float(spec["duration_ms"]), steps * .1, rtol=0, atol=1e-12),
|
||||
"The frozen APL protocol requires dt_ms=0.1 and a matching duration")
|
||||
_check(spec.get("precision", "float64") == "float64", "Author state must be float64")
|
||||
_check(spec.get("training", False) is False and spec.get("poi_used", False) is False,
|
||||
"Training and original poi are outside this gate")
|
||||
_check(spec["apl"] == {"tau_ms": 20, "gain": .01, "normalization": 1}, "Frozen APL parameters changed")
|
||||
record_history = spec["record_history"]
|
||||
_check(isinstance(record_history, bool), "record_history must be a boolean")
|
||||
budget_root = Path(spec.get("budget_root", out_dir.parent)).resolve()
|
||||
budget_limit = int(spec.get("output_budget_bytes", 15 * 1024 ** 3))
|
||||
_check(budget_limit > 0 and out_dir.is_relative_to(budget_root), "Invalid output budget/root")
|
||||
out_dir.mkdir(parents=True, exist_ok=False)
|
||||
mapped_files = []
|
||||
saved_preferences, old_dt, worker = None, None, None
|
||||
last_complete_tick = -1
|
||||
try:
|
||||
arrays, identity_edges = _graph_inputs(graph_dir)
|
||||
nodes, n, edges = arrays["nodes"], len(arrays["nodes"]), len(arrays["edge_id"])
|
||||
_check(n < 166700 or not record_history, "Full probes must not save dense V/g/ref histories")
|
||||
bundle = {name: np.load(bundle_dir / (name + ".npy"), mmap_mode="r", allow_pickle=False)
|
||||
for name in ("input_matrix", "output_matrix", "kc_globals", "apl_globals")}
|
||||
for name, shape in (("input_matrix", (2, 166700)), ("output_matrix", (2, 4064))):
|
||||
value = bundle[name]
|
||||
_check(value.dtype == np.dtype("float32") and value.shape == shape and value.flags.c_contiguous,
|
||||
f"Frozen APL {name} shape/dtype/layout changed")
|
||||
_check(np.isfinite(value).all() and np.all(value >= 0), f"Invalid positive raw-normalized APL {name}")
|
||||
for name, count in (("kc_globals", 4064), ("apl_globals", 2)):
|
||||
value = bundle[name]
|
||||
_check(value.dtype == np.dtype("int32") and value.shape == (count,) and value[0] >= 0 and
|
||||
value[-1] < 166700 and np.all(np.diff(value) > 0), f"Frozen APL identity {name} changed")
|
||||
_check(not np.intersect1d(bundle["kc_globals"], bundle["apl_globals"]).size, "APL and KC identities overlap")
|
||||
kc_positions = np.searchsorted(nodes, bundle["kc_globals"])
|
||||
present = kc_positions < n
|
||||
present[present] &= nodes[kc_positions[present]] == bundle["kc_globals"][present]
|
||||
kc_local, kc_in_bundle = kc_positions[present], np.flatnonzero(present)
|
||||
requested_globals, request_local, request_ptr, request_count = _requests(case_dir, nodes, steps)
|
||||
initial = None
|
||||
if (case_dir / "arrays.npz").exists():
|
||||
with np.load(case_dir / "arrays.npz", allow_pickle=False) as archive:
|
||||
_check(set(archive.files) == {"initial_v_mv", "initial_g_mv"}, "Unexpected initial-state schema")
|
||||
initial = {name: archive[name].copy() for name in archive.files}
|
||||
for name, value in initial.items():
|
||||
_check(value.dtype == np.dtype("float64") and value.shape == (n,) and np.isfinite(value).all(),
|
||||
f"Invalid diagnostic initial state: {name}")
|
||||
history_bytes = steps * n * 40 if record_history else 0
|
||||
_budget(budget_root, budget_limit, history_bytes + steps * (2 + 2 + 4064) * 4 + edges * 8 + (64 << 20))
|
||||
versions = {name: importlib.metadata.version(name) for name in
|
||||
("brian2", "numpy", "pandas", "pyarrow", "sympy", "scipy", "cython")}
|
||||
_save(out_dir / "environment.json", {"python": sys.version, "platform": platform.platform(),
|
||||
"packages": versions, "codegen": "numpy", "author_precision": "float64",
|
||||
"APL_precision": "float32 in separate science-Python process", "Torch_imported_into_Brian": False})
|
||||
for name, required in (("brian2", "2.5.1"), ("numpy", "1.22.3"), ("sympy", "1.12")):
|
||||
_check(versions[name] == required, f"{name} must remain pinned to {required}")
|
||||
import brian2 as b2
|
||||
import pandas as pd
|
||||
from brian2.core.base import BrianObject
|
||||
|
||||
ae = _load_checked(ae_path, AE_SHA256, "phase4ag_sealed_ae_helpers")
|
||||
af = _load_checked(af_path, AF_SHA256, "phase4ag_sealed_af_helpers")
|
||||
author = _load_checked(author_path, AUTHOR_SHA256, "phase4ag_unchanged_author")
|
||||
_check("torch" not in sys.modules, "Torch must not be imported into the pinned Brian process")
|
||||
comp = pd.read_csv(graph_dir / "completeness.csv", index_col=0)
|
||||
_check(np.array_equal(comp.index.to_numpy(), nodes) and
|
||||
np.array_equal(comp["local_index"].to_numpy(), np.arange(n)), "Author CSV neuron identity differs")
|
||||
del comp
|
||||
graph_hashes = {name: _digest(graph_dir / name) for name in
|
||||
tuple(key + ".npy" for key in arrays) + ("completeness.csv", "connectivity.parquet")}
|
||||
bundle_hashes = {name: _digest(bundle_dir / name) for name in BUNDLE_FILES}
|
||||
case_hashes = {name: _digest(case_dir / name) for name in ("spec.json", "requests.npz")}
|
||||
if initial is not None:
|
||||
case_hashes["arrays.npz"] = _digest(case_dir / "arrays.npz")
|
||||
worker_hash = _digest(worker_path)
|
||||
coefficient_hash = _digest(coefficients_path)
|
||||
baseline_coefficients = json.loads(coefficients_path.read_text(encoding="utf-8"))
|
||||
if "legacy_apl_worker_sha256" in spec:
|
||||
_check(worker_hash == spec["legacy_apl_worker_sha256"], "APL worker differs from preregistered source")
|
||||
_save(out_dir / "input-audit.json", {"spec": spec, "case_hashes": case_hashes,
|
||||
"graph_hashes": graph_hashes, "bundle_hashes": bundle_hashes,
|
||||
"author_sha256": AUTHOR_SHA256, "ae_sha256": AE_SHA256, "af_sha256": AF_SHA256,
|
||||
"legacy_apl_worker_sha256": worker_hash, "coefficients_sha256": coefficient_hash,
|
||||
"candidate_trace_used_as_input": False,
|
||||
"scope": "shared-APL-component closed-loop oracle, not all-components-independent CPU equivalence"})
|
||||
saved_preferences = {key: b2.prefs[key] for key in
|
||||
("codegen.target", "core.default_float_dtype", "legacy.refractory_timing")}
|
||||
old_dt = b2.defaultclock.dt
|
||||
b2.start_scope()
|
||||
b2.prefs["codegen.target"] = "numpy"
|
||||
b2.prefs["core.default_float_dtype"] = np.float64
|
||||
b2.prefs["legacy.refractory_timing"] = False
|
||||
b2.defaultclock.dt = .1 * b2.ms
|
||||
parameters = dict(author.default_params)
|
||||
_save(out_dir / "parameter-checks.json", ae._parameters_match(parameters, spec["model"], b2))
|
||||
neu, syn, spikes = author.create_model(graph_dir / "completeness.csv", graph_dir / "connectivity.parquet", parameters)
|
||||
_check(len(neu) == n and len(syn) == edges, "Author loaded an unexpected graph size")
|
||||
_check(all(np.dtype(neu.variables[key].dtype) == np.float64 for key in ("v", "g")) and
|
||||
np.dtype(syn.variables["w"].dtype) == np.float64, "Author states/weights are not float64")
|
||||
effective = _new_memmap(out_dir / "weights.npy", "<f8", (edges,))
|
||||
mapped_files.append(effective)
|
||||
width, w_unit, mask_count = BLOCK_BYTES // 8, float(parameters["w_syn"] / b2.volt), 0
|
||||
mask_hasher = hashlib.sha256()
|
||||
base_hasher = hashlib.sha256()
|
||||
effective_hasher = hashlib.sha256()
|
||||
for begin in range(0, edges, width):
|
||||
stop = min(edges, begin + width)
|
||||
np.testing.assert_array_equal(np.asarray(syn.i[begin:stop]), arrays["src"][begin:stop])
|
||||
np.testing.assert_array_equal(np.asarray(syn.j[begin:stop]), arrays["dst"][begin:stop])
|
||||
original = np.asarray(syn.w_[begin:stop], dtype=np.float64).copy()
|
||||
raw_expected = arrays["signed_raw"][begin:stop] * w_unit
|
||||
_check(np.array_equal(original.view(np.uint64), raw_expected.view(np.uint64)) and
|
||||
np.array_equal(original.view(np.uint64), arrays["weights"][begin:stop].view(np.uint64)),
|
||||
"Actual original Brian weights differ from unchanged signed raw or AF base weights")
|
||||
base_hasher.update(np.ascontiguousarray(original, dtype="<f8").tobytes())
|
||||
mask = np.isin(nodes[arrays["src"][begin:stop]], bundle["apl_globals"])
|
||||
mask_count += int(mask.sum())
|
||||
mask_hasher.update(np.ascontiguousarray(arrays["edge_id"][begin:stop][mask], dtype="<i8").tobytes())
|
||||
original[mask] = np.float64(0.0)
|
||||
effective_hasher.update(np.ascontiguousarray(original, dtype="<f8").tobytes())
|
||||
effective[begin:stop] = original
|
||||
syn.w_[begin:stop] = original
|
||||
if n == 166700:
|
||||
_check(mask_count == 5845, "Full legacy APL fast-edge mask must contain exactly 5845 existing edges")
|
||||
effective.flush()
|
||||
_save(out_dir / "apl-mask-audit.json", {"count": mask_count, "ordered_original_edge_id_sha256": mask_hasher.hexdigest(),
|
||||
"base_weight_raw_bytes_sha256": base_hasher.hexdigest(), "effective_weights_file_sha256": _digest(out_dir / "weights.npy"),
|
||||
"effective_weights_raw_sha256": effective_hasher.hexdigest(),
|
||||
"rule": "zero +0 all existing fast edges whose source GLOBAL neuron is in apl_globals; no graph edge removal",
|
||||
"graded_targets": "only KC identities present in this LIF case", "KC_targets_in_case": len(kc_local),
|
||||
"base_graph_files_modified": False, "fullgraph_required_mask_count": 5845})
|
||||
if initial is not None:
|
||||
neu.v = initial["initial_v_mv"] * b2.mV
|
||||
neu.g = initial["initial_g_mv"] * b2.mV
|
||||
v_raw = neu.variables["v"].get_value()
|
||||
g_raw = neu.variables["g"].get_value()
|
||||
last_raw = neu.variables["lastspike"].get_value()
|
||||
available_raw = neu.variables["not_refractory"].get_value()
|
||||
histories = {}
|
||||
pre_apl_histories = {}
|
||||
if record_history:
|
||||
for name, dtype in (("v", "float64"), ("g", "float64"), ("ref", "int32")):
|
||||
histories[name] = _new_memmap(out_dir / f"{name}-history.{dtype}.npy", np.dtype(dtype), (steps, n))
|
||||
mapped_files.append(histories[name])
|
||||
pre_apl_histories[name] = _new_memmap(out_dir / f"preAPL-{name}.{dtype}.npy", np.dtype(dtype), (steps, n))
|
||||
mapped_files.append(pre_apl_histories[name])
|
||||
apl_history = {}
|
||||
for name, count in (("drive", 2), ("state", 2), ("inhibition", 4064)):
|
||||
apl_history[name] = _new_memmap(out_dir / f"apl-{name}.float32.npy", "<f4", (steps, count))
|
||||
mapped_files.append(apl_history[name])
|
||||
_save(out_dir / "output-schema.json", {"history_recorded": record_history, "history_shape": [steps, n],
|
||||
"state_hash_file": "tick-state-hashes.jsonl", "state_hash_encoding": "SHA256 of contiguous little-endian raw f64 V/g and i32 ref",
|
||||
"APL_dtype": "float32", "APL_h_units": "mV; gain already applied in float32",
|
||||
"APL_SI_conversion": "h.astype(float64) * float64(0.001), then separate float64 g subtraction",
|
||||
"APL_write_schedule": "end, order=0, after original reset, unconditional raw g write; recorder end/order=1",
|
||||
"APL_input": "Brian's own actual global spikes, never candidate spikes or candidate inhibition",
|
||||
"APL_shapes": {"drive": [steps, 2], "state": [steps, 2], "inhibition": [steps, 4064]},
|
||||
"request_columns": ["tick", "neuron_idx", "attempted", "accepted", "rejected"],
|
||||
"queue_protocol": "sealed Phase4AF queue_summary; original global edge IDs, including zero-weight/ref-rejected arrivals",
|
||||
"state_npz": ["step", "neuron_idx", "counts", "v", "g", "ref"],
|
||||
"complete_marker": "summary.json complete=true only after worker exit and all checks"})
|
||||
delay_steps = int(round(float(parameters["t_dly"] / b2.ms) / .1))
|
||||
ref_steps = int(round(float(parameters["t_rfc"] / b2.ms) / .1))
|
||||
_check(delay_steps == 18 and ref_steps == 22, "Author delay/refractory changed")
|
||||
queue_len = delay_steps + 1
|
||||
due_counts = np.zeros(queue_len, np.int64)
|
||||
spike_ring = [np.empty(0, np.int32) for _ in range(queue_len)]
|
||||
outdegree = np.diff(arrays["ptr"])
|
||||
request_seen = np.zeros(steps, np.uint8)
|
||||
apl_seen = np.zeros(steps, np.uint8)
|
||||
applicable_count = np.zeros(steps, np.int32)
|
||||
rejected_count = np.zeros(steps, np.int32)
|
||||
spike_count = np.zeros(steps, np.int32)
|
||||
total_counts = np.zeros(n, np.int32)
|
||||
records = {"rows": [], "peek_calls": 0, "advance_calls": 0, "installations": []}
|
||||
last_ticks = np.zeros(n, np.int64)
|
||||
final_ref = np.zeros(n, np.int32)
|
||||
permitted = np.empty(0, np.int32)
|
||||
total_events = 0
|
||||
|
||||
def tick_now():
|
||||
return int(round(float(neu.clock.t / b2.second) / .0001))
|
||||
|
||||
def current_ref(tick):
|
||||
seen = np.isfinite(last_raw) & (last_raw >= 0)
|
||||
last_ticks.fill(0)
|
||||
last_ticks[seen] = np.rint(last_raw[seen] / .0001).astype(np.int64)
|
||||
result = np.where(seen, np.maximum(ref_steps - (tick - last_ticks), 0), 0).astype(np.int32)
|
||||
_check(np.array_equal(result == 0, available_raw), "Ref history disagrees with author not_refractory")
|
||||
return result
|
||||
|
||||
with ExitStack() as stack:
|
||||
stderr = stack.enter_context((out_dir / "apl-worker.stderr.log").open("xb"))
|
||||
worker = _APLWorker(science_python, worker_path, bundle_dir, stderr, float(spec.get("worker_timeout_seconds", 180)))
|
||||
_save(out_dir / "apl-worker-handshake.json", {"command": worker.command, "handshake": worker.hello})
|
||||
queue_file = stack.enter_context((out_dir / "queue.jsonl").open("x", encoding="utf-8"))
|
||||
event_file = stack.enter_context((out_dir / "events.bin").open("xb"))
|
||||
hash_file = stack.enter_context((out_dir / "tick-state-hashes.jsonl").open("x", encoding="utf-8"))
|
||||
tick_file = stack.enter_context((out_dir / "ticks.jsonl").open("x", encoding="utf-8"))
|
||||
requests_file = stack.enter_context((out_dir / "requests.csv").open("x", encoding="utf-8", newline=""))
|
||||
request_writer = csv.DictWriter(requests_file, fieldnames=["tick", "neuron_idx", "attempted", "accepted", "rejected"])
|
||||
request_writer.writeheader()
|
||||
|
||||
def request_operation():
|
||||
nonlocal permitted
|
||||
tick = tick_now()
|
||||
_check(0 <= tick < steps and not request_seen[tick], "Invalid/duplicate common request tick")
|
||||
request_seen[tick] = 1
|
||||
begin, stop = request_ptr[tick:tick + 2]
|
||||
local = request_local[begin:stop]
|
||||
available = available_raw[local].copy()
|
||||
permitted = local[available]
|
||||
applicable_count[tick] = len(permitted)
|
||||
rejected_count[tick] = len(local) - len(permitted)
|
||||
for node, accepted in zip(requested_globals[begin:stop], available):
|
||||
request_writer.writerow({"tick": tick, "neuron_idx": int(node), "attempted": 1,
|
||||
"accepted": int(accepted), "rejected": int(not accepted)})
|
||||
if len(permitted):
|
||||
neu.v[permitted] = parameters["v_th"] + 1 * b2.mV
|
||||
|
||||
def apl_operation():
|
||||
tick = tick_now()
|
||||
_check(0 <= tick < steps and not apl_seen[tick] and request_seen[tick], "Invalid/duplicate APL tick")
|
||||
active = np.asarray(neu.spikes, dtype=np.int32)
|
||||
_check(not len(active) or np.all(np.diff(active) > 0), "Brian actual spikes are not canonical/unique")
|
||||
_check(np.all(g_raw[active] == 0.0), "APL ran before the original author g reset")
|
||||
if record_history:
|
||||
pre_apl_histories["v"][tick] = v_raw
|
||||
pre_apl_histories["g"][tick] = g_raw
|
||||
pre_apl_histories["ref"][tick] = current_ref(tick)
|
||||
drive, state, inhibition = worker.step(tick, np.asarray(nodes[active], dtype=np.int32))
|
||||
apl_history["drive"][tick] = drive
|
||||
apl_history["state"][tick] = state
|
||||
apl_history["inhibition"][tick] = inhibition
|
||||
# These are separate array operations. Do not move the unit
|
||||
# conversion into a fused subtract; do not guard by refractory.
|
||||
delta_si = inhibition[kc_in_bundle].astype(np.float64) * np.float64(.001)
|
||||
g_raw[kc_local] = g_raw[kc_local] - delta_si
|
||||
apl_seen[tick] = 1
|
||||
|
||||
def record_end():
|
||||
nonlocal last_complete_tick, total_events, final_ref
|
||||
tick = tick_now()
|
||||
_check(tick == last_complete_tick + 1 and tick < steps and apl_seen[tick], "End tick is incomplete/out of order")
|
||||
_check(len(records["rows"]) == tick + 1, "End state lacks queue observation")
|
||||
_check(np.isfinite(v_raw).all() and np.isfinite(g_raw).all(), "Nonfinite author/APL state")
|
||||
final_ref = current_ref(tick)
|
||||
if record_history:
|
||||
histories["v"][tick], histories["g"][tick], histories["ref"][tick] = v_raw, g_raw, final_ref
|
||||
active = np.asarray(neu.spikes, dtype=np.int32)
|
||||
_check(np.isin(permitted, active).all(), "An applicable request did not produce a real spike")
|
||||
spike_count[tick] = len(active)
|
||||
total_counts[active] += 1
|
||||
if len(active):
|
||||
events = np.empty(len(active), EVENT_DTYPE)
|
||||
events["step"], events["neuron_idx"] = tick, nodes[active]
|
||||
event_file.write(memoryview(events).cast("B"))
|
||||
total_events += len(active)
|
||||
spike_ring[tick % queue_len] = active.copy()
|
||||
due_counts[(tick + delay_steps) % queue_len] = int(outdegree[active].sum(dtype=np.int64))
|
||||
hash_file.write(json.dumps({"tick": tick, "v_sha256": _array_hash(v_raw, "<f8"),
|
||||
"g_sha256": _array_hash(g_raw, "<f8"), "ref_sha256": _array_hash(final_ref, "<i4")}, sort_keys=True) + "\n")
|
||||
tick_file.write(json.dumps({"tick": tick, "spikes": len(active), "requests": int(request_count[tick]),
|
||||
"applicable_requests": int(applicable_count[tick]), "ref_rejected_requests": int(rejected_count[tick]),
|
||||
"event_sha256": _array_hash(nodes[active], "<i4"),
|
||||
"due_arrivals": records["rows"][tick]["count"], "accepted_arrivals": records["rows"][tick]["accepted_count"],
|
||||
"APL_drive_sha256": _array_hash(apl_history["drive"][tick], "<f4"),
|
||||
"APL_state_sha256": _array_hash(apl_history["state"][tick], "<f4"),
|
||||
"APL_inhibition_sha256": _array_hash(apl_history["inhibition"][tick], "<f4")}, sort_keys=True) + "\n")
|
||||
_budget(budget_root, budget_limit, reserve=1 << 20)
|
||||
last_complete_tick = tick
|
||||
if (tick + 1) % 100 == 0 or tick + 1 == steps:
|
||||
for mapped in mapped_files:
|
||||
mapped.flush()
|
||||
for handle in (queue_file, event_file, hash_file, tick_file, requests_file):
|
||||
handle.flush()
|
||||
print(json.dumps({"oracle": spec["name"], "completed_ticks": tick + 1,
|
||||
"spikes": total_events, "APL_worker_ticks": worker.tick}), flush=True)
|
||||
|
||||
class QueueInstaller(BrianObject):
|
||||
def __init__(self):
|
||||
super().__init__(clock=neu.clock, when="end", order=1_000_000, name="zz_apl_queue_installer")
|
||||
|
||||
def before_run(self, run_namespace):
|
||||
installed = []
|
||||
for code in syn.pre.code_objects:
|
||||
if "_queue" not in code.namespace or "_queue.peek()" not in str(code.code):
|
||||
continue
|
||||
original = code.namespace["_queue"]
|
||||
if isinstance(original, _QueueProxy):
|
||||
original = original._original_queue
|
||||
code.namespace["_queue"] = _QueueProxy(original, tick_now, available_raw, arrays, identity_edges,
|
||||
due_counts, spike_ring, delay_steps, queue_file, records, af.queue_summary)
|
||||
installed.append(code.name)
|
||||
_check(len(installed) == 1, "Expected one original anatomical queue consumer")
|
||||
records["installations"].append(installed[0])
|
||||
|
||||
objects = [neu, syn, spikes,
|
||||
b2.NetworkOperation(request_operation, clock=neu.clock, when="before_thresholds", order=0, name="oracle_common_requests"),
|
||||
b2.NetworkOperation(apl_operation, clock=neu.clock, when="end", order=0, name="oracle_legacy_apl_unconditional"),
|
||||
b2.NetworkOperation(record_end, clock=neu.clock, when="end", order=1, name="oracle_after_apl_record"), QueueInstaller()]
|
||||
network = b2.Network(*objects)
|
||||
_save(out_dir / "schedule.json", {"schedule": list(network.schedule), "summary": str(network.scheduling_summary())})
|
||||
network.run(0 * b2.ms, namespace={})
|
||||
_save(out_dir / "compiled-code.json", ae._code_records(network))
|
||||
compiled_coefficients = ae._coefficients(neu, parameters, b2, .0001)
|
||||
coefficient_keys = ("gd", "c0", "cg", "cv", "threshold", "reset", "seed_volt")
|
||||
_check(all(float(compiled_coefficients[key]).hex() == float(baseline_coefficients[key]).hex()
|
||||
for key in coefficient_keys), "Compiled Brian coefficients differ from sealed AF")
|
||||
_check(last_complete_tick == -1 and worker.tick == 0 and not records["rows"] and not request_seen.any(),
|
||||
"Zero-duration compilation unexpectedly advanced state")
|
||||
network.run(steps * .1 * b2.ms, namespace={})
|
||||
_save(out_dir / "generated-code.json", ae._code_records(network))
|
||||
worker_status = worker.finish(steps)
|
||||
worker = None
|
||||
|
||||
_check(last_complete_tick == steps - 1 and np.all(request_seen == 1) and np.all(apl_seen == 1), "Incomplete trial")
|
||||
_check(records["peek_calls"] == records["advance_calls"] == steps, "Incomplete queue audit")
|
||||
_check(total_events == int(spikes.num_spikes) and np.array_equal(total_counts, np.asarray(spikes.count[:])),
|
||||
"Sparse events disagree with original SpikeMonitor")
|
||||
_check(np.array_equal(request_count, applicable_count + rejected_count), "Incomplete request accounting")
|
||||
_check(np.all(np.asarray(neu.rfc[:] / b2.ms) == float(parameters["t_rfc"] / b2.ms)), "Author refractory duration changed")
|
||||
for begin in range(0, edges, width):
|
||||
stop = min(edges, begin + width)
|
||||
_check(np.array_equal(np.asarray(syn.w_[begin:stop]).view(np.uint64), effective[begin:stop].view(np.uint64)),
|
||||
"Effective synaptic weights changed during the trial")
|
||||
event_values = np.memmap(out_dir / "events.bin", dtype=EVENT_DTYPE, mode="r", shape=(total_events,)) if total_events else np.empty(0, EVENT_DTYPE)
|
||||
_check((out_dir / "events.bin").stat().st_size == total_events * 8, "Sparse event byte count mismatch")
|
||||
for begin in range(0, total_events, width):
|
||||
stop = min(total_events, begin + width)
|
||||
spike_local = np.asarray(spikes.i[begin:stop], dtype=np.int32)
|
||||
np.testing.assert_array_equal(event_values["neuron_idx"][begin:stop], nodes[spike_local])
|
||||
times = np.rint(np.asarray(spikes.t[begin:stop] / b2.second) / .0001).astype(np.int32)
|
||||
np.testing.assert_array_equal(event_values["step"][begin:stop], times)
|
||||
_budget(budget_root, budget_limit, total_events * 8 + n * 24 + (8 << 20))
|
||||
with (out_dir / "state.npz").open("xb") as handle:
|
||||
np.savez_compressed(handle, step=event_values["step"], neuron_idx=event_values["neuron_idx"],
|
||||
counts=total_counts, v=v_raw, g=g_raw, ref=final_ref)
|
||||
with (out_dir / "tick-summary.npz").open("xb") as handle:
|
||||
np.savez_compressed(handle, request_count=request_count, applicable_count=applicable_count,
|
||||
rejected_count=rejected_count, spike_count=spike_count,
|
||||
arrival_count=np.asarray([row["count"] for row in records["rows"]], np.int64),
|
||||
accepted_arrival_count=np.asarray([row["accepted_count"] for row in records["rows"]], np.int64))
|
||||
coefficients = ae._coefficients(neu, parameters, b2, .0001)
|
||||
keys = ("gd", "c0", "cg", "cv", "threshold", "reset", "seed_volt")
|
||||
_check(all(float(coefficients[key]).hex() == float(baseline_coefficients[key]).hex() for key in keys),
|
||||
"Actual Brian coefficients differ from sealed AF coefficients")
|
||||
_save(out_dir / "coefficients.json", coefficients)
|
||||
_save(out_dir / "queue-audit.json", {"peek_calls": records["peek_calls"], "advance_calls": records["advance_calls"],
|
||||
"installations": records["installations"], "same_peek_object_returned": True,
|
||||
"original_edge_ids_not_sorted_or_filtered": True, "zero_weight_and_refractory_rejected_edges_retained": True,
|
||||
"post_threshold_pre_synaptic_guard_observed": True, "forwarding_contract": "sealed AE/AF design",
|
||||
"new_unwrapped_repeat_performed": False, "limit": "Current wrapper was not independently rerun without the queue proxy"})
|
||||
for path, expected in ((author_path, AUTHOR_SHA256), (ae_path, AE_SHA256), (af_path, AF_SHA256), (worker_path, worker_hash)):
|
||||
_check(_digest(path) == expected, f"Source changed during the run: {path}")
|
||||
_check(_digest(coefficients_path) == coefficient_hash, "Sealed coefficient reference changed during the run")
|
||||
for directory, hashes in ((graph_dir, graph_hashes), (bundle_dir, bundle_hashes), (case_dir, case_hashes)):
|
||||
for name, expected in hashes.items():
|
||||
_check(_digest(directory / name) == expected, f"Read-only input changed during the run: {name}")
|
||||
for mapped in mapped_files:
|
||||
mapped.flush()
|
||||
_check("torch" not in sys.modules, "Pinned Brian process unexpectedly imported Torch")
|
||||
output_bytes = _budget(budget_root, budget_limit, 1 << 20)
|
||||
summary = {"name": spec["name"], "complete": True, "precision": "float64", "APL_precision": "float32",
|
||||
"backend": "shared-APL-component closed-loop Brian2 oracle", "steps": steps, "completed_ticks": last_complete_tick + 1,
|
||||
"nodes": n, "edges": edges, "spikes": total_events, "requests": int(request_count.sum()),
|
||||
"applicable_requests": int(applicable_count.sum()), "ref_rejected_requests": int(rejected_count.sum()),
|
||||
"APL_fast_edges_zeroed": mask_count, "APL_worker_exit_status": worker_status, "APL_worker_ticks": steps,
|
||||
"apl_fast_edges_zeroed": mask_count, "effective_weights_raw_sha256": effective_hasher.hexdigest(),
|
||||
"kc_targets_present": len(kc_local),
|
||||
"candidate_trace_used_as_input": False, "independent_CPU_all_components_claim": False,
|
||||
"history_recorded": record_history, "single_continuous_run": True, "no_training": True, "no_poi": True,
|
||||
"source_and_inputs_unchanged": True, "pass_all": True,
|
||||
"pass_all_scope": "Oracle construction, protocol and internal self-checks only; candidate comparison is a separate gate",
|
||||
"scope": "Only this fixed trajectory; legacy Torch APL is a shared numerical component, not an independent CPU APL implementation",
|
||||
"output_budget_bytes": budget_limit, "output_bytes_before_summary": output_bytes,
|
||||
"state_hashes_sha256": _digest(out_dir / "tick-state-hashes.jsonl"), "queue_sha256": _digest(out_dir / "queue.jsonl"),
|
||||
"events_sha256": _digest(out_dir / "events.bin"), "weights_sha256": _digest(out_dir / "weights.npy")}
|
||||
_save(out_dir / "summary.json", summary)
|
||||
return summary
|
||||
except Exception as error:
|
||||
for mapped in mapped_files:
|
||||
mapped.flush()
|
||||
_save(out_dir / "error.json", {"complete": False, "last_complete_tick": last_complete_tick,
|
||||
"type": type(error).__name__, "message": str(error), "partial_files_preserved": True,
|
||||
"inputs_or_duration_adjusted": False})
|
||||
raise
|
||||
finally:
|
||||
if worker is not None:
|
||||
worker.abort()
|
||||
if saved_preferences is not None:
|
||||
for key, value in saved_preferences.items():
|
||||
b2.prefs[key] = value
|
||||
b2.defaultclock.dt = old_dt
|
||||
|
||||
@@ -0,0 +1,155 @@
|
||||
"""Preregister exact legacy APL data and unmodified fixed CAL requests."""
|
||||
from datetime import datetime,timezone
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
from context import ROOT,MODEL,EXP,AF_MODEL,AF_EXP,AD_EXP,CFG,save,digest,sources,integrity,budget
|
||||
|
||||
def arr(path,value):
|
||||
with Path(path).open('xb') as f:np.save(f,value,allow_pickle=False)
|
||||
|
||||
def graph(folder,nodes,full):
|
||||
folder.mkdir();nodes=np.asarray(nodes,np.int32);ptr,dst,src,raw,base=full
|
||||
if len(nodes)==CFG['neurons']:ids=np.arange(len(dst),dtype=np.int64)
|
||||
else:
|
||||
ids=np.concatenate([np.arange(ptr[s],ptr[s+1],dtype=np.int64) for s in nodes])
|
||||
ids=ids[np.isin(dst[ids],nodes)]
|
||||
lookup=np.full(CFG['neurons'],-1,np.int32);lookup[nodes]=np.arange(len(nodes),dtype=np.int32)
|
||||
ss,dd=lookup[src[ids]],lookup[dst[ids]];pp=np.r_[np.int64(0),np.bincount(ss,minlength=len(nodes)).cumsum(dtype=np.int64)]
|
||||
for name,value in dict(nodes=nodes,edge_id=ids,src=ss,dst=dd,ptr=pp,signed_raw=raw[ids],weights=base[ids]).items():arr(folder/(name+'.npy'),value)
|
||||
pd.DataFrame({'local_index':np.arange(len(nodes),dtype=np.int32)},index=pd.Index(nodes,name='global_neuron_idx')).to_csv(folder/'completeness.csv')
|
||||
pd.DataFrame({'Presynaptic_Index':ss,'Postsynaptic_Index':dd,'Excitatory x Connectivity':raw[ids],'edge_id':ids},copy=False).to_parquet(folder/'connectivity.parquet',index=False)
|
||||
save(folder/'audit.json',dict(nodes=len(nodes),edges=len(ids),canonical_original_edge_ids=True,anatomical_weights_not_modified=True,
|
||||
effective_APL_zeroing='Only in new runtime clone, never these original weight files'))
|
||||
return nodes
|
||||
|
||||
def case(name,kind,steps,nodes,graphdir,requests,initial=None,extra=None):
|
||||
folder=EXP/'cases'/name;folder.mkdir()
|
||||
model=json.loads((AD_EXP/'protocol/preregistered.json').read_text())['config']['model']
|
||||
spec=dict(name=name,kind=kind,steps=steps,dt_ms=.1,duration_ms=steps*.1,model=model,
|
||||
graph_dir=str(graphdir),apl_bundle_dir=str(EXP/'apl-bundle'),apl=CFG['apl'],record_history=kind=='fixture',
|
||||
coefficients_path=str(AF_EXP/'oracle/no_input/coefficients.json'),
|
||||
science_python=str(ROOT/'models/male-cns-lif/.venv/bin/python'),legacy_apl_worker=str(MODEL/'legacy_apl_worker.py'),
|
||||
budget_root=str(EXP),output_budget_bytes=CFG['output_budget_bytes'],precision='float64',apl_precision='float32',training=False)
|
||||
if extra:spec.update(extra)
|
||||
requests=np.asarray(requests,dtype=np.int32).reshape(-1,2)
|
||||
if len(requests):
|
||||
requests=requests[np.lexsort((requests[:,1],requests[:,0]))]
|
||||
assert len(np.unique(requests.astype(np.int64)[:,0]*CFG['neurons']+requests[:,1]))==len(requests)
|
||||
assert requests[:,0].min()>=0 and requests[:,0].max()<steps and np.isin(requests[:,1],nodes).all()
|
||||
with (folder/'requests.npz').open('xb') as f:np.savez_compressed(f,tick=requests[:,0],neuron_idx=requests[:,1])
|
||||
if kind=='fixture':
|
||||
v=np.full(len(nodes),model['resting_potential_mv'],np.float64);g=np.zeros(len(nodes),np.float64)
|
||||
if initial:
|
||||
for node,value in initial.get('v',{}).items():v[np.flatnonzero(nodes==node)[0]]=value
|
||||
for node,value in initial.get('g',{}).items():g[np.flatnonzero(nodes==node)[0]]=value
|
||||
with (folder/'arrays.npz').open('xb') as f:np.savez_compressed(f,initial_v_mv=v,initial_g_mv=g)
|
||||
save(folder/'spec.json',spec)
|
||||
|
||||
def main():
|
||||
assert not (EXP/'protocol/preregistered.json').exists()
|
||||
assert all(not (EXP/d).exists() or not any((EXP/d).iterdir()) for d in ('protocol','graphs','apl-bundle','cases','gpu','oracle'))
|
||||
assert shutil.disk_usage(EXP.parent).free>30*1024**3
|
||||
for d in ('protocol','apl-bundle','graphs','cases','gpu','oracle','component-audit','regression','calibration','audit','extension-build','logs'):(EXP/d).mkdir(parents=True,exist_ok=True)
|
||||
can=ROOT/'datasets/male-cns-v1.0/processed/neuron-only';legacy=ROOT/'models/male-cns-lif';lp=legacy/'olfactory-grounding-v1/phase4l.yaml'
|
||||
lock=ROOT/'experiments/phase4j-mb-physiology-repair/calibration/locked-parameters.json'
|
||||
files=[AF_MODEL/'manifest.json',AF_EXP/'manifest.json',AF_EXP/'oracle/no_input/coefficients.json',
|
||||
legacy/'deterministic-runtime-v2/phase4p_revalidate.py',lp,lock,can/'neuron-index/unified-metadata.parquet',can/'gpu/csr-weights.int64.npy',
|
||||
legacy/'author-runtime-lab-v1/author_ext.cpp',legacy/'author-runtime-lab-v1/author_ext.cu',legacy/'author-runtime-lab-v1/gpu_backend.py',
|
||||
legacy/'author-runtime-lab-v1/oracle_stress.py',AD_EXP/'external-resources/model.py',AF_MODEL/'oracle_full.py']
|
||||
lroot=ROOT/'experiments/phase4l-olfactory-grounding'
|
||||
files += [lroot/'odor-encoder'/x for x in ('frozen-odor-panel.parquet','mapped-glomerular-vectors.npz','selection.json')]
|
||||
files += [lroot/'receptor-mapping/olfactory-mapping.parquet']
|
||||
save(EXP/'protocol/preregistered.json',dict(timestamp_utc=datetime.now(timezone.utc).isoformat(),sources=sources(),config=CFG,
|
||||
inputs=[dict(path=str(p.relative_to(ROOT)),sha256=digest(p)) for p in files],before_any_trajectory=True,
|
||||
parent_commit=subprocess.check_output(['git','-C',str(ROOT),'rev-parse','HEAD'],text=True).strip()))
|
||||
save(EXP/'regression/old-integrity-before.json',integrity())
|
||||
locked=json.loads(lock.read_text());assert locked['parameters_locked'] and locked['calibration_only']
|
||||
assert {k:locked['selected'][k] for k in ('gain','normalization','tau_ms')}==CFG['apl']
|
||||
meta=pd.read_parquet(can/'neuron-index/unified-metadata.parquet').sort_values('neuron_idx').reset_index(drop=True)
|
||||
assert len(meta)==CFG['neurons'] and np.array_equal(meta.neuron_idx,np.arange(CFG['neurons']))
|
||||
cls=meta.annotation_class.fillna('').astype(str).to_numpy();typ=meta.annotation_type.fillna('').astype(str).to_numpy()
|
||||
kc=np.flatnonzero(cls=='Kenyon_Cell').astype(np.int32);pn=np.flatnonzero(cls=='ALPN').astype(np.int32);apl=np.flatnonzero(typ=='APL').astype(np.int32)
|
||||
assert len(kc)==4064 and len(apl)==2
|
||||
fg=AF_EXP/'graph';ptr=np.load(fg/'ptr.npy',mmap_mode='r');dst=np.load(fg/'dst.npy',mmap_mode='r');src=np.load(fg/'src.npy',mmap_mode='r')
|
||||
signed=np.load(fg/'signed_raw.npy',mmap_mode='r');base=np.load(AF_EXP/'oracle/no_input/weights.npy',mmap_mode='r');raw=np.load(can/'gpu/csr-weights.int64.npy',mmap_mode='r')
|
||||
assert len(dst)==CFG['edges'] and int(raw.sum())==CFG['raw_weight_sum']
|
||||
aplout=np.isin(src,apl);aplids=np.flatnonzero(aplout).astype(np.int64);assert len(aplids)==5845
|
||||
I=np.zeros((2,len(meta)),np.float32);O=np.zeros((2,len(kc)),np.float32)
|
||||
alook={int(v):i for i,v in enumerate(apl)};kl=np.full(len(meta),-1,np.int32);kl[kc]=np.arange(len(kc),dtype=np.int32)
|
||||
allowed=np.isin(src,np.r_[kc,pn])&np.isin(dst,apl)
|
||||
for eid in np.flatnonzero(allowed):I[alook[int(dst[eid])],int(src[eid])]+=float(raw[eid])
|
||||
for eid in np.flatnonzero(aplout&np.isin(dst,kc)):O[alook[int(src[eid])],kl[dst[eid]]]+=float(raw[eid])
|
||||
iscale=float(I[I>0].sum()/np.count_nonzero(np.any(I>0,axis=0)));oscale=float(O[O>0].mean());bundle=EXP/'apl-bundle'
|
||||
for name,value in dict(input_matrix=I/iscale,output_matrix=O/oscale,input_raw=I,output_raw=O,kc_globals=kc,pn_globals=pn,apl_globals=apl,apl_out_ids=aplids).items():arr(bundle/(name+'.npy'),value)
|
||||
save(bundle/'scales.json',dict(input_scale=iscale,output_scale=oscale))
|
||||
save(bundle/'provenance.json',dict(canonical_root=str(can),source=str(legacy/'deterministic-runtime-v2/phase4p_revalidate.py'),source_sha256=digest(legacy/'deterministic-runtime-v2/phase4p_revalidate.py'),
|
||||
parameters=CFG['apl'],lock_file=str(lock),lock_sha256=digest(lock),full_dimensions=[list(I.shape),list(O.shape)],
|
||||
APL=len(apl),KC=len(kc),ALPN=len(pn),APL_fast_edges=len(aplids),APL_nonKC_edges=int(np.count_nonzero(aplout&~np.isin(dst,kc))),
|
||||
APL_nonKC_contacts=int(raw[aplout&~np.isin(dst,kc)].sum()),normalization_recomputed_in_original_order=True,
|
||||
raw_files_modified=False))
|
||||
full=(ptr,dst,src,signed,base);fullnodes=graph(EXP/'graphs/full',np.arange(len(meta),dtype=np.int32),full)
|
||||
pscore=I.sum(axis=0);pnrank=pn[np.lexsort((pn,-pscore[pn]))];kscore=O.sum(axis=0);kcrank=kc[np.lexsort((kc,-kscore))]
|
||||
candidates=np.flatnonzero(np.isin(src,pn)&np.isin(dst,kc));best=int(candidates[np.argmax(raw[candidates])]);pairpn=int(src[best])
|
||||
pe=np.arange(ptr[pairpn],ptr[pairpn+1],dtype=np.int64);pe=pe[np.isin(dst[pe],kc)];pe=pe[np.lexsort((dst[pe],-raw[pe]))]
|
||||
collision_kc=np.unique(dst[pe[:3]]);assert len(collision_kc)==3
|
||||
nonkc=aplids[~np.isin(dst[aplids],kc)];nonkc_target=int(dst[nonkc[np.argmax(raw[nonkc])]])
|
||||
nodes=np.unique(np.r_[apl,pnrank[:16],kcrank[:16],pairpn,collision_kc,nonkc_target,[1197,1381,1511,82312,133666]]).astype(np.int32)
|
||||
fixturegraph=EXP/'graphs/fixtures';graph(fixturegraph,nodes,full)
|
||||
p=int(pnrank[0]);k=int(kcrank[0]);kc1,kc2,kc3=map(int,collision_kc)
|
||||
designs=[('zero',100,[],None),('pn_pulse',300,[(0,p)],None),
|
||||
('kc_ref_boundary',100,[(t,k) for t in (0,1,21,22,23,44)],None),
|
||||
('two_sources',300,[(t,n) for t in (0,22,44,66,88) for n in (p,k)],None),
|
||||
('arrival_collision',120,[(0,pairpn),(0,kc1),(18,kc2),(21,kc1)],None),
|
||||
('natural_request',100,[(0,k)],dict(v={k:-44.0})),
|
||||
('apl_only',100,[(0,int(a)) for a in apl],None),
|
||||
('high_fanin',400,[(t,int(n)) for t in (0,22,44) for n in nodes if pscore[n]>0],dict(g={k:.123}))]
|
||||
assert [v[0] for v in designs]==CFG['fixtures']
|
||||
for name,steps,requests,initial in designs:case(name,'fixture',steps,nodes,fixturegraph,requests,initial)
|
||||
meta.iloc[nodes][['neuron_idx','body_id','annotation_class','annotation_type','neurotransmitter']].to_csv(EXP/'protocol/fixture-identities.csv',index=False)
|
||||
save(EXP/'protocol/fixture-selection.json',dict(pn=p,kc=k,pair_PN=pairpn,arrival_KCs=collision_kc.tolist(),real_PN_KC_edge=best,
|
||||
nonKC_APL_target=nonkc_target,method='Static raw-count ranks and exact metadata; retain ALL existing induced edges and full APL matrices/norms',runtime_selection=False))
|
||||
os.environ['CUDA_VISIBLE_DEVICES']=CFG['gpu_uuid'];sys.path.insert(0,str(legacy));sys.path.insert(0,str(legacy/'deterministic-runtime-v2'))
|
||||
sys.path.insert(0,str(legacy/'terminal-scope-diagnostic-v1'));from diagnostic import queues
|
||||
queues()
|
||||
import torch
|
||||
from phase4p_revalidate import door_inputs,make_drive
|
||||
oc,panel,vectors,selection,mapping=door_inputs();cal=panel[panel.split=='calibration'].copy();assert len(panel)==12 and len(cal)==6
|
||||
assert oc['odor_encoder']['evaluation_seeds']==CFG['calibration']['seeds']
|
||||
assert float(oc['odor_encoder']['duration_ms'])==CFG['calibration']['duration_ms']
|
||||
cal.to_parquet(EXP/'protocol/calibration-panel.parquet',index=False);calrecords=[]
|
||||
for mode in CFG['calibration']['modes']:
|
||||
for oi,(_,odor) in enumerate(cal.iterrows()):
|
||||
targets,rates=make_drive(oc,panel,vectors,selection,mapping,odor.odor_id,mode)
|
||||
assert len(np.unique(targets))==len(targets),'Original encoder emits duplicate PN targets; audit instead of silently merging'
|
||||
rates=np.asarray(rates,np.float32);steps=round(CFG['calibration']['duration_ms']/.1)
|
||||
for seed in CFG['calibration']['seeds']:
|
||||
gen=torch.Generator(device='cuda');gen.manual_seed(seed)
|
||||
original_probability=rates*.1/1000.0
|
||||
explicit_probability=rates*np.float32(.1)/np.float32(1000.)
|
||||
assert original_probability.dtype==np.float32 and np.array_equal(original_probability.view(np.uint32),explicit_probability.view(np.uint32))
|
||||
probs=torch.as_tensor(original_probability,dtype=torch.float32,device='cuda')
|
||||
forced=(torch.rand((steps,len(targets)),generator=gen,device='cuda')<probs[None,:]).cpu().numpy()
|
||||
tt,cc=np.nonzero(forced);req=np.column_stack([tt,targets[cc]]).astype(np.int32)
|
||||
name=f'cal-{mode}-{oi:02d}-s{seed}'
|
||||
case(name,'calibration',steps,fullnodes,EXP/'graphs/full',req,extra=dict(odor_id=str(odor.odor_id),odor_name=str(odor.odor_name),
|
||||
mode=mode,seed=seed,odor_index=oi,expected_external_events=float(rates.sum())*steps*.1/1000.,target_count=len(targets),rate_sum_hz=float(rates.sum())))
|
||||
with (EXP/'cases'/name/'pn-drive.npz').open('xb') as f:np.savez_compressed(f,targets=targets,rates_hz=rates)
|
||||
calrecords.append(dict(name=name,mode=mode,odor_index=oi,odor_id=str(odor.odor_id),seed=seed,requests=len(req),rates_sum=float(rates.sum())))
|
||||
save(EXP/'protocol/calibration-cases.json',calrecords)
|
||||
save(EXP/'protocol/encoder-audit.json',dict(panel_count=12,cal_count=6,median_mapped_response_sum=float(np.median(panel.mapped_response_sum)),
|
||||
target_rng_column_order='Original pn_drive, no sorting before torch.rand',seeds=CFG['calibration']['seeds'],
|
||||
requested_before_any_dynamics=True,formal_AB_run=False,heldout_run=False,torch=torch.__version__,numpy=np.__version__,
|
||||
generation='Original rates*.1/1000 float32 GPU Bernoulli, independent per-case original seed'))
|
||||
save(EXP/'protocol/prepared-hashes.json',[dict(path=str(p.relative_to(EXP)),bytes=p.stat().st_size,sha256=digest(p))
|
||||
for d in ('graphs','apl-bundle','cases','protocol') for p in sorted((EXP/d).rglob('*')) if p.is_file() and p.name!='prepared-hashes.json'])
|
||||
budget();print('PREPARED8 fixtures and60 fixed CAL inputs; no trajectories; frozen integrity PASS',flush=True)
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,48 @@
|
||||
"""A fresh process per declared case; no concurrent scientific GPU jobs."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
from context import ROOT,MODEL,EXP,CFG,check,budget
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser()
|
||||
parser.add_argument('backend',choices=['gpu','oracle','component'])
|
||||
parser.add_argument('case')
|
||||
parser.add_argument('--repeat',type=int,default=0)
|
||||
args=parser.parse_args()
|
||||
assert '/' not in args.case and '..' not in args.case and 0<=args.repeat<3
|
||||
check();budget()
|
||||
os.environ['CUDA_VISIBLE_DEVICES']=CFG['gpu_uuid']
|
||||
os.environ['CUDA_HOME']='/usr/local/cuda'
|
||||
os.environ['PATH']=str(ROOT/'models/male-cns-lif/.venv/bin')+':/usr/local/cuda/bin:'+os.environ['PATH']
|
||||
os.environ['MAX_JOBS']='2';os.environ['TORCH_CUDA_ARCH_LIST']='8.9'
|
||||
# The pinned Brian environment intentionally has neither Torch nor PyYAML.
|
||||
# Keep this preflight standard-library-only; never install into that env.
|
||||
for port in (8188,8189,8190):
|
||||
with urllib.request.urlopen(f'http://192.168.10.51:{port}/queue',timeout=5) as response:
|
||||
queue=json.load(response)
|
||||
assert not queue['queue_running'] and not queue['queue_pending'],'ComfyUI busy: defer, never evict'
|
||||
case=EXP/'cases'/args.case
|
||||
assert case.is_dir()
|
||||
out=EXP/'gpu'/f'{args.case}-r{args.repeat}'
|
||||
if args.backend=='gpu':
|
||||
from gpu_apl import run_gpu
|
||||
result=run_gpu(case,out,EXP/'extension-build')
|
||||
elif args.backend=='oracle':
|
||||
assert args.repeat==0
|
||||
from oracle_apl import run_case
|
||||
result=run_case(case,EXP/'oracle'/args.case)
|
||||
else:
|
||||
assert args.repeat==0
|
||||
from validate_components import validate
|
||||
result=validate(out,EXP/'apl-bundle',EXP/'component-audit'/(args.case+'.json'))
|
||||
budget()
|
||||
print(json.dumps(dict(backend=args.backend,case=args.case,repeat=args.repeat,
|
||||
complete=result.get('complete'),pass_all=result.get('pass_all')),allow_nan=False),flush=True)
|
||||
if not result.get('pass_all',False):raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Ordered numerical gates before any full CAL; never tune from outcomes."""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
import traceback
|
||||
from context import ROOT,MODEL,EXP,CFG,check,save,budget
|
||||
|
||||
SCIENCE=ROOT/'models/male-cns-lif/.venv/bin/python'
|
||||
ORACLE=ROOT/'experiments/phase4ad-author-semantics/environment/.venv/bin/python'
|
||||
|
||||
|
||||
def child(backend,name,repeat=0):
|
||||
log=EXP/'logs'/f'{backend}-{name}-r{repeat}.log'
|
||||
cmd=[str(ORACLE if backend=='oracle' else SCIENCE),'-B',str(MODEL/'run_case.py'),backend,name,'--repeat',str(repeat)]
|
||||
print('START',backend,name,repeat,flush=True);t0=time.perf_counter()
|
||||
with log.open('x',encoding='utf-8') as f:
|
||||
result=subprocess.run(cmd,stdout=f,stderr=subprocess.STDOUT,timeout=1800,check=False)
|
||||
print('FINISH',backend,name,repeat,'exit',result.returncode,'wall',round(time.perf_counter()-t0,2),flush=True)
|
||||
if result.returncode:raise RuntimeError(f'{backend} {name} failed; inspect {log}')
|
||||
|
||||
|
||||
def gate_case(name,repeats):
|
||||
from compare import compare,compare_gpu_repeats
|
||||
child('gpu',name)
|
||||
for r in range(1,repeats):child('gpu',name,r)
|
||||
child('oracle',name)
|
||||
result=compare(EXP/'cases'/name,EXP/'gpu'/(name+'-r0'),EXP/'oracle'/name,EXP/'audit'/('compare-'+name+'.json'))
|
||||
assert result['pass_all'],f'Closed-loop exact comparison failed: {name}; no later CAL permitted'
|
||||
for r in range(1,repeats):
|
||||
report=compare_gpu_repeats(EXP/'gpu'/(name+'-r0'),EXP/'gpu'/f'{name}-r{r}',EXP/'audit'/f'repeat-{name}-r{r}.json')
|
||||
assert report['pass_all'],f'Exact GPU repeat failed: {name}'
|
||||
child('component',name)
|
||||
return dict(case=name,pass_all=True,gpu_repeats=repeats)
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser();parser.add_argument('stage',choices=['fixtures','probes','calibration']);args=parser.parse_args()
|
||||
check();budget();receipt=EXP/'audit'/(args.stage+'-gate.json');assert not receipt.exists()
|
||||
try:
|
||||
if args.stage=='fixtures':
|
||||
rows=[gate_case(n,CFG['fixture_repeats']) for n in CFG['fixtures']]
|
||||
save(receipt,dict(pass_all=True,cases=rows,scope='Fixed real-edge induced fixtures only'))
|
||||
elif args.stage=='probes':
|
||||
assert json.loads((EXP/'audit/fixtures-gate.json').read_text())['pass_all']
|
||||
rows=[gate_case(n,1) for n in CFG['calibration']['full_oracle_probes']]
|
||||
save(receipt,dict(pass_all=True,cases=rows,scope='Only two original first-CAL fixed fullgraph trajectories'))
|
||||
else:
|
||||
assert json.loads((EXP/'audit/fixtures-gate.json').read_text())['pass_all']
|
||||
assert json.loads((EXP/'audit/probes-gate.json').read_text())['pass_all']
|
||||
records=json.loads((EXP/'protocol/calibration-cases.json').read_text());assert len(records)==60
|
||||
probes=CFG['calibration']['full_oracle_probes']
|
||||
for rec in records:
|
||||
if rec['name'] not in probes:child('gpu',rec['name'])
|
||||
if rec['seed']==CFG['calibration']['seeds'][0]:child('gpu',rec['name'],1)
|
||||
from calibration import summarize
|
||||
result=summarize()
|
||||
save(receipt,dict(pass_all=result['pass_all'],scope='Complete original six-CAL representation gate, not heldout or learning',summary='calibration/summary.json'))
|
||||
if not result['pass_all']:raise SystemExit(2)
|
||||
except Exception as e:
|
||||
save(EXP/'audit'/('stopped-'+args.stage+'.json'),dict(stage=args.stage,type=type(e).__name__,message=str(e),traceback=traceback.format_exc(),
|
||||
later_stages_not_permitted=True,partial_outputs_preserved=True,parameters_changed=False))
|
||||
raise
|
||||
budget()
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Local, synthetic tests only: no CUDA, scientific trajectories, or heldout data."""
|
||||
import copy
|
||||
import unittest
|
||||
|
||||
import numpy as np
|
||||
|
||||
from calibration import (ORIGINAL_MODES, ORIGINAL_SEEDS, ORIGINAL_CAL_IDS,
|
||||
APL_FILES, STATE_DTYPES, arrays_bitwise_equal, evaluate_gates, exact_repeat, pair_metrics,
|
||||
sample_cv, summarize_mode, validate_inventory, validate_state)
|
||||
|
||||
|
||||
GATES = {"sparse_min_fraction": .03, "sparse_max_fraction": .09,
|
||||
"minimum_separable_odor_pairs": 3, "max_fraction_strict": .5, "min_fraction_strict": 0.,
|
||||
"same_jaccard_gt_between": True, "between_euclidean_gt_within": True,
|
||||
"both_modes_required": True, "finite_required": True, "repeat_exact_required": True}
|
||||
|
||||
|
||||
def synthetic_runs(silent=False):
|
||||
result = {}
|
||||
for mode in ORIGINAL_MODES:
|
||||
for odor in range(6):
|
||||
for seed in ORIGINAL_SEEDS:
|
||||
counts = np.zeros(1000, dtype=np.int32)
|
||||
if not silent:
|
||||
counts[odor * 50:(odor + 1) * 50] = 1
|
||||
result[(mode, str(odor), seed)] = {"kc_counts": counts,
|
||||
"expected_external_events": 10., "external_events": 0 if silent else 10}
|
||||
return result
|
||||
|
||||
|
||||
class CalibrationTests(unittest.TestCase):
|
||||
def test_complete_gate_and_original_pair_counts(self):
|
||||
summary = summarize_mode(ORIGINAL_MODES[0], list(map(str, range(6))), ORIGINAL_SEEDS, synthetic_runs(), GATES)
|
||||
self.assertTrue(summary["gate"])
|
||||
self.assertEqual(summary["within_distance_count"], 60)
|
||||
self.assertEqual(summary["between_distance_count"], 75)
|
||||
self.assertEqual(summary["pair_count"], 15)
|
||||
self.assertEqual(summary["pairwise_separation_pass_count"], 15)
|
||||
self.assertTrue(all(row["within_distance_count"] == 20 and row["between_distance_count"] == 5 for row in summary["pairs"]))
|
||||
|
||||
def test_silent_is_not_pass(self):
|
||||
summary = summarize_mode(ORIGINAL_MODES[0], list(map(str, range(6))), ORIGINAL_SEEDS, synthetic_runs(True), GATES)
|
||||
self.assertFalse(summary["gate"])
|
||||
self.assertFalse(summary["conditions"]["minimum_strictly_positive"])
|
||||
self.assertFalse(summary["conditions"]["same_jaccard_gt_different"])
|
||||
self.assertFalse(summary["conditions"]["different_euclidean_gt_same"])
|
||||
self.assertEqual(summary["pairwise_separation_pass_count"], 0)
|
||||
self.assertIsNone(summary["actual_event_count_cv"])
|
||||
|
||||
def test_every_original_conjunct_is_necessary(self):
|
||||
baseline = summarize_mode(ORIGINAL_MODES[0], list(map(str, range(6))), ORIGINAL_SEEDS, synthetic_runs(), GATES)
|
||||
changes = (("KC_active_fraction_mean", .029), ("KC_active_fraction_mean", .091),
|
||||
("KC_active_fraction_max", .5), ("KC_active_fraction_min", 0.),
|
||||
("same_odor_jaccard_mean", baseline["different_odor_jaccard_mean"]),
|
||||
("different_odor_euclidean_mean", baseline["same_odor_euclidean_mean"]),
|
||||
("pairwise_separation_pass_count", 2))
|
||||
for key, value in changes:
|
||||
with self.subTest(key=key, value=value):
|
||||
modified = dict(baseline, **{key: value})
|
||||
self.assertFalse(all(evaluate_gates(modified, GATES).values()))
|
||||
for value in (.03, .09):
|
||||
self.assertTrue(evaluate_gates(dict(baseline, KC_active_fraction_mean=value), GATES)["mean_sparsity_in_closed_interval"])
|
||||
|
||||
def test_pair_specific_not_global_baseline(self):
|
||||
runs = synthetic_runs()
|
||||
mode = ORIGINAL_MODES[0]
|
||||
for odor in range(6):
|
||||
for index, seed in enumerate(ORIGINAL_SEEDS):
|
||||
counts = np.array([index + odor * 100] if odor < 2 else [10000 * odor + index * 1000], dtype=np.int32)
|
||||
runs[(mode, str(odor), seed)]["kc_counts"] = counts
|
||||
summary = summarize_mode(mode, list(map(str, range(6))), ORIGINAL_SEEDS, runs, GATES)
|
||||
first_pair = summary["pairs"][0]
|
||||
self.assertEqual(first_pair["within_this_pair_mean"], 2.)
|
||||
self.assertEqual(first_pair["between_same_seed_mean"], 100.)
|
||||
self.assertTrue(first_pair["gate"])
|
||||
self.assertGreater(summary["same_odor_euclidean_mean"], 100.)
|
||||
|
||||
def test_original_metric_zero_and_count_semantics(self):
|
||||
zero = np.zeros(3, dtype=np.int32)
|
||||
self.assertEqual(pair_metrics(zero, zero)["active_jaccard"], 1.)
|
||||
self.assertEqual(pair_metrics(zero, zero)["correlation"], 1.)
|
||||
result = pair_metrics(np.array([2, 0, 0]), np.array([1, 0, 0]))
|
||||
self.assertEqual(result["euclidean_distance"], 1.)
|
||||
self.assertEqual(result["active_jaccard"], 1.)
|
||||
self.assertAlmostEqual(sample_cv([1., 2., 3.]), .5)
|
||||
|
||||
def test_repeat_comparison_is_bits_and_dtype(self):
|
||||
self.assertFalse(arrays_bitwise_equal(np.array([0.], np.float64), np.array([-0.], np.float64)))
|
||||
self.assertFalse(arrays_bitwise_equal(np.array([1], np.int32), np.array([1], np.int64)))
|
||||
self.assertTrue(arrays_bitwise_equal(np.array([1.], np.float32), np.array([1.], np.float32)))
|
||||
|
||||
def test_exact_repeat_covers_all_recorded_semantics(self):
|
||||
first = {"state": {name: np.zeros(1, dtype) for name, dtype in STATE_DTYPES.items()},
|
||||
"apl": {name: np.zeros((1, width), np.float32) for name, width in APL_FILES.items()},
|
||||
"hashes": [{"tick": 0}], "queues": [{"tick": 0}],
|
||||
"requests": [], "component": {"environment": {"tf32": False}}}
|
||||
self.assertTrue(exact_repeat(first, copy.deepcopy(first))["pass_all"])
|
||||
for category in ("state", "apl"):
|
||||
for name in first[category]:
|
||||
bad = copy.deepcopy(first)
|
||||
bad[category][name].flat[0] = 1
|
||||
with self.subTest(category=category, name=name):
|
||||
self.assertFalse(exact_repeat(first, bad)["pass_all"])
|
||||
for key in ("hashes", "queues", "requests", "component"):
|
||||
bad = copy.deepcopy(first)
|
||||
bad[key] = {} if isinstance(first[key], list) else []
|
||||
with self.subTest(key=key):
|
||||
self.assertFalse(exact_repeat(first, bad)["pass_all"])
|
||||
|
||||
def test_raw_state_rejects_nonfinite_duplicate_and_wrong_counts(self):
|
||||
state = {"step": np.array([0], np.int32), "neuron_idx": np.array([1], np.int32),
|
||||
"counts": np.array([0, 1], np.int32), "v": np.zeros(2, np.float64),
|
||||
"g": np.zeros(2, np.float64), "ref": np.zeros(2, np.int32)}
|
||||
validate_state(state, 2, 10)
|
||||
for mutation in ("nan", "counts", "duplicate", "wrong_dtype"):
|
||||
bad = copy.deepcopy(state)
|
||||
if mutation == "nan": bad["g"][0] = np.nan
|
||||
if mutation == "counts": bad["counts"][0] = 1
|
||||
if mutation == "duplicate":
|
||||
bad["step"] = np.array([0, 0], np.int32)
|
||||
bad["neuron_idx"] = np.array([1, 1], np.int32)
|
||||
bad["counts"][1] = 2
|
||||
if mutation == "wrong_dtype": bad["v"] = bad["v"].astype(np.float32)
|
||||
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
|
||||
validate_state(bad, 2, 10)
|
||||
|
||||
def test_inventory_rejects_missing_duplicate_or_heldout_identity(self):
|
||||
odors = sorted(ORIGINAL_CAL_IDS)
|
||||
records = [{"name": f"cal-{mode}-{index:02d}-s{seed}", "mode": mode, "odor_index": index,
|
||||
"odor_id": odors[index], "seed": seed} for mode in ORIGINAL_MODES
|
||||
for index in range(6) for seed in ORIGINAL_SEEDS]
|
||||
cfg = {"calibration": {"modes": list(ORIGINAL_MODES), "seeds": list(ORIGINAL_SEEDS),
|
||||
"duration_ms": 100., "odor_count": 6, "split": "calibration", "repeat_first_seed_each_odor": True}, "gates": GATES}
|
||||
self.assertEqual(validate_inventory(records, cfg), odors)
|
||||
for malformed in (records[:-1], records[:-1] + [records[0]],
|
||||
[dict(row, odor_id="heldout") if row["odor_index"] == 0 else row for row in records]):
|
||||
with self.assertRaises(ValueError):
|
||||
validate_inventory(malformed, cfg)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,266 @@
|
||||
"""Local/remote pure protocol tests; no Brian, Torch, GPU or network execution.
|
||||
|
||||
Run with the already-provisioned Python and ``-B test_protocol.py``. The
|
||||
sidecar process is mocked with in-memory stdin and an anonymous stdout pipe.
|
||||
These tests validate transport and declared arithmetic/input contracts, not
|
||||
the scientific trajectory or independent correctness of the Torch component.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import threading
|
||||
import types
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def load_no_cache(path, name):
|
||||
module = types.ModuleType(name)
|
||||
module.__file__ = str(path)
|
||||
exec(compile(path.read_bytes(), str(path), "exec"), module.__dict__)
|
||||
return module
|
||||
|
||||
|
||||
ORACLE = load_no_cache(HERE / "oracle_apl.py", "phase4ag_protocol_oracle")
|
||||
|
||||
|
||||
def handshake(**updates):
|
||||
result = {"ok": True, "protocol": "phase4ag-apl-v1", "n_neurons": 166700,
|
||||
"n_apl": 2, "n_kc": 4064, "dt_ms": .1,
|
||||
"files_sha256": {name: "frozen-test-hash" for name in ORACLE.BUNDLE_FILES},
|
||||
"decay_f32_bits": f"{int(np.float32(math.exp(-.1 / 20.)).view(np.uint32)):08x}",
|
||||
"gain_f32_bits": f"{int(np.float32(.01).view(np.uint32)):08x}"}
|
||||
result.update(updates)
|
||||
return (json.dumps(result) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
def frame(tick=0, values=None, byte_count=None):
|
||||
values = np.arange(4068, dtype="<f4") if values is None else np.asarray(values, dtype="<f4")
|
||||
count = values.nbytes if byte_count is None else byte_count
|
||||
return struct.pack("<qQ", tick, count) + values.tobytes()
|
||||
|
||||
|
||||
class MemoryInput(io.BytesIO):
|
||||
def close(self):
|
||||
if not self.closed:
|
||||
self.closed_data = self.getvalue()
|
||||
super().close()
|
||||
|
||||
|
||||
class FakeProcess:
|
||||
"""The only use of OS descriptors is an anonymous read-only response pipe."""
|
||||
def __init__(self, response, status=0, chunk_size=97):
|
||||
read_fd, write_fd = os.pipe()
|
||||
self.stdin = MemoryInput()
|
||||
self.stdout = os.fdopen(read_fd, "rb", buffering=0)
|
||||
self.status = status
|
||||
self.terminated = False
|
||||
|
||||
def write_response():
|
||||
try:
|
||||
for begin in range(0, len(response), chunk_size):
|
||||
view = memoryview(response[begin:begin + chunk_size])
|
||||
while view:
|
||||
view = view[os.write(write_fd, view):]
|
||||
except (BrokenPipeError, OSError):
|
||||
# Invalid-response tests deliberately close the reader early.
|
||||
pass
|
||||
finally:
|
||||
os.close(write_fd)
|
||||
|
||||
self.thread = threading.Thread(target=write_response, daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
def poll(self):
|
||||
return self.status
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return self.status
|
||||
|
||||
def terminate(self):
|
||||
self.terminated = True
|
||||
|
||||
def kill(self):
|
||||
self.terminated = True
|
||||
|
||||
def close(self):
|
||||
if not self.stdin.closed:
|
||||
self.stdin.close()
|
||||
if not self.stdout.closed:
|
||||
self.stdout.close()
|
||||
self.thread.join(timeout=2)
|
||||
|
||||
|
||||
class ProtocolTests(unittest.TestCase):
|
||||
def create_worker(self, response, status=0):
|
||||
process = FakeProcess(response, status=status)
|
||||
self.addCleanup(process.close)
|
||||
source_patch = mock.patch.object(ORACLE, "_digest", return_value="frozen-test-hash")
|
||||
process_patch = mock.patch.object(ORACLE.subprocess, "Popen", return_value=process)
|
||||
source_patch.start()
|
||||
process_mock = process_patch.start()
|
||||
self.addCleanup(source_patch.stop)
|
||||
self.addCleanup(process_patch.stop)
|
||||
worker = ORACLE._APLWorker(Path("science-python"), Path("legacy_apl_worker.py"), Path("bundle"), None, timeout=2)
|
||||
self.addCleanup(worker.abort)
|
||||
return worker, process, process_mock
|
||||
|
||||
def test_exact_handshake_frame_and_clean_eof(self):
|
||||
worker, process, process_mock = self.create_worker(handshake() + frame())
|
||||
globals_ = np.array([1, 40, 166699], dtype=np.int32)
|
||||
drive, state, inhibition = worker.step(0, globals_)
|
||||
values = np.arange(4068, dtype=np.float32)
|
||||
np.testing.assert_array_equal(drive, values[:2])
|
||||
np.testing.assert_array_equal(state, values[2:4])
|
||||
np.testing.assert_array_equal(inhibition, values[4:])
|
||||
request = process.stdin.getvalue()
|
||||
self.assertEqual(struct.unpack("<qQ", request[:16]), (0, 3))
|
||||
np.testing.assert_array_equal(np.frombuffer(request[16:], dtype="<i4"), globals_)
|
||||
self.assertEqual(worker.finish(1), 0)
|
||||
command = process_mock.call_args.args[0]
|
||||
self.assertEqual(command, ["science-python", "-B", "legacy_apl_worker.py", "--bundle-dir", "bundle", "--dt-ms", "0.1"])
|
||||
self.assertFalse(process_mock.call_args.kwargs["shell"])
|
||||
|
||||
def test_empty_spike_frame_still_advances_one_tick(self):
|
||||
worker, process, _ = self.create_worker(handshake() + frame(values=np.zeros(4068, np.float32)))
|
||||
arrays = worker.step(0, np.empty(0, np.int32))
|
||||
self.assertTrue(all(np.all(value == 0) for value in arrays))
|
||||
self.assertEqual(process.stdin.getvalue(), struct.pack("<qQ", 0, 0))
|
||||
self.assertEqual(worker.finish(1), 0)
|
||||
|
||||
def test_two_consecutive_ticks(self):
|
||||
worker, _, _ = self.create_worker(handshake() + frame(0) + frame(1))
|
||||
worker.step(0, np.array([4], np.int32))
|
||||
worker.step(1, np.array([8, 10], np.int32))
|
||||
self.assertEqual(worker.finish(2), 0)
|
||||
|
||||
def test_rejects_wrong_protocol_or_dimensions(self):
|
||||
for update in ({"protocol": "other"}, {"ok": False}, {"n_neurons": 42},
|
||||
{"n_kc": 3}, {"dt_ms": .2}, {"gain_f32_bits": "00000000"},
|
||||
{"files_sha256": {}}):
|
||||
with self.subTest(update=update):
|
||||
with self.assertRaises(ValueError):
|
||||
self.create_worker(handshake(**update))
|
||||
|
||||
def test_rejects_wrong_tick_or_payload_length(self):
|
||||
for response in (frame(tick=1), frame(byte_count=8)):
|
||||
with self.subTest(response_header=response[:16]):
|
||||
worker, _, _ = self.create_worker(handshake() + response)
|
||||
with self.assertRaises(ValueError):
|
||||
worker.step(0, np.empty(0, np.int32))
|
||||
|
||||
def test_rejects_truncated_response(self):
|
||||
worker, _, _ = self.create_worker(handshake() + frame()[:-1])
|
||||
with self.assertRaises(EOFError):
|
||||
worker.step(0, np.empty(0, np.int32))
|
||||
|
||||
def test_rejects_nonfinite_or_negative_apl_payload(self):
|
||||
for bad_value in (np.nan, np.inf, -1):
|
||||
with self.subTest(value=bad_value):
|
||||
values = np.zeros(4068, np.float32)
|
||||
values[-1] = bad_value
|
||||
worker, _, _ = self.create_worker(handshake() + frame(values=values))
|
||||
with self.assertRaises(ValueError):
|
||||
worker.step(0, np.empty(0, np.int32))
|
||||
|
||||
def test_rejects_invalid_spike_dtype_order_and_range(self):
|
||||
for values in (np.array([1], np.int64), np.array([2, 1], np.int32),
|
||||
np.array([1, 1], np.int32), np.array([-1], np.int32),
|
||||
np.array([166700], np.int32)):
|
||||
with self.subTest(values=values):
|
||||
worker, process, _ = self.create_worker(handshake() + frame())
|
||||
with self.assertRaises(ValueError):
|
||||
worker.step(0, values)
|
||||
self.assertEqual(process.stdin.getvalue(), b"")
|
||||
|
||||
def test_rejects_skipped_request_tick(self):
|
||||
worker, process, _ = self.create_worker(handshake() + frame())
|
||||
with self.assertRaises(ValueError):
|
||||
worker.step(1, np.empty(0, np.int32))
|
||||
self.assertEqual(process.stdin.getvalue(), b"")
|
||||
|
||||
def test_rejects_failed_exit_and_trailing_stdout(self):
|
||||
for suffix, status in ((b"", 7), (b"unexpected", 0)):
|
||||
with self.subTest(status=status, suffix=suffix):
|
||||
worker, _, _ = self.create_worker(handshake() + frame() + suffix, status=status)
|
||||
worker.step(0, np.empty(0, np.int32))
|
||||
with self.assertRaises(ValueError):
|
||||
worker.finish(1)
|
||||
|
||||
def test_pipe_reader_timeout(self):
|
||||
read_fd, write_fd = os.pipe()
|
||||
stream = os.fdopen(read_fd, "rb", buffering=0)
|
||||
reader = ORACLE._PipeReader(stream, .05)
|
||||
try:
|
||||
with self.assertRaises(TimeoutError):
|
||||
reader.read(1)
|
||||
finally:
|
||||
os.close(write_fd)
|
||||
reader.thread.join(timeout=2)
|
||||
stream.close()
|
||||
self.assertFalse(reader.thread.is_alive())
|
||||
|
||||
|
||||
class ArithmeticAndIdentityContractTests(unittest.TestCase):
|
||||
def test_si_conversion_must_follow_float64_promotion(self):
|
||||
h32 = np.array([.01, 1 / 7, 123.456], dtype=np.float32)
|
||||
correct = h32.astype(np.float64) * np.float64(.001)
|
||||
early_conversion = (h32 * np.float32(.001)).astype(np.float64)
|
||||
self.assertFalse(np.array_equal(correct.view(np.uint64), early_conversion.view(np.uint64)))
|
||||
for actual, original in zip(correct, h32):
|
||||
self.assertEqual(actual.hex(), (float(original) * float(.001)).hex())
|
||||
|
||||
def test_declared_overlay_does_not_guard_refractory_targets(self):
|
||||
before = np.array([0, -.002, .001], dtype=np.float64)
|
||||
refractory = np.array([22, 21, 0], dtype=np.int32)
|
||||
h32 = np.array([1, 2, 3], dtype=np.float32)
|
||||
delta = h32.astype(np.float64) * np.float64(.001)
|
||||
after = before - delta
|
||||
self.assertTrue(np.all(after != before))
|
||||
np.testing.assert_array_equal(refractory, [22, 21, 0])
|
||||
self.assertEqual(after[0], -.001)
|
||||
|
||||
def test_af_queue_hash_keeps_original_noncontiguous_ids_and_rejections(self):
|
||||
path = HERE.parent / "author-fullgraph-lab-v1" / "oracle_full.py"
|
||||
self.assertEqual(hashlib.sha256(path.read_bytes()).hexdigest(), ORACLE.AF_SHA256)
|
||||
af = load_no_cache(path, "phase4ag_protocol_af")
|
||||
edges = np.array([81, 2, 500], dtype=np.int64)
|
||||
accepted = np.array([1, 0, 1], dtype=np.uint8)
|
||||
original_edges, original_mask = edges.copy(), accepted.copy()
|
||||
result = af.queue_summary(18, edges, accepted, chunk_bytes=8)
|
||||
packet = b"phase4af-queue-v1\0" + struct.pack("<qq", 18, 3) + edges.astype("<i8").tobytes() + accepted.tobytes()
|
||||
self.assertEqual(result["joint_sha256"], hashlib.sha256(packet).hexdigest())
|
||||
self.assertEqual((result["count"], result["accepted_count"]), (3, 2))
|
||||
self.assertEqual(result, af.queue_summary(18, edges, accepted, chunk_bytes=8192))
|
||||
np.testing.assert_array_equal(edges, original_edges)
|
||||
np.testing.assert_array_equal(accepted, original_mask)
|
||||
|
||||
def test_empty_queue_hash_includes_tick(self):
|
||||
af = load_no_cache(HERE.parent / "author-fullgraph-lab-v1" / "oracle_full.py", "phase4ag_empty_af")
|
||||
first = af.queue_summary(0, np.empty(0, np.int64), np.empty(0, np.uint8))
|
||||
second = af.queue_summary(1, np.empty(0, np.int64), np.empty(0, np.uint8))
|
||||
self.assertEqual(first["ordered_edge_sha256"], hashlib.sha256(b"").hexdigest())
|
||||
self.assertNotEqual(first["joint_sha256"], second["joint_sha256"])
|
||||
|
||||
def test_oracle_import_has_no_brian_or_torch_dependency(self):
|
||||
before = set(sys.modules)
|
||||
load_no_cache(HERE / "oracle_apl.py", "phase4ag_second_import")
|
||||
added = set(sys.modules) - before
|
||||
self.assertFalse(any(name == "brian2" or name.startswith("brian2.") or
|
||||
name == "torch" or name.startswith("torch.") for name in added))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,431 @@
|
||||
"""Independent extraction/replay of the frozen legacy APL component.
|
||||
|
||||
Only the initializer prefix before ``self.base`` and two original APL update
|
||||
statements are executed. No old simulator is instantiated, no learning code is
|
||||
executed, and no frozen artifact is written. The Torch replay is a component
|
||||
oracle, not an independent CPU whole-brain closed-loop oracle.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import copy
|
||||
from functools import lru_cache
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import types
|
||||
|
||||
import numpy as np
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
|
||||
|
||||
ROOT = Path('/data/digital-drosophila')
|
||||
CANONICAL = ROOT/'datasets/male-cns-v1.0/processed/neuron-only'
|
||||
SOURCE = ROOT/'models/male-cns-lif/deterministic-runtime-v2/phase4p_revalidate.py'
|
||||
SOURCE_SHA256 = 'c7e693a2da70acdce4f53af5c2853e52e8333acc93f88a46a5e8083eac639346'
|
||||
N, A, K = 166700, 2, 4064
|
||||
BUNDLE_ARRAYS = {
|
||||
'input_matrix': (np.float32, (A, N)), 'output_matrix': (np.float32, (A, K)),
|
||||
'input_raw': (np.float32, (A, N)), 'output_raw': (np.float32, (A, K)),
|
||||
'kc_globals': (np.int32, (K,)), 'apl_globals': (np.int32, (A,)),
|
||||
'pn_globals': (np.int32, None), 'apl_out_ids': (np.int64, (5845,)),
|
||||
}
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def digest(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open('rb') as source:
|
||||
for block in iter(lambda: source.read(8 << 20), b''):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def array_hash(value):
|
||||
return hashlib.sha256(np.ascontiguousarray(value).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def bit_equal(a, b):
|
||||
a, b = np.asarray(a), np.asarray(b)
|
||||
return bool(a.dtype == b.dtype and a.shape == b.shape and
|
||||
np.array_equal(np.ascontiguousarray(a).view(np.uint8),
|
||||
np.ascontiguousarray(b).view(np.uint8)))
|
||||
|
||||
|
||||
def save(path, value):
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open('x', encoding='utf-8') as target:
|
||||
json.dump(value, target, indent=2, sort_keys=True, allow_nan=False)
|
||||
target.write('\n')
|
||||
|
||||
|
||||
class ReadOnlyNamespace:
|
||||
"""The extracted prefix can read rt, but cannot mutate or add rt fields."""
|
||||
def __init__(self, **values):
|
||||
object.__setattr__(self, '_values', types.MappingProxyType(values))
|
||||
|
||||
def __getattr__(self, name):
|
||||
return self._values[name]
|
||||
|
||||
def __setattr__(self, name, value):
|
||||
raise AttributeError('Read-only canonical runtime view')
|
||||
|
||||
|
||||
def _code(nodes, filename):
|
||||
module = ast.Module(body=copy.deepcopy(nodes), type_ignores=[])
|
||||
return compile(ast.fix_missing_locations(module), str(filename), 'exec')
|
||||
|
||||
|
||||
def _self_attribute(target, field):
|
||||
return isinstance(target, ast.Attribute) and isinstance(target.value, ast.Name) and \
|
||||
target.value.id == 'self' and target.attr == field
|
||||
|
||||
|
||||
def extract_statements():
|
||||
data = SOURCE.read_bytes()
|
||||
require(hashlib.sha256(data).hexdigest() == SOURCE_SHA256, 'Frozen legacy source SHA-256 mismatch')
|
||||
text = data.decode('utf-8')
|
||||
tree = ast.parse(text, filename=str(SOURCE))
|
||||
classes = [node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == 'DeterministicAPLSim']
|
||||
require(len(classes) == 1, 'Expected exactly one frozen DeterministicAPLSim')
|
||||
methods = {node.name: node for node in classes[0].body if isinstance(node, ast.FunctionDef)}
|
||||
init, run = methods['__init__'], methods['run']
|
||||
cuts = [i for i, node in enumerate(init.body) if isinstance(node, ast.Assign) and
|
||||
any(_self_attribute(target, 'base') for target in node.targets)]
|
||||
require(len(cuts) == 1 and cuts[0] > 0, 'Missing/ambiguous self.base cutoff')
|
||||
prefix = init.body[:cuts[0]]
|
||||
# Executing no import or file-writing call is allowed in the extracted prefix.
|
||||
require(not any(isinstance(node, (ast.Import, ast.ImportFrom, ast.With, ast.AsyncWith))
|
||||
for stmt in prefix for node in ast.walk(stmt)), 'Unexpected side-effect construct in initializer prefix')
|
||||
expressions = [node for node in ast.walk(run) if isinstance(node, ast.Expr) and
|
||||
isinstance(node.value, ast.Call) and isinstance(node.value.func, ast.Attribute) and
|
||||
node.value.func.attr == 'add_' and isinstance(node.value.func.value, ast.Call) and
|
||||
isinstance(node.value.func.value.func, ast.Attribute) and
|
||||
isinstance(node.value.func.value.func.value, ast.Name) and
|
||||
node.value.func.value.func.value.id == 'apl_state' and node.value.func.value.func.attr == 'mul_']
|
||||
inhibition = [node for node in ast.walk(run) if isinstance(node, ast.Assign) and
|
||||
any(isinstance(target, ast.Name) and target.id == 'inhibition' for target in node.targets)]
|
||||
decay = [node for node in ast.walk(run) if isinstance(node, ast.Assign) and
|
||||
any(isinstance(target, ast.Name) and target.id == 'apl_decay' for target in node.targets)]
|
||||
require(len(expressions) == len(inhibition) == len(decay) == 1, 'Ambiguous frozen APL operator extraction')
|
||||
require(expressions[0].lineno < inhibition[0].lineno, 'Unexpected frozen APL update order')
|
||||
rhs = inhibition[0].value
|
||||
require(isinstance(rhs, ast.BinOp) and isinstance(rhs.op, ast.Mult), 'Unexpected inhibition gain expression')
|
||||
gain_expression = ast.Expression(copy.deepcopy(rhs.right))
|
||||
provenance = {'source': str(SOURCE), 'sha256': SOURCE_SHA256,
|
||||
'initializer_line_start': prefix[0].lineno, 'initializer_line_end': prefix[-1].end_lineno,
|
||||
'excluded_self_base_line': init.body[cuts[0]].lineno,
|
||||
'decay_line': decay[0].lineno, 'state_update_line': expressions[0].lineno,
|
||||
'inhibition_line': inhibition[0].lineno,
|
||||
'initializer_source': '\n'.join(ast.get_source_segment(text, node) for node in prefix),
|
||||
'decay_source': ast.get_source_segment(text, decay[0]),
|
||||
'state_update_source': ast.get_source_segment(text, expressions[0]),
|
||||
'inhibition_source': ast.get_source_segment(text, inhibition[0]),
|
||||
'learning_initializer_suffix_executed': False, 'original_run_loop_executed': False}
|
||||
return (_code(prefix, SOURCE), _code(decay, SOURCE),
|
||||
_code([expressions[0], inhibition[0]], SOURCE),
|
||||
compile(ast.fix_missing_locations(gain_expression), str(SOURCE), 'eval'), provenance)
|
||||
|
||||
|
||||
def _canonical_files():
|
||||
return {name: CANONICAL/path for name, path in {
|
||||
'metadata': 'neuron-index/unified-metadata.parquet',
|
||||
'indptr': 'gpu/csr-indptr.int64.npy', 'indices': 'gpu/csr-indices.int32.npy',
|
||||
'raw': 'gpu/csr-weights.int64.npy'}.items()}
|
||||
|
||||
|
||||
@lru_cache(maxsize=2)
|
||||
def reconstruct(canonical_hashes, device_name):
|
||||
"""Cache only reconstructions tied to exact canonical bytes and source hash."""
|
||||
files = _canonical_files()
|
||||
require(dict(canonical_hashes) == {name: digest(path) for name, path in files.items()},
|
||||
'Canonical bytes changed before matrix reconstruction')
|
||||
arrays = {name: np.load(files[name], mmap_mode='r', allow_pickle=False)
|
||||
for name in ('indptr', 'indices', 'raw')}
|
||||
require(arrays['indptr'].dtype == np.int64 and arrays['indices'].dtype == np.int32 and
|
||||
arrays['raw'].dtype == np.int64, 'Canonical CSR dtypes changed')
|
||||
require(arrays['indptr'].shape == (N + 1,) and arrays['indptr'][-1] == 25582938 and
|
||||
arrays['indices'].shape == arrays['raw'].shape == (25582938,), 'Canonical graph shape changed')
|
||||
require(int(arrays['raw'].sum()) == 124177617 and bool(np.all(arrays['raw'] > 0)), 'Canonical raw weights changed')
|
||||
table = pq.read_table(files['metadata'])
|
||||
require(table.num_rows == N, 'Canonical metadata count changed')
|
||||
rt = ReadOnlyNamespace(n=N, dt=.1, device=torch.device(device_name), table=table,
|
||||
indptr_np=arrays['indptr'], indices_np=arrays['indices'], raw_np=arrays['raw'])
|
||||
view = types.SimpleNamespace()
|
||||
init_code, decay_code, update_code, gain_code, provenance = extract_statements()
|
||||
scope = {'self': view, 'rt': rt, 'np': np, 'torch': torch}
|
||||
exec(init_code, scope)
|
||||
require(not hasattr(view, 'base') and not hasattr(view, 'identity'), 'Initializer crossed the learning cutoff')
|
||||
expected = {'input_matrix': view.input_gpu.detach().cpu().numpy().copy(),
|
||||
'output_matrix': view.output_gpu.detach().cpu().numpy().copy(),
|
||||
'input_raw': scope['input_matrix'], 'output_raw': scope['output_matrix'],
|
||||
'kc_globals': view.kc, 'apl_globals': view.apl, 'pn_globals': view.pn,
|
||||
'apl_out_ids': view.apl_out_ids}
|
||||
for value in expected.values():
|
||||
value.flags.writeable = False
|
||||
return view, expected, provenance, (decay_code, update_code, gain_code)
|
||||
|
||||
|
||||
class TorchObservationProxy:
|
||||
"""Observe the result of the actual extracted torch.mv call unchanged."""
|
||||
def __init__(self):
|
||||
self.drive = None
|
||||
self.calls = 0
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(torch, name)
|
||||
|
||||
def mv(self, *args, **kwargs):
|
||||
value = torch.mv(*args, **kwargs)
|
||||
self.drive = value
|
||||
self.calls += 1
|
||||
return value
|
||||
|
||||
|
||||
def _up(value):
|
||||
return np.nextafter(np.asarray(value, dtype=np.float64), np.inf)
|
||||
|
||||
|
||||
def _add(a, b):
|
||||
return _up(np.asarray(a, dtype=np.float64) + np.asarray(b, dtype=np.float64))
|
||||
|
||||
|
||||
def _mul(a, b):
|
||||
return _up(np.asarray(a, dtype=np.float64) * np.asarray(b, dtype=np.float64))
|
||||
|
||||
|
||||
def _gamma(count, unit_roundoff):
|
||||
require(count * unit_roundoff < 1, 'Gamma bound is undefined')
|
||||
# Outward rounding on the division; the denominator is rounded downward.
|
||||
numerator = _mul(count, unit_roundoff)
|
||||
denominator = np.nextafter(1.0 - numerator, -np.inf)
|
||||
return _up(numerator / denominator)
|
||||
|
||||
|
||||
def _positive_dot_upper(state, matrix):
|
||||
result = np.zeros(matrix.shape[1], np.float64)
|
||||
for index in range(len(state)):
|
||||
result = _add(result, _mul(state[index], matrix[index]))
|
||||
return result
|
||||
|
||||
|
||||
def _normal_or_zero(value, tiny):
|
||||
value = np.asarray(value)
|
||||
return bool(np.all((value == 0) | (value >= tiny)))
|
||||
|
||||
|
||||
def _new_metric():
|
||||
return {'max_abs_error': 0.0, 'sum_abs_error': 0.0, 'elements': 0,
|
||||
'max_relative_error_nonzero_reference': 0.0, 'zero_reference_nonzero_actual': 0,
|
||||
'max_error_bound': 0.0, 'max_bound_utilization': 0.0, 'bound_violations': 0}
|
||||
|
||||
|
||||
def _accumulate(metric, actual, reference, bound):
|
||||
error = np.abs(np.asarray(actual, np.float64) - reference)
|
||||
error_upper = np.where(error == 0, 0.0, _up(error))
|
||||
nonzero = reference != 0
|
||||
ratio = np.divide(error, np.abs(reference), out=np.zeros_like(error), where=nonzero)
|
||||
usage = np.divide(error_upper, bound, out=np.zeros_like(error), where=bound > 0)
|
||||
metric['max_abs_error'] = max(metric['max_abs_error'], float(error.max(initial=0)))
|
||||
metric['sum_abs_error'] += float(error.sum(dtype=np.float64))
|
||||
metric['elements'] += error.size
|
||||
metric['max_relative_error_nonzero_reference'] = max(metric['max_relative_error_nonzero_reference'], float(ratio.max(initial=0)))
|
||||
metric['zero_reference_nonzero_actual'] += int(np.count_nonzero((~nonzero) & (error != 0)))
|
||||
metric['max_error_bound'] = max(metric['max_error_bound'], float(np.max(bound, initial=0)))
|
||||
metric['max_bound_utilization'] = max(metric['max_bound_utilization'], float(usage.max(initial=0)))
|
||||
metric['bound_violations'] += int(np.count_nonzero(error_upper > bound))
|
||||
|
||||
|
||||
def validate(gpu_case_dir, bundle_dir, out_json):
|
||||
"""Validate one completed fixture; write one exclusive pure-JSON audit."""
|
||||
gpu_case_dir, bundle_dir, out_json = Path(gpu_case_dir), Path(bundle_dir), Path(out_json)
|
||||
require(not out_json.exists(), 'Refusing to overwrite an existing component audit')
|
||||
report = {'pass_all': False, 'scope': 'Frozen APL component extraction and recorded-spike replay only',
|
||||
'independent_cpu_closed_loop_oracle': False, 'brain_or_learning_parameters_changed': False,
|
||||
'candidate_function_called': False, 'gpu_case_dir': str(gpu_case_dir), 'bundle_dir': str(bundle_dir)}
|
||||
try:
|
||||
require(digest(SOURCE) == SOURCE_SHA256, 'Frozen source changed before component validation')
|
||||
require(torch.cuda.is_available() and torch.cuda.device_count() == 1 and
|
||||
'4090' in torch.cuda.get_device_name(), 'Require the one declared RTX4090')
|
||||
environment = {'torch': str(torch.__version__), 'numpy': np.__version__,
|
||||
'gpu': torch.cuda.get_device_name(), 'cuda': torch.version.cuda,
|
||||
'cuda_matmul_allow_tf32': torch.backends.cuda.matmul.allow_tf32,
|
||||
'float32_matmul_precision': torch.get_float32_matmul_precision(),
|
||||
'deterministic_algorithms': torch.are_deterministic_algorithms_enabled(),
|
||||
'precision_flags_changed': False}
|
||||
report['environment'] = environment
|
||||
canonical_hashes = tuple(sorted((name, digest(path)) for name, path in _canonical_files().items()))
|
||||
view, reconstructed, provenance, code = reconstruct(canonical_hashes, 'cuda:0')
|
||||
bundle = {}
|
||||
matrix_checks = {}
|
||||
for name, (dtype, shape) in BUNDLE_ARRAYS.items():
|
||||
value = np.load(bundle_dir/(name+'.npy'), mmap_mode='r', allow_pickle=False)
|
||||
require(value.dtype == dtype and value.ndim == len(shape or (0,)) and
|
||||
(shape is None or value.shape == shape), 'Bundle schema mismatch: '+name)
|
||||
bundle[name] = value
|
||||
matrix_checks[name] = bit_equal(value, reconstructed[name])
|
||||
scales = json.loads((bundle_dir/'scales.json').read_text())
|
||||
for name in ('input_scale', 'output_scale'):
|
||||
matrix_checks[name] = struct.pack('<d', float(scales[name])) == struct.pack('<d', float(getattr(view, name)))
|
||||
report['matrix_checks'] = matrix_checks
|
||||
report['extracted_source'] = provenance
|
||||
report['canonical_sha256'] = dict(canonical_hashes)
|
||||
report['bundle_sha256'] = {name+'.npy': digest(bundle_dir/(name+'.npy')) for name in BUNDLE_ARRAYS}
|
||||
report['bundle_sha256']['scales.json'] = digest(bundle_dir/'scales.json')
|
||||
report['reconstructed_array_sha256'] = {name: array_hash(value) for name, value in reconstructed.items()}
|
||||
bundle_provenance = json.loads((bundle_dir/'provenance.json').read_text())
|
||||
require(Path(bundle_provenance['canonical_root']).resolve() == CANONICAL.resolve(),
|
||||
'Bundle provenance names another canonical dataset')
|
||||
report['bundle_sha256']['provenance.json'] = digest(bundle_dir/'provenance.json')
|
||||
require(all(matrix_checks.values()), 'Bundle differs from the actual frozen initializer prefix')
|
||||
|
||||
traces = {name: np.load(gpu_case_dir/('apl-'+name+'.float32.npy'), mmap_mode='r', allow_pickle=False)
|
||||
for name in ('drive', 'state', 'inhibition')}
|
||||
steps = len(traces['state'])
|
||||
require(steps > 0, 'Empty APL trajectory cannot pass')
|
||||
for name, width in (('drive', A), ('state', A), ('inhibition', K)):
|
||||
require(traces[name].dtype == np.float32 and traces[name].shape == (steps, width) and
|
||||
bool(np.isfinite(traces[name]).all()) and bool(np.all(traces[name] >= 0)), 'Invalid APL trace: '+name)
|
||||
summary = json.loads((gpu_case_dir/'summary.json').read_text())
|
||||
require(summary.get('complete') is True and summary.get('pass_all') is True and
|
||||
summary.get('steps') == steps and summary.get('completed_ticks') == steps,
|
||||
'GPU fixture did not complete its internal execution checks')
|
||||
with np.load(gpu_case_dir/'state.npz', allow_pickle=False) as archive:
|
||||
event_step, event_node = archive['step'].copy(), archive['neuron_idx'].copy()
|
||||
require(event_step.dtype == event_node.dtype == np.int32 and event_step.ndim == 1 and
|
||||
event_step.shape == event_node.shape, 'Invalid sparse actual spike event schema')
|
||||
require(bool(np.all((event_step >= 0) & (event_step < steps))) and
|
||||
bool(np.all((event_node >= 0) & (event_node < N))), 'Sparse event outside its clock/identity range')
|
||||
require(np.array_equal(np.lexsort((event_node, event_step)), np.arange(len(event_step))), 'Actual events not sorted')
|
||||
keys = event_step.astype(np.int64)*N + event_node
|
||||
require(len(keys) < 2 or bool(np.all(np.diff(keys) > 0)), 'Duplicate actual spikes')
|
||||
boundaries = np.searchsorted(event_step, np.arange(steps + 1))
|
||||
|
||||
proxy = TorchObservationProxy()
|
||||
# Reconstructed original matrices are already on the original GPU/dtype.
|
||||
replay_rt = types.SimpleNamespace(spike_u8=torch.zeros(N, dtype=torch.uint8, device=view.device))
|
||||
shadow = types.SimpleNamespace(input_gpu=view.input_gpu, output_gpu=view.output_gpu, rt=replay_rt, dt=.1)
|
||||
scope = {'self': shadow, 'np': np, 'math': math, 'torch': proxy,
|
||||
'apl_state': torch.zeros(A, dtype=torch.float32, device=view.device)}
|
||||
decay_code, update_code, gain_code = code
|
||||
exec(decay_code, scope)
|
||||
decay32 = np.float32(scope['apl_decay'])
|
||||
gain32 = np.float32(eval(gain_code, {'np': np}))
|
||||
require(0 < decay32 < 1 and gain32 > 0, 'Invalid extracted positive APL constants')
|
||||
exact = {name: True for name in traces}
|
||||
first_difference = None
|
||||
I64 = np.asarray(bundle['input_matrix'], np.float64)
|
||||
O64 = np.asarray(bundle['output_matrix'], np.float64)
|
||||
require(bool(np.all(I64 >= 0)) and bool(np.all(O64 >= 0)), 'Positive-sum theory requires nonnegative matrices')
|
||||
d, q = float(decay32), float(gain32)
|
||||
u32, u64 = 2.0**-24, 2.0**-53
|
||||
gd32, gd64 = _gamma(N-1, u32), _gamma(N-1, u64)
|
||||
g232, g264 = _gamma(2, u32), _gamma(2, u64)
|
||||
gh32, gh64 = _gamma(2*A+1, u32), _gamma(2*A+1, u64)
|
||||
state64 = np.zeros(A, np.float64)
|
||||
e32 = np.zeros(A, np.float64)
|
||||
e64 = np.zeros(A, np.float64)
|
||||
previous32 = np.zeros(A, np.float64)
|
||||
statistics = {name: _new_metric() for name in traces}
|
||||
normal_arithmetic = _normal_or_zero(I64, np.finfo(np.float32).tiny) and _normal_or_zero(O64, np.finfo(np.float32).tiny)
|
||||
max_active_terms = 0
|
||||
with torch.no_grad():
|
||||
for tick in range(steps):
|
||||
fired = event_node[boundaries[tick]:boundaries[tick+1]]
|
||||
replay_rt.spike_u8.zero_()
|
||||
if len(fired):
|
||||
replay_rt.spike_u8[torch.as_tensor(fired.astype(np.int64), device=view.device)] = 1
|
||||
before_calls = proxy.calls
|
||||
exec(update_code, scope)
|
||||
require(proxy.calls == before_calls + 1, 'Original update did not issue exactly one mv')
|
||||
actual = {'drive': proxy.drive.detach().cpu().numpy(),
|
||||
'state': scope['apl_state'].detach().cpu().numpy(),
|
||||
'inhibition': scope['inhibition'].detach().cpu().numpy()}
|
||||
for name in traces:
|
||||
same = bit_equal(actual[name], traces[name][tick])
|
||||
exact[name] &= same
|
||||
if not same and first_difference is None:
|
||||
first_difference = {'tick': tick, 'field': name,
|
||||
'max_abs_error': float(np.max(np.abs(actual[name].astype(np.float64)-traces[name][tick])))}
|
||||
# Float64 reference is measurement only and never feeds either brain.
|
||||
selected = I64[:, fired]
|
||||
drive64 = selected.sum(axis=1, dtype=np.float64)
|
||||
max_active_terms = max(max_active_terms, int(np.count_nonzero(selected, axis=1).max(initial=0)))
|
||||
Dupper = _up(drive64 / np.nextafter(1.0-gd64, -np.inf))
|
||||
drive_bound = _add(_mul(gd32, Dupper), _mul(gd64, Dupper))
|
||||
state32 = traces['state'][tick].astype(np.float64)
|
||||
normal_arithmetic &= _normal_or_zero(d*previous32, np.finfo(np.float32).tiny)
|
||||
normal_arithmetic &= _normal_or_zero(O64*state32[:, None], np.finfo(np.float32).tiny)
|
||||
e32 = _add(_mul(d, e32), _add(_mul(g232, _add(_mul(d, previous32), Dupper)),
|
||||
_mul(1.0+u32, _mul(gd32, Dupper))))
|
||||
e64 = _add(_mul(d, e64), _add(_mul(g264, _add(_mul(d, state64), Dupper)),
|
||||
_mul(1.0+u64, _mul(gd64, Dupper))))
|
||||
state64 = d*state64 + drive64
|
||||
inhibition64 = np.matmul(state64, O64)*q
|
||||
h32bound = _add(_mul(q, _positive_dot_upper(e32, O64)),
|
||||
_mul(gh32, _mul(q, _positive_dot_upper(state32, O64))))
|
||||
h64bound = _add(_mul(q, _positive_dot_upper(e64, O64)),
|
||||
_mul(gh64, _mul(q, _positive_dot_upper(state64, O64))))
|
||||
normal_arithmetic &= _normal_or_zero(q*np.matmul(state32, O64), np.finfo(np.float32).tiny)
|
||||
_accumulate(statistics['drive'], traces['drive'][tick], drive64, drive_bound)
|
||||
_accumulate(statistics['state'], traces['state'][tick], state64, _add(e32, e64))
|
||||
_accumulate(statistics['inhibition'], traces['inhibition'][tick], inhibition64, _add(h32bound, h64bound))
|
||||
previous32 = state32
|
||||
for metric in statistics.values():
|
||||
metric['mean_abs_error'] = metric.pop('sum_abs_error') / metric['elements']
|
||||
theory_applicable = bool(normal_arithmetic and not environment['cuda_matmul_allow_tf32'] and
|
||||
environment['float32_matmul_precision'] == 'highest')
|
||||
report.update(steps=steps, actual_spike_events=len(event_step),
|
||||
ast_replay_bitwise_exact=exact, first_ast_difference=first_difference,
|
||||
value_units={'drive': 'legacy normalized-contact spike drive',
|
||||
'state': 'legacy normalized-drive accumulator', 'inhibition': 'legacy mV'},
|
||||
decay_f32_bits=f'{int(decay32.view(np.uint32)):08x}', gain_f32_bits=f'{int(gain32.view(np.uint32)):08x}',
|
||||
float64_measurement=statistics,
|
||||
theoretical_bound={'applicable': theory_applicable, 'normal_or_zero_arithmetic': bool(normal_arithmetic),
|
||||
'gamma_definition': 'gamma_n=(n*u)/(1-n*u), upward-rounded; u32=2^-24,u64=2^-53',
|
||||
'drive_n': N-1, 'max_observed_nonzero_drive_terms': max_active_terms,
|
||||
'gamma_drive_f32': float(gd32), 'gamma_drive_f64': float(gd64),
|
||||
'state_recurrence': 'E_p[t] <= d*E_p[t-1] + gamma_2,p*(d*abs(s_p[t-1])+D_upper) + (1+u_p)*gamma_(N-1),p*D_upper',
|
||||
'output_bound': 'q*sum(B*E_p) + gamma_(2*A+1),p*q*sum(B*abs(s_p)); compare with E32+E64',
|
||||
'assumptions': 'nonnegative f32 matrices, spikes exactly 0/1, fixed stored f32 decay/gain, normal finite RN binary32/64 operations; no reduced-precision matmul',
|
||||
'bound_arithmetic': 'Each positive bound operation is rounded toward +infinity via nextafter; denominator toward -infinity',
|
||||
'tolerance_fitted_to_observed_error': False,
|
||||
'f64_values_used_to_drive_brain': False,
|
||||
'no_cpu_gpu_bitwise_expectation_for_f64_measurement': True},
|
||||
input_sha256={name: digest(gpu_case_dir/name) for name in
|
||||
('state.npz', 'summary.json', 'apl-drive.float32.npy', 'apl-state.float32.npy', 'apl-inhibition.float32.npy')})
|
||||
report['pass_all'] = bool(all(matrix_checks.values()) and all(exact.values()) and theory_applicable and
|
||||
all(value['bound_violations'] == 0 for value in statistics.values()))
|
||||
report['limitations'] = [
|
||||
'AST shadow and candidate use the same original Torch/CUDA primitives; this validates preserved component semantics, not an independent CPU whole-brain oracle.',
|
||||
'Float64 is a non-feedback mathematical measurement with explicit gamma error bounds, not replacement APL dynamics.',
|
||||
'Replay uses the recorded actual spike train; it does not establish equivalence for untested closed-loop inputs.',
|
||||
'The N-1 positive-sum bound is deliberately conservative; passing it does not prove physiological validity or learning.']
|
||||
save(out_json, report)
|
||||
return report
|
||||
except Exception as error:
|
||||
report['error'] = {'type': type(error).__name__, 'message': str(error)}
|
||||
if not out_json.exists():
|
||||
save(out_json, report)
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('gpu_case_dir', type=Path)
|
||||
parser.add_argument('bundle_dir', type=Path)
|
||||
parser.add_argument('out_json', type=Path)
|
||||
args = parser.parse_args()
|
||||
result = validate(args.gpu_case_dir, args.bundle_dir, args.out_json)
|
||||
print(json.dumps({'pass_all': result['pass_all'], 'steps': result['steps']}), flush=True)
|
||||
if not result['pass_all']:
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Run old reference unchanged, writing only new experiment output."""
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
from context import ROOT,EXP,CFG,save,check
|
||||
|
||||
|
||||
def main():
|
||||
check();assert not (EXP/'regression/frozen-five.json').exists()
|
||||
rtpath=ROOT/'models/male-cns-lif/deterministic-runtime-v2';pc=yaml.safe_load((rtpath/'phase4p.yaml').read_text())
|
||||
os.environ['CUDA_VISIBLE_DEVICES']=CFG['gpu_uuid'];os.environ['CUDA_HOME']='/usr/local/cuda'
|
||||
os.environ['PATH']=str(ROOT/'models/male-cns-lif/.venv/bin')+':/usr/local/cuda/bin:'+os.environ['PATH']
|
||||
os.environ['MAX_JOBS']='2';os.environ['TORCH_CUDA_ARCH_LIST']='8.9'
|
||||
sys.path.insert(0,str(ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'))
|
||||
from diagnostic import queues
|
||||
queues();sys.path.insert(0,str(ROOT/'models/male-cns-lif'));sys.path.insert(0,str(rtpath))
|
||||
import torch,phase4p
|
||||
assert torch.cuda.mem_get_info()[0]>2*1024**3
|
||||
torch.cuda.reset_peak_memory_stats();t0=time.perf_counter();rt=phase4p.CanonicalRuntime(pc,rtpath,EXP);rows=[]
|
||||
for spec in pc['reference_trials']:
|
||||
queues();base=ROOT/'experiments/phase4p-deterministic-runtime/phase4c-v2/reference'/spec['name']
|
||||
ev=pd.read_parquet(base/'spike-events.parquet')
|
||||
ref=dict(step=ev.step.to_numpy(),neuron_idx=ev.neuron_idx.to_numpy(),counts=np.load(base/'per-neuron-spike-counts.int32.npy'),v=np.load(base/'final-voltage.float32.npy'),g=np.load(base/'final-g.float32.npy'))
|
||||
result=rt.run(np.flatnonzero(rt.pops[spec['population']]),spec['rate_hz'],spec['seed'],spec['duration_ms'])
|
||||
row=dict(trial=spec['name'],exact={k:bool(np.array_equal(v,result[k])) for k,v in ref.items()},wall_seconds=result['metrics']['wall_seconds'])
|
||||
rows.append(row);print(row,flush=True)
|
||||
save(EXP/'regression/frozen-five.json',dict(pass_all=all(all(r['exact'].values()) for r in rows),trials=rows,
|
||||
wall_seconds=time.perf_counter()-t0,peak_allocated_vram=int(torch.cuda.max_memory_allocated()),oracle='old frozen deterministic-v2 only'))
|
||||
assert all(all(r['exact'].values()) for r in rows)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,84 @@
|
||||
# Phase4AG: author LIF + frozen legacy graded APL
|
||||
|
||||
Execution revision r2: the first registered v1 attempt stopped before the first
|
||||
Brian tick because resolving the science-venv Python symlink launched system
|
||||
Python without NumPy. All v1 files and partial outputs are preserved separately.
|
||||
This revision retains the declared venv invocation path. It does not change any
|
||||
equation, arithmetic, mask, stimulus, gate, parameter or choice of trial.
|
||||
The complete original protocol is rerun from fresh state; no failed trial is
|
||||
overwritten or omitted, and the first version cannot be claimed numerical PASS.
|
||||
|
||||
Authorized independent hypothesis branch. All old source, raw connectome, signs,
|
||||
W_syn, APL calibration, eligibility-v2 and 14,232-edge mask remain read-only.
|
||||
No learning, gain/threshold changes, new edges, top-K, heldout activity, game or RL.
|
||||
|
||||
Author code provenance is the already archived MIT-licensed
|
||||
https://github.com/philshiu/Drosophila_brain_model at
|
||||
91bdd1e7dcf193f3e7ca5a8933497fcef63b7960; original model.py SHA-256
|
||||
fc45837d7122c6ce2a7f3f2f23c515992e4b232aadb919efabb72337fac88e4e.
|
||||
Associated paper: https://www.nature.com/articles/s41586-024-07763-9.
|
||||
This new MaleCNS/APL combination is not a strict reproduction of that model.
|
||||
|
||||
The declared combination is author float64 LIF plus the exact old float32 graded
|
||||
APL operator after synapses/resets at end. All 5,845 original APL fast out-edges
|
||||
are zeroed only in the effective in-memory model, as in the legacy overlay.
|
||||
Only KC receives graded replacement; known 1,212 non-KC edges remain unreplaced.
|
||||
This existing scope limitation is not silently repaired in a compatibility test.
|
||||
|
||||
Use the original full normalized I[2,166700]/O[2,4064] float32 bytes and original
|
||||
normalizers, even for induced real subgraphs. APL reads actual binary spikes,
|
||||
never requests, and keeps independent float32 state. Preserve unfused operations:
|
||||
state.mul_(float(f32(exp(-.1/20)))).add_(torch.mv(I, spikes_f32));
|
||||
h32=torch.matmul(state,O)*f32(.01). Promote h32 to float64, multiply by float64
|
||||
.001, then separately subtract from SI g. Do not alter TF32/default CuBLAS flags;
|
||||
record them. No added APL delay, drive dt/tau scaling, truncation or clipping.
|
||||
|
||||
APL writes KC g unconditionally after author reset, including refractory KC.
|
||||
Author g integration is frozen during refractory; therefore negative overlay g
|
||||
may accumulate and differs from old dynamics. This is an explicit hypothesis,
|
||||
not an assertion of original author or measured physiology. Guarded writes or
|
||||
reset-before/after changes would be different hypotheses, not fixes to this one.
|
||||
|
||||
Eight fixed metadata/connectivity-selected fixtures cover zero/reset, real PN/KC
|
||||
drive, current actual event feedback, duplicate natural/request event, 18-step
|
||||
arrival vs22-step refractory boundary, full fast APL mask, non-KC scope and mixed
|
||||
precision/high fan-in. No choice is made from runtime responses. Each GPU case
|
||||
has3 independent fresh runs. A Brian2 author oracle starts from its own state,
|
||||
computes its own actual spikes, and uses an isolated standard-I/O legacy-APL
|
||||
worker with independent state. No candidate inhibition trace is fed into Brian.
|
||||
All tick states, events, requests, queue order/guard, APL drive/state/h must agree
|
||||
exactly. The APL component is shared, so this is NOT a fully independent CPU
|
||||
oracle. Independently reconstruct old matrices using frozen initializer AST,
|
||||
replay actual events through original update AST, require exact component bytes;
|
||||
also quantify an independent float64 positive-sum arithmetic error bound. It is
|
||||
not a replacement for exact legacy equivalence. No arithmetic bound is adjusted.
|
||||
|
||||
Only after all fixture gates pass, run two predeclared full-graph probes: first
|
||||
old calibration odor, first original seed, both encoder modes. Compare each with
|
||||
the independently running Brian + isolated APL worker. Save state hashes and sparse
|
||||
events, not a time-by-all-neurons dense spike matrix. If a numerical gate fails,
|
||||
stop CAL; keep diagnostic evidence and do not widen tolerance.
|
||||
|
||||
Then run the complete original6 CAL odors ×5 original seeds ×2 original modes,
|
||||
100ms per run. Retain the ORIGINAL12-odor median for matched-energy encoding,
|
||||
original pn_drive target/RNG column order, original evaluation seeds51101..51105.
|
||||
Save Poisson requests before any trajectory. Only calibration dynamic cases are
|
||||
used; formal A/B and heldout remain unrun. Repeat first seed for every odor/mode.
|
||||
No per-odor pulse, amplitude, timing or threshold search. No learning.
|
||||
|
||||
Recompute the full original4L representation criteria on CAL only: mean KC active
|
||||
fraction .03..09, min>0, max<.5, same-odor Jaccard>different, between Euclidean>
|
||||
within, >=3 pairs whose mean between exceeds THAT PAIR's pooled within distances,
|
||||
both modes. Add finiteness and exact repeat requirements. CAL PASS does not imply
|
||||
heldout generalization, timing/credit specificity, memory or learning usability.
|
||||
If CAL fails stop promotion and report it, not a least-bad candidate.
|
||||
|
||||
Before/after verify SHA-256 of all original230239 files and AA--AF seals; rerun original five
|
||||
deterministic reference trials to new output. Archive code/env/source SHA,
|
||||
matrices/norms/masks/identities, requests and all outputs. No published candidate.
|
||||
|
||||
Technical references (queried2026-09-30): Brian2 2.5.1 refractory documentation
|
||||
https://brian2.readthedocs.io/en/2.5.1/user/refractoriness.html (actual generated code
|
||||
remains authoritative); PyTorch numerical accuracy notes
|
||||
https://docs.pytorch.org/docs/2.14/notes/numerical_accuracy.html (cross-backend
|
||||
floating operations are not assumed bitwise equivalent).
|
||||
@@ -0,0 +1,396 @@
|
||||
"""Complete original Phase 4L representation metrics on fixed CAL inputs only.
|
||||
|
||||
No simulation, fitting, heldout trajectory access, or candidate generation is
|
||||
performed here. The original 4L pair-specific within-odor baseline is retained;
|
||||
the weaker Phase 4P r4 aggregate gate is deliberately not used.
|
||||
"""
|
||||
import csv
|
||||
import hashlib
|
||||
import itertools
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
STATE_DTYPES = {"step": np.int32, "neuron_idx": np.int32, "counts": np.int32,
|
||||
"v": np.float64, "g": np.float64, "ref": np.int32}
|
||||
REQUEST_FIELDS = ("tick", "neuron_idx", "attempted", "accepted", "rejected")
|
||||
APL_FILES = {"apl-drive.float32.npy": 2, "apl-state.float32.npy": 2,
|
||||
"apl-inhibition.float32.npy": 4064}
|
||||
ORIGINAL_CAL_IDS = {
|
||||
"OMSUIQOIVADKIM-RXMQYKEDSA-N", "LABTWGUMFABVFG-ARJAWSKDSA-N",
|
||||
"QNVRIHYSUZMSGM-ZCFIWIBFSA-N", "VLSVVMPLPMNWBH-UHFFFAOYSA-N",
|
||||
"XHIUFYZDQBSEMF-UHFFFAOYSA-N", "OMSUIQOIVADKIM-YFKPBYRVSA-N"}
|
||||
ORIGINAL_MODES = ("physiological", "matched_energy")
|
||||
ORIGINAL_SEEDS = (51101, 51102, 51103, 51104, 51105)
|
||||
HEX_SHA = re.compile(r"[0-9a-f]{64}\Z")
|
||||
_DEFAULT_OUTPUT = object()
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def read_json(path):
|
||||
with Path(path).open(encoding="utf-8") as stream:
|
||||
return json.load(stream)
|
||||
|
||||
|
||||
def file_hash(path):
|
||||
digest = hashlib.sha256()
|
||||
with Path(path).open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(8 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def raw_hash(value):
|
||||
return hashlib.sha256(np.ascontiguousarray(value).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def arrays_bitwise_equal(first, second):
|
||||
"""Unlike array_equal, also reject different dtypes and signed-zero bits."""
|
||||
return bool(first.dtype == second.dtype and first.shape == second.shape and
|
||||
np.array_equal(np.ascontiguousarray(first).view(np.uint8),
|
||||
np.ascontiguousarray(second).view(np.uint8)))
|
||||
|
||||
|
||||
def pair_metrics(a, b):
|
||||
"""Exact definitions copied from phase4i.pair_metrics/safe_corr/cosine."""
|
||||
x, y = a.astype(np.float64), b.astype(np.float64)
|
||||
aa, bb = a > 0, b > 0
|
||||
union = int(np.count_nonzero(aa | bb))
|
||||
intersection = int(np.count_nonzero(aa & bb))
|
||||
denominator = float(np.linalg.norm(x) * np.linalg.norm(y))
|
||||
cosine = 1.0 if denominator == 0 and np.array_equal(x, y) else (
|
||||
0.0 if denominator == 0 else float(np.dot(x, y) / denominator))
|
||||
correlation = (1.0 if np.array_equal(x, y) else 0.0) if np.std(x) == 0 or np.std(y) == 0 else float(np.corrcoef(x, y)[0, 1])
|
||||
return {"active_intersection": intersection, "active_union": union,
|
||||
"active_jaccard": 1.0 if union == 0 else float(intersection / union),
|
||||
"cosine_similarity": cosine, "correlation": correlation,
|
||||
"euclidean_distance": float(np.linalg.norm(x - y))}
|
||||
|
||||
|
||||
def evaluate_gates(summary, gates):
|
||||
"""Each conjunct in original phase4l.py:239, kept visible for audit."""
|
||||
return {
|
||||
"mean_sparsity_in_closed_interval": bool(float(gates["sparse_min_fraction"]) <=
|
||||
summary["KC_active_fraction_mean"] <= float(gates["sparse_max_fraction"])),
|
||||
"maximum_strictly_below_half": bool(summary["KC_active_fraction_max"] < 0.5),
|
||||
"minimum_strictly_positive": bool(summary["KC_active_fraction_min"] > 0),
|
||||
"same_jaccard_gt_different": bool(summary["same_odor_jaccard_mean"] > summary["different_odor_jaccard_mean"]),
|
||||
"different_euclidean_gt_same": bool(summary["different_odor_euclidean_mean"] > summary["same_odor_euclidean_mean"]),
|
||||
"at_least_original_pair_count": bool(summary["pairwise_separation_pass_count"] >= int(gates["minimum_separable_odor_pairs"]))}
|
||||
|
||||
|
||||
def sample_cv(values):
|
||||
"""Original pandas Series.std uses ddof=1. Undefined diagnostic -> null."""
|
||||
values = np.asarray(values, dtype=np.float64)
|
||||
mean = float(values.mean())
|
||||
return None if mean == 0 else float(values.std(ddof=1) / mean)
|
||||
|
||||
|
||||
def summarize_mode(mode, odor_ids, seeds, runs, gates):
|
||||
within, within_j, between, between_j, between_cos, between_corr = [], [], [], [], [], []
|
||||
pairs = []
|
||||
for odor in odor_ids:
|
||||
arrays = [runs[(mode, odor, seed)]["kc_counts"] for seed in seeds]
|
||||
for first, second in itertools.combinations(arrays, 2):
|
||||
metric = pair_metrics(first, second)
|
||||
within.append(metric["euclidean_distance"])
|
||||
within_j.append(metric["active_jaccard"])
|
||||
for first_odor, second_odor in itertools.combinations(odor_ids, 2):
|
||||
distances = []
|
||||
for seed in seeds:
|
||||
metric = pair_metrics(runs[(mode, first_odor, seed)]["kc_counts"],
|
||||
runs[(mode, second_odor, seed)]["kc_counts"])
|
||||
distances.append(metric["euclidean_distance"])
|
||||
between.append(metric["euclidean_distance"])
|
||||
between_j.append(metric["active_jaccard"])
|
||||
between_cos.append(metric["cosine_similarity"])
|
||||
between_corr.append(metric["correlation"])
|
||||
pair_within = []
|
||||
for odor in (first_odor, second_odor):
|
||||
arrays = [runs[(mode, odor, seed)]["kc_counts"] for seed in seeds]
|
||||
pair_within.extend(pair_metrics(first, second)["euclidean_distance"]
|
||||
for first, second in itertools.combinations(arrays, 2))
|
||||
cross_mean, within_mean = float(np.mean(distances)), float(np.mean(pair_within))
|
||||
pairs.append({"odor_id_a": first_odor, "odor_id_b": second_odor,
|
||||
"between_same_seed_mean": cross_mean, "within_this_pair_mean": within_mean,
|
||||
"between_distance_count": len(distances), "within_distance_count": len(pair_within),
|
||||
"gate": bool(cross_mean > within_mean)})
|
||||
selected = [runs[(mode, odor, seed)] for odor in odor_ids for seed in seeds]
|
||||
fractions = [int(np.count_nonzero(run["kc_counts"])) / len(run["kc_counts"]) for run in selected]
|
||||
summary = {"KC_active_fraction_mean": float(np.mean(fractions)),
|
||||
"KC_active_fraction_min": float(np.min(fractions)), "KC_active_fraction_max": float(np.max(fractions)),
|
||||
"same_odor_jaccard_mean": float(np.mean(within_j)),
|
||||
"different_odor_jaccard_mean": float(np.mean(between_j)),
|
||||
"same_odor_euclidean_mean": float(np.mean(within)),
|
||||
"different_odor_euclidean_mean": float(np.mean(between)),
|
||||
"different_odor_cosine_mean": float(np.mean(between_cos)),
|
||||
"different_odor_correlation_mean": float(np.mean(between_corr)),
|
||||
"within_variance": float(np.var(within)), "between_distance_mean": float(np.mean(between)),
|
||||
"pairwise_separation_pass_count": sum(row["gate"] for row in pairs), "pair_count": len(pairs),
|
||||
"expected_event_count_cv": sample_cv([run["expected_external_events"] for run in selected]),
|
||||
"actual_event_count_cv": sample_cv([run["external_events"] for run in selected]),
|
||||
"within_distance_count": len(within), "between_distance_count": len(between),
|
||||
"pairwise_baseline": "Each pair's two odors: 10+10 within-seed distances; never pooled across all odors",
|
||||
"pairs": pairs}
|
||||
summary["conditions"] = evaluate_gates(summary, gates)
|
||||
summary["gate"] = bool(all(summary["conditions"].values()))
|
||||
return summary
|
||||
|
||||
|
||||
def validate_state(state, neurons, steps):
|
||||
require(set(state) == set(STATE_DTYPES), "state.npz must have exactly the six declared arrays")
|
||||
for name, dtype in STATE_DTYPES.items():
|
||||
value = state[name]
|
||||
require(value.dtype == dtype and value.ndim == 1, "Invalid state dtype/rank: " + name)
|
||||
require(bool(np.isfinite(value).all()), "Nonfinite state: " + name)
|
||||
if name not in ("step", "neuron_idx"):
|
||||
require(value.shape == (neurons,), "Invalid state length: " + name)
|
||||
ticks, nodes = state["step"], state["neuron_idx"]
|
||||
require(ticks.shape == nodes.shape and bool(np.all((ticks >= 0) & (ticks < steps))) and
|
||||
bool(np.all((nodes >= 0) & (nodes < neurons))), "Invalid sparse event indices")
|
||||
keys = ticks.astype(np.int64) * neurons + nodes
|
||||
require(bool(np.all(np.diff(keys) > 0)), "Sparse events not sorted unique by tick/global neuron")
|
||||
require(np.array_equal(np.bincount(nodes, minlength=neurons), state["counts"]), "Counts disagree with sparse events")
|
||||
require(bool(np.all((state["ref"] >= 0) & (state["ref"] <= 22))), "Final refractory outside frozen range")
|
||||
return keys
|
||||
|
||||
|
||||
def read_jsonl(path, steps, kind):
|
||||
with Path(path).open(encoding="utf-8") as source:
|
||||
rows = [json.loads(line) for line in source if line.strip()]
|
||||
require(len(rows) == steps and [row.get("tick") for row in rows] == list(range(steps)),
|
||||
"Missing/duplicate/reordered ticks in " + kind)
|
||||
for row in rows:
|
||||
names = ("v_sha256", "g_sha256", "ref_sha256") if kind == "state hashes" else (
|
||||
"ordered_edge_sha256", "accepted_mask_sha256", "joint_sha256")
|
||||
for name in names:
|
||||
require(isinstance(row.get(name), str) and HEX_SHA.fullmatch(row[name]), "Invalid SHA256 in " + kind)
|
||||
if kind == "queue":
|
||||
require(type(row.get("count")) is int and type(row.get("accepted_count")) is int and
|
||||
0 <= row["accepted_count"] <= row["count"], "Invalid queue counts")
|
||||
return rows
|
||||
|
||||
|
||||
def read_trial(output, case, spec, kc_globals, neurons, edges):
|
||||
summary = read_json(output / "summary.json")
|
||||
steps = spec["steps"]
|
||||
require(summary.get("complete") is True and summary.get("pass_all") is True and
|
||||
summary.get("all_finite") is True and summary.get("steps") == steps and
|
||||
summary.get("completed_ticks") == steps and summary.get("nodes") == neurons and
|
||||
summary.get("edges") == edges and summary.get("precision") == "float64" and
|
||||
summary.get("apl_precision") == "float32" and summary.get("case") == spec["name"] and
|
||||
summary.get("record_history") is False and summary.get("learning") is False,
|
||||
"Incomplete/failed/wrong-scope GPU summary")
|
||||
with np.load(output / "state.npz", allow_pickle=False) as source:
|
||||
state = {name: source[name].copy() for name in source.files}
|
||||
event_keys = validate_state(state, neurons, steps)
|
||||
hashes = read_jsonl(output / "tick-state-hashes.jsonl", steps, "state hashes")
|
||||
for name in ("v", "g", "ref"):
|
||||
require(hashes[-1][name + "_sha256"] == raw_hash(state[name]), "Final state does not match last tick hash: " + name)
|
||||
queues = read_jsonl(output / "queue.jsonl", steps, "queue")
|
||||
apl = {}
|
||||
for name, width in APL_FILES.items():
|
||||
value = np.load(output / name, mmap_mode="r", allow_pickle=False)
|
||||
require(value.dtype == np.float32 and value.shape == (steps, width) and
|
||||
bool(np.isfinite(value).all()) and bool(np.all(value >= 0)), "Invalid full APL trace: " + name)
|
||||
apl[name] = value
|
||||
with (output / "requests.csv").open(encoding="utf-8", newline="") as source:
|
||||
reader = csv.DictReader(source)
|
||||
require(tuple(reader.fieldnames or ()) == REQUEST_FIELDS, "Request CSV schema mismatch")
|
||||
request_rows = [{name: int(row[name]) for name in REQUEST_FIELDS} for row in reader]
|
||||
with np.load(case / "requests.npz", allow_pickle=False) as source:
|
||||
require(set(source.files) == {"tick", "neuron_idx"}, "Input request schema mismatch")
|
||||
expected_ticks, expected_nodes = source["tick"], source["neuron_idx"]
|
||||
require(expected_ticks.dtype == np.int32 and expected_nodes.dtype == np.int32 and
|
||||
expected_ticks.ndim == 1 and expected_ticks.shape == expected_nodes.shape,
|
||||
"Input request dtype/shape mismatch")
|
||||
require(len(request_rows) == len(expected_ticks), "Requested event coverage mismatch")
|
||||
require([row["tick"] for row in request_rows] == expected_ticks.tolist() and
|
||||
[row["neuron_idx"] for row in request_rows] == expected_nodes.tolist(), "Requests differ from frozen input")
|
||||
require(all(row["attempted"] == 1 and row["accepted"] in (0, 1) and
|
||||
row["rejected"] == 1 - row["accepted"] for row in request_rows), "Invalid request acceptance bookkeeping")
|
||||
requested_keys = expected_ticks.astype(np.int64) * neurons + expected_nodes
|
||||
require(bool(np.all(np.diff(requested_keys) > 0)) and bool(np.all((expected_ticks >= 0) & (expected_ticks < steps))) and
|
||||
bool(np.all((expected_nodes >= 0) & (expected_nodes < neurons))), "Input requests not sorted unique/in range")
|
||||
actual_accepted = np.array([row["accepted"] for row in request_rows], dtype=np.uint8)
|
||||
require(np.array_equal(np.isin(requested_keys, event_keys, assume_unique=True), actual_accepted.astype(bool)),
|
||||
"Request accepted/rejected values disagree with actual spikes")
|
||||
accepted = int(actual_accepted.sum())
|
||||
require(summary.get("requested") == len(request_rows) and summary.get("accepted_requests") == accepted and
|
||||
summary.get("rejected_requests") == len(request_rows) - accepted and
|
||||
summary.get("total_spikes") == len(state["step"]), "Summary counts disagree with raw output")
|
||||
component = read_json(output / "apl-component.json")
|
||||
require(component.get("ok") is True and component.get("n_neurons") == neurons and
|
||||
component.get("n_kc") == len(kc_globals) and component.get("n_apl") == 2 and
|
||||
component.get("dt_ms") == .1 and summary.get("apl_component") == component,
|
||||
"APL component provenance mismatch")
|
||||
bundle = Path(spec["apl_bundle_dir"])
|
||||
require(component.get("files_sha256") == {name: file_hash(bundle / name) for name in
|
||||
("input_matrix.npy", "output_matrix.npy", "kc_globals.npy", "apl_globals.npy")}, "APL bundle hashes changed")
|
||||
inputs = summary.get("input_sha256", {})
|
||||
require(inputs.get("spec.json") == file_hash(case / "spec.json") and
|
||||
inputs.get("requests.npz") == file_hash(case / "requests.npz"), "Case input provenance mismatch")
|
||||
kc_counts = state["counts"][kc_globals].copy()
|
||||
kc_event_mask = np.isin(state["neuron_idx"], kc_globals)
|
||||
record = {"case": spec["name"], "mode": spec["mode"], "odor_id": spec["odor_id"],
|
||||
"odor_name": spec["odor_name"], "seed": spec["seed"], "split": "calibration",
|
||||
"KC_active": int(np.count_nonzero(kc_counts)), "KC_active_fraction": float(np.count_nonzero(kc_counts) / len(kc_globals)),
|
||||
"KC_spikes": int(kc_counts.sum()), "KC_first_latency_ms":
|
||||
float(state["step"][kc_event_mask][0] * .1) if bool(np.any(kc_event_mask)) else None,
|
||||
"external_events": accepted, "expected_external_events": float(spec["expected_external_events"]),
|
||||
"total_PN_rate_hz": float(spec["rate_sum_hz"]), "finite": True,
|
||||
"total_spikes": len(state["step"]), "requested": len(request_rows)}
|
||||
require(np.isfinite(record["expected_external_events"]) and record["expected_external_events"] > 0,
|
||||
"CAL expected external events must be finite and positive")
|
||||
return {"state": state, "hashes": hashes, "queues": queues, "apl": apl,
|
||||
"requests": request_rows, "component": component, "kc_counts": kc_counts,
|
||||
"expected_external_events": record["expected_external_events"], "external_events": accepted,
|
||||
"record": record, "output": output}
|
||||
|
||||
|
||||
def exact_repeat(first, repeat):
|
||||
checks = {"state/" + name: arrays_bitwise_equal(first["state"][name], repeat["state"][name]) for name in STATE_DTYPES}
|
||||
checks.update({name: arrays_bitwise_equal(first["apl"][name], repeat["apl"][name]) for name in APL_FILES})
|
||||
checks.update(tick_state_hashes=first["hashes"] == repeat["hashes"],
|
||||
queues=first["queues"] == repeat["queues"], requests=first["requests"] == repeat["requests"],
|
||||
apl_component=first["component"] == repeat["component"])
|
||||
return {"pass_all": bool(all(checks.values())), "checks": checks,
|
||||
"scope": "All six state arrays, every tick all-neuron hashes, APL raw float32 traces, queues, requests and APL environment"}
|
||||
|
||||
|
||||
def validate_inventory(records, cfg):
|
||||
require(tuple(cfg["calibration"]["modes"]) == ORIGINAL_MODES and
|
||||
tuple(cfg["calibration"]["seeds"]) == ORIGINAL_SEEDS and
|
||||
cfg["calibration"]["duration_ms"] == 100.0 and cfg["calibration"]["odor_count"] == 6 and
|
||||
cfg["calibration"]["split"] == "calibration" and
|
||||
cfg["calibration"]["repeat_first_seed_each_odor"] is True, "Frozen CAL design changed")
|
||||
gates = cfg["gates"]
|
||||
require(gates["sparse_min_fraction"] == .03 and gates["sparse_max_fraction"] == .09 and
|
||||
gates["minimum_separable_odor_pairs"] == 3 and gates["max_fraction_strict"] == .5 and
|
||||
gates["min_fraction_strict"] == 0 and all(gates[key] is True for key in
|
||||
("same_jaccard_gt_between", "between_euclidean_gt_within", "both_modes_required", "finite_required", "repeat_exact_required")),
|
||||
"Original representation gates or additional integrity gates changed")
|
||||
expected = {(mode, index, seed) for mode in ORIGINAL_MODES for index in range(6) for seed in ORIGINAL_SEEDS}
|
||||
require(len(records) == 60 and {(row["mode"], row["odor_index"], row["seed"]) for row in records} == expected,
|
||||
"CAL inventory must cover exactly 2 modes x 6 odors x 5 seeds")
|
||||
odors = []
|
||||
for index in range(6):
|
||||
ids = {row["odor_id"] for row in records if row["odor_index"] == index}
|
||||
require(len(ids) == 1, "Odor identity changes across seeds/modes")
|
||||
odors.append(next(iter(ids)))
|
||||
require(set(odors) == ORIGINAL_CAL_IDS, "CAL identities differ from the original frozen six")
|
||||
for row in records:
|
||||
require(row["name"] == f"cal-{row['mode']}-{row['odor_index']:02d}-s{row['seed']}", "Unexpected CAL case name")
|
||||
return odors
|
||||
|
||||
|
||||
def summarize(out_json=_DEFAULT_OUTPUT):
|
||||
"""Recompute; default writes exclusively, out_json=None is strictly read-only."""
|
||||
from context import EXP, ROOT, CFG, check
|
||||
target = EXP / "calibration/summary.json" if out_json is _DEFAULT_OUTPUT else (
|
||||
None if out_json is None else Path(out_json))
|
||||
require(target is None or not target.exists(), "Refusing existing calibration summary")
|
||||
failures, runs, run_records, repeats, provenance = [], {}, [], [], []
|
||||
modes, odors = {}, []
|
||||
prerequisites_ok = False
|
||||
encoder = None
|
||||
records = []
|
||||
try:
|
||||
check()
|
||||
records = read_json(EXP / "protocol/calibration-cases.json")
|
||||
odors = validate_inventory(records, CFG)
|
||||
encoder = read_json(EXP / "protocol/encoder-audit.json")
|
||||
require(encoder["panel_count"] == 12 and encoder["cal_count"] == 6 and
|
||||
encoder["seeds"] == list(ORIGINAL_SEEDS) and encoder["requested_before_any_dynamics"] is True and
|
||||
encoder["formal_AB_run"] is False and encoder["heldout_run"] is False, "Invalid frozen encoder scope")
|
||||
# Only static original-panel metadata is read, never heldout trajectories.
|
||||
import pandas as pd
|
||||
original = pd.read_parquet(ROOT / "experiments/phase4l-olfactory-grounding/odor-encoder/frozen-odor-panel.parquet")
|
||||
require(len(original) == 12 and set(original[original.split == "calibration"].odor_id) == ORIGINAL_CAL_IDS,
|
||||
"Original static panel identity mismatch")
|
||||
require(float(np.median(original.mapped_response_sum)) == encoder["median_mapped_response_sum"],
|
||||
"Matched-energy reference is not the original frozen 12-panel median")
|
||||
kc_globals = np.load(EXP / "apl-bundle/kc_globals.npy", allow_pickle=False)
|
||||
require(kc_globals.dtype == np.int32 and kc_globals.shape == (4064,) and
|
||||
bool(np.all(np.diff(kc_globals) > 0)) and bool(np.all((kc_globals >= 0) & (kc_globals < CFG["neurons"]))),
|
||||
"Invalid frozen KC identities")
|
||||
prerequisites_ok = True
|
||||
except Exception as error:
|
||||
failures.append({"stage": "prerequisites", "type": type(error).__name__, "message": str(error)})
|
||||
if prerequisites_ok:
|
||||
for row in records:
|
||||
name = row["name"]
|
||||
first = None
|
||||
try:
|
||||
case = EXP / "cases" / name
|
||||
spec = read_json(case / "spec.json")
|
||||
require(spec["name"] == name and spec["kind"] == "calibration" and spec["steps"] == 1000 and
|
||||
spec["duration_ms"] == 100.0 and spec["dt_ms"] == .1 and spec["training"] is False and
|
||||
all(spec[key] == row[key] for key in ("mode", "odor_id", "odor_index", "seed")),
|
||||
"Case spec differs from fixed CAL inventory")
|
||||
first = read_trial(EXP / "gpu" / (name + "-r0"), case, spec, kc_globals, CFG["neurons"], CFG["edges"])
|
||||
require(first["record"]["requested"] == row["requests"], "Frozen inventory request count mismatch")
|
||||
runs[(row["mode"], row["odor_id"], row["seed"])] = {key: first[key] for key in
|
||||
("kc_counts", "expected_external_events", "external_events")}
|
||||
run_records.append(first["record"])
|
||||
provenance.append({"case": name, "repeat": 0, "state_npz_sha256": file_hash(first["output"] / "state.npz"),
|
||||
"summary_sha256": file_hash(first["output"] / "summary.json")})
|
||||
except Exception as error:
|
||||
failures.append({"stage": "raw_trial", "case": name, "repeat": 0,
|
||||
"type": type(error).__name__, "message": str(error)})
|
||||
if row["seed"] == ORIGINAL_SEEDS[0]:
|
||||
try:
|
||||
require(first is not None, "Baseline unavailable for exact repeat")
|
||||
repeat = read_trial(EXP / "gpu" / (name + "-r1"), case, spec, kc_globals, CFG["neurons"], CFG["edges"])
|
||||
repeat_report = {"case": name, **exact_repeat(first, repeat)}
|
||||
repeats.append(repeat_report)
|
||||
provenance.append({"case": name, "repeat": 1, "state_npz_sha256": file_hash(repeat["output"] / "state.npz"),
|
||||
"summary_sha256": file_hash(repeat["output"] / "summary.json")})
|
||||
require(repeat_report["pass_all"], "Exact repeat differs: " + ", ".join(
|
||||
key for key, value in repeat_report["checks"].items() if not value))
|
||||
except Exception as error:
|
||||
failures.append({"stage": "exact_repeat", "case": name, "repeat": 1,
|
||||
"type": type(error).__name__, "message": str(error)})
|
||||
if len(runs) == 60:
|
||||
for mode in ORIGINAL_MODES:
|
||||
try:
|
||||
modes[mode] = summarize_mode(mode, odors, ORIGINAL_SEEDS, runs, CFG["gates"])
|
||||
except Exception as error:
|
||||
failures.append({"stage": "representation", "mode": mode,
|
||||
"type": type(error).__name__, "message": str(error)})
|
||||
integrity_pass = bool(prerequisites_ok and len(runs) == 60 and not any(row["stage"] in
|
||||
("prerequisites", "raw_trial") for row in failures))
|
||||
repeat_pass = bool(len(repeats) == 12 and all(row["pass_all"] for row in repeats) and
|
||||
not any(row["stage"] == "exact_repeat" for row in failures))
|
||||
representation_pass = bool(len(modes) == 2 and all(row["gate"] for row in modes.values()))
|
||||
result = {"phase": "4AG", "scope": "Original fixed six CAL odors only; not heldout generalization or learning",
|
||||
"pass_all": bool(integrity_pass and repeat_pass and representation_pass),
|
||||
"numerical_integrity_pass": integrity_pass, "exact_repeat_pass": repeat_pass,
|
||||
"original_representation_gate_pass": representation_pass,
|
||||
"expected_base_trials": 60, "valid_base_trials": len(runs), "expected_repeats": 12,
|
||||
"checked_repeats": len(repeats), "seeds": list(ORIGINAL_SEEDS), "duration_ms": 100.0,
|
||||
"odor_ids": odors, "modes": modes, "runs": run_records, "repeats": repeats,
|
||||
"failures": failures, "input_output_provenance": provenance, "encoder_audit": encoder,
|
||||
"metric_source": "phase4l.representation (full original gate), phase4i.pair_metrics",
|
||||
"finite_gate_note": "Added integrity prerequisite; original 4L recorded finite but omitted it from its gate",
|
||||
"repeat_gate_note": "Additional exact repeat requirement; original metrics and numerical thresholds unchanged",
|
||||
"shared_apl_component_oracle_limit": True, "independent_oracle_equivalence_claim": False,
|
||||
"candidate_generated": False, "training": False, "heldout_dynamic_access": False}
|
||||
if target is not None:
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
with target.open("x", encoding="utf-8") as output:
|
||||
json.dump(result, output, ensure_ascii=False, indent=2, allow_nan=False)
|
||||
output.write("\n")
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
print(json.dumps({key: value for key, value in summarize().items() if key in
|
||||
("pass_all", "numerical_integrity_pass", "exact_repeat_pass", "original_representation_gate_pass")}), flush=True)
|
||||
@@ -0,0 +1,18 @@
|
||||
49f53924f758317bfcbd2b53cab9b8b785121eb99f3234a8343c9b483c5701d8 PROTOCOL.md
|
||||
edb9cb2d890471fd07afd88c959ef3540b0cd40af490fac3f26396b5e7f5bf9c calibration.py
|
||||
6e1616b086f667fa4e4b5087f500f8080281977ca9922e7b055acfddd2003bd5 compare.py
|
||||
a8bafb9deec49d918976f2836d977ef9f13c26a6fb7f70e015879e29cf986c9a config.json
|
||||
274eb13ccf38dbbc5c7f14d9d642130eb0f69072d91a090f208c30c27f87dd2b context.py
|
||||
b6e3c67e668a971b544df32d76435ab5e537061b9fc2e8d9dedb5d2a726f5782 finalize.py
|
||||
69f5d893f47dafff4ef7c3186c3df0b9014666b32c8f4e943b57016bec92b18a gpu_apl.py
|
||||
21ceec65fe42581dc55872010e0025d660e80358905f1cf4df1dadf341a3b291 legacy_apl_worker.py
|
||||
357a7d3debb5abc34a8714c0233b456547a154d9d5855d28661b4f01019a80ea oracle_apl.py
|
||||
e9328567753e144fc315491970f7e02af13725b93f35e4d4b97f7c4c48ce3338 prepare.py
|
||||
afbe6d5a37c29bda66d38c470093584130d1852e67881c681662aec3c76b33c7 run_case.py
|
||||
84c681030b0032119b5392d0c3f3c136a9c6b0d133eb548c40791a10bbbb8b63 run_suite.py
|
||||
bab3136eecbc5bb09a985d715844ee845cb72b07930ce376bfb39be33bd1845a test_calibration.py
|
||||
1a233f47943e01bbd845ac1dacdb1eec3f88c270c65a5f75bce1cf679b5de097 test_protocol.py
|
||||
0ed665f686e3499f8543e48f1712ccb4523946971cb7a0e096cf4f6d10d3bcba test_python_entry.py
|
||||
5cfc9a4505da6b8fb4e7d2617cdaedaa8475ed46a11ec9bdf0d4d25b1d22c2d3 validate_components.py
|
||||
0d0e72b2f1348f6648124c8c34f75094f76fc5e1a281fbcca130a5fad2f35293 verify_frozen.py
|
||||
208c00961b9d895d14ac050bc23fbc18db49d0df1fcf5c20d4c3eaa8e15ff7ff manifest.json
|
||||
@@ -0,0 +1,508 @@
|
||||
"""Fail-closed byte-exact audit of the shared-APL closed-loop fixtures.
|
||||
|
||||
Comparison is independent of either backend's stored PASS. No tolerance is
|
||||
used, timing fields are never compared, and reports contain no wall timestamp.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import re
|
||||
import struct
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
N_GLOBAL = 166700
|
||||
STATE_KEYS = ('step', 'neuron_idx', 'counts', 'v', 'g', 'ref')
|
||||
REQUEST_FIELDS = ('tick', 'neuron_idx', 'attempted', 'accepted', 'rejected')
|
||||
HASH_FIELDS = {'tick', 'v_sha256', 'g_sha256', 'ref_sha256'}
|
||||
QUEUE_FIELDS = {'tick', 'count', 'accepted_count', 'ordered_edge_sha256',
|
||||
'accepted_mask_sha256', 'joint_sha256'}
|
||||
BUNDLE_FILES = ('input_matrix.npy', 'output_matrix.npy', 'kc_globals.npy', 'apl_globals.npy')
|
||||
COEFFICIENTS = ('gd', 'c0', 'cg', 'cv', 'threshold', 'reset', 'seed_volt')
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def file_hash(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open('rb') as source:
|
||||
for block in iter(lambda: source.read(8 << 20), b''):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def raw_hash(array, dtype=None):
|
||||
return hashlib.sha256(np.ascontiguousarray(array, dtype=dtype).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def save(path, report):
|
||||
if path is None:
|
||||
return
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open('x', encoding='utf-8') as target:
|
||||
json.dump(report, target, indent=2, sort_keys=True, allow_nan=False)
|
||||
target.write('\n')
|
||||
|
||||
|
||||
def _read_json(path, used):
|
||||
used.add(Path(path))
|
||||
return json.loads(Path(path).read_text(encoding='utf-8'))
|
||||
|
||||
|
||||
def _read_jsonl(path, used):
|
||||
used.add(Path(path))
|
||||
with Path(path).open(encoding='utf-8') as source:
|
||||
return [json.loads(line) for line in source if line.strip()]
|
||||
|
||||
|
||||
def _read_array(path, used):
|
||||
used.add(Path(path))
|
||||
return np.load(path, mmap_mode='r', allow_pickle=False)
|
||||
|
||||
|
||||
def _check(checks, name, condition):
|
||||
checks[name] = bool(condition)
|
||||
|
||||
|
||||
def _bit_mask(a, b):
|
||||
if a.dtype.kind == 'f':
|
||||
unsigned = np.dtype('uint'+str(a.dtype.itemsize*8))
|
||||
return np.ascontiguousarray(a).view(unsigned) != np.ascontiguousarray(b).view(unsigned)
|
||||
return a != b
|
||||
|
||||
|
||||
def _compare_array(name, a, b, checks, differences, dtype=None, shape=None):
|
||||
a, b = np.asarray(a), np.asarray(b)
|
||||
schema = a.shape == b.shape and a.dtype == b.dtype and \
|
||||
(dtype is None or a.dtype == np.dtype(dtype)) and (shape is None or a.shape == shape)
|
||||
_check(checks, name+'.schema', schema)
|
||||
if not schema:
|
||||
differences[name] = {'left_shape': list(a.shape), 'right_shape': list(b.shape),
|
||||
'left_dtype': str(a.dtype), 'right_dtype': str(b.dtype), 'expected_shape': None if shape is None else list(shape),
|
||||
'expected_dtype': None if dtype is None else str(np.dtype(dtype))}
|
||||
_check(checks, name+'.bitwise_exact', False)
|
||||
return False
|
||||
finite = bool(np.isfinite(a).all() and np.isfinite(b).all())
|
||||
_check(checks, name+'.finite', finite)
|
||||
mask = _bit_mask(a, b)
|
||||
exact = not bool(np.any(mask))
|
||||
_check(checks, name+'.bitwise_exact', exact)
|
||||
if not exact or not finite:
|
||||
flat = np.flatnonzero(mask.ravel())
|
||||
detail = {'mismatching_elements': int(len(flat)),
|
||||
'first_positions': [list(map(int, np.unravel_index(int(index), a.shape))) for index in flat[:5]],
|
||||
'left_nonfinite': int(np.count_nonzero(~np.isfinite(a))),
|
||||
'right_nonfinite': int(np.count_nonzero(~np.isfinite(b)))}
|
||||
if finite and a.size:
|
||||
error = np.abs(a.astype(np.float64)-b.astype(np.float64))
|
||||
detail.update(max_abs_error=float(error.max()), mean_abs_error=float(error.mean()))
|
||||
if a.dtype.kind == 'f' and len(flat):
|
||||
unsigned = np.dtype('uint'+str(a.dtype.itemsize*8))
|
||||
av = np.ascontiguousarray(a).view(unsigned).ravel()
|
||||
bv = np.ascontiguousarray(b).view(unsigned).ravel()
|
||||
detail['first_bit_patterns'] = [{'left': hex(int(av[index])), 'right': hex(int(bv[index]))} for index in flat[:5]]
|
||||
differences[name] = detail
|
||||
return exact and finite
|
||||
|
||||
|
||||
def _sha(value):
|
||||
return isinstance(value, str) and re.fullmatch('[0-9a-f]{64}', value) is not None
|
||||
|
||||
|
||||
def _read_requests(path, steps, used):
|
||||
used.add(Path(path))
|
||||
with Path(path).open(encoding='utf-8', newline='') as source:
|
||||
reader = csv.DictReader(source)
|
||||
require(tuple(reader.fieldnames or ()) == REQUEST_FIELDS, 'Request CSV schema mismatch: '+str(path))
|
||||
rows = [{key: int(row[key]) for key in REQUEST_FIELDS} for row in reader]
|
||||
previous = (-1, -1)
|
||||
for row in rows:
|
||||
key = row['tick'], row['neuron_idx']
|
||||
require(previous < key and 0 <= key[0] < steps and 0 <= key[1] < N_GLOBAL,
|
||||
'Request rows must be sorted, unique, and in range: '+str(path))
|
||||
require(row['attempted'] == 1 and row['accepted'] in (0, 1) and row['rejected'] == 1-row['accepted'],
|
||||
'Request accounting is incomplete: '+str(path))
|
||||
previous = key
|
||||
return rows
|
||||
|
||||
|
||||
def _load_run(directory, role, n, edges, steps, record_history, name, used, checks):
|
||||
directory = Path(directory)
|
||||
summary = _read_json(directory/'summary.json', used)
|
||||
_check(checks, role+'.summary_complete', summary.get('complete') is True and summary.get('pass_all') is True and
|
||||
summary.get('steps') == steps and summary.get('completed_ticks') == steps and
|
||||
summary.get('nodes') == n and summary.get('edges') == edges and summary.get('precision') == 'float64')
|
||||
is_cpu = role == 'cpu'
|
||||
_check(checks, role+'.case_name', summary.get('name' if is_cpu else 'case') == name)
|
||||
_check(checks, role+'.history_mode', summary.get('history_recorded' if is_cpu else 'record_history') is record_history)
|
||||
_check(checks, role+'.apl_precision', summary.get('APL_precision' if is_cpu else 'apl_precision') == 'float32')
|
||||
_check(checks, role+'.no_error_marker', not (directory/'error.json').exists())
|
||||
used.add(directory/'state.npz')
|
||||
with np.load(directory/'state.npz', allow_pickle=False) as source:
|
||||
require(set(source.files) == set(STATE_KEYS), 'State NPZ keys changed: '+str(directory))
|
||||
state = {key: source[key].copy() for key in STATE_KEYS}
|
||||
for key, dtype in (('v', 'float64'), ('g', 'float64'), ('ref', 'int32'), ('counts', 'int32')):
|
||||
require(state[key].shape == (n,) and state[key].dtype == np.dtype(dtype), 'Invalid final '+key+': '+str(directory))
|
||||
require(state['step'].dtype == state['neuron_idx'].dtype == np.int32 and state['step'].ndim == 1 and
|
||||
state['step'].shape == state['neuron_idx'].shape, 'Invalid sparse event schema: '+str(directory))
|
||||
require(bool(np.all((state['step'] >= 0) & (state['step'] < steps))) and
|
||||
bool(np.all((state['neuron_idx'] >= 0) & (state['neuron_idx'] < N_GLOBAL))), 'Sparse event range error')
|
||||
event_keys = state['step'].astype(np.int64)*N_GLOBAL+state['neuron_idx']
|
||||
_check(checks, role+'.events_sorted_unique', len(event_keys) < 2 or np.all(np.diff(event_keys) > 0))
|
||||
_check(checks, role+'.count_nonnegative_total', np.all(state['counts'] >= 0) and int(state['counts'].sum()) == len(event_keys))
|
||||
_check(checks, role+'.ref_range', np.all((state['ref'] >= 0) & (state['ref'] <= 22)))
|
||||
_check(checks, role+'.reported_events', summary.get('spikes' if is_cpu else 'total_spikes') == len(event_keys))
|
||||
hashes = _read_jsonl(directory/'tick-state-hashes.jsonl', used)
|
||||
require(len(hashes) == steps and [row.get('tick') for row in hashes] == list(range(steps)), 'Incomplete tick hash coverage')
|
||||
require(all(set(row) == HASH_FIELDS and all(_sha(row[key]) for key in HASH_FIELDS-{'tick'}) for row in hashes),
|
||||
'Invalid tick state hash schema')
|
||||
queues = _read_jsonl(directory/'queue.jsonl', used)
|
||||
require(len(queues) == steps and [row.get('tick') for row in queues] == list(range(steps)), 'Incomplete queue coverage')
|
||||
require(all(set(row) == QUEUE_FIELDS and type(row['count']) is int and type(row['accepted_count']) is int and
|
||||
0 <= row['accepted_count'] <= row['count'] <= edges and
|
||||
all(_sha(row[key]) for key in QUEUE_FIELDS-{'tick', 'count', 'accepted_count'}) for row in queues),
|
||||
'Invalid queue row schema/count/hash')
|
||||
for key in ('v', 'g', 'ref'):
|
||||
_check(checks, role+'.last_hash_'+key, hashes[-1][key+'_sha256'] == raw_hash(state[key], '<i4' if key == 'ref' else '<f8'))
|
||||
requests = _read_requests(directory/'requests.csv', steps, used)
|
||||
requested, accepted = len(requests), sum(row['accepted'] for row in requests)
|
||||
_check(checks, role+'.request_summary', summary.get('requests' if is_cpu else 'requested') == requested and
|
||||
summary.get('applicable_requests' if is_cpu else 'accepted_requests') == accepted and
|
||||
summary.get('ref_rejected_requests' if is_cpu else 'rejected_requests') == requested-accepted)
|
||||
apl = {field: _read_array(directory/('apl-'+field+'.float32.npy'), used) for field in ('drive', 'state', 'inhibition')}
|
||||
for field, width in (('drive', 2), ('state', 2), ('inhibition', 4064)):
|
||||
require(apl[field].dtype == np.float32 and apl[field].shape == (steps, width), 'Invalid APL '+field+' history schema')
|
||||
_check(checks, role+'.apl_positive_'+field, np.all(apl[field] >= 0))
|
||||
histories, pre = {}, {}
|
||||
for field, dtype in (('v', 'float64'), ('g', 'float64'), ('ref', 'int32')):
|
||||
files = (directory/(field+'-history.'+dtype+'.npy'), directory/('preAPL-'+field+'.'+dtype+'.npy'))
|
||||
if record_history:
|
||||
histories[field], pre[field] = (_read_array(path, used) for path in files)
|
||||
require(all(array.dtype == np.dtype(dtype) and array.shape == (steps, n) for array in (histories[field], pre[field])),
|
||||
'Small pre/post history schema mismatch')
|
||||
_check(checks, role+'.history_hashes_'+field, all(row[field+'_sha256'] == raw_hash(histories[field][tick])
|
||||
for tick, row in enumerate(hashes)))
|
||||
_check(checks, role+'.history_final_'+field, not np.any(_bit_mask(histories[field][-1], state[field])))
|
||||
else:
|
||||
_check(checks, role+'.no_dense_'+field, not any(path.exists() for path in files))
|
||||
if is_cpu:
|
||||
component = _read_json(directory/'apl-worker-handshake.json', used)['handshake']
|
||||
else:
|
||||
component = _read_json(directory/'apl-component.json', used)
|
||||
_check(checks, role+'.component_summary_consistent', summary.get('apl_component') == component)
|
||||
return dict(directory=directory, summary=summary, state=state, hashes=hashes, queues=queues,
|
||||
requests=requests, apl=apl, histories=histories, pre=pre, component=component)
|
||||
|
||||
|
||||
def _pair_outputs(left, right, checks, differences, n, steps, record_history):
|
||||
for field in STATE_KEYS:
|
||||
shape = None if field in ('step', 'neuron_idx') else (n,)
|
||||
dtype = 'float64' if field in ('v', 'g') else 'int32'
|
||||
_compare_array('state.'+field, left['state'][field], right['state'][field], checks, differences, dtype, shape)
|
||||
for field, width in (('drive', 2), ('state', 2), ('inhibition', 4064)):
|
||||
_compare_array('apl.'+field, left['apl'][field], right['apl'][field], checks, differences, 'float32', (steps, width))
|
||||
for field in ('hashes', 'queues', 'requests', 'component'):
|
||||
same = left[field] == right[field]
|
||||
_check(checks, field+'.parsed_exact', same)
|
||||
if not same:
|
||||
if isinstance(left[field], list):
|
||||
first = next((i for i, (a, b) in enumerate(zip(left[field], right[field])) if a != b), None)
|
||||
differences[field] = {'left_count': len(left[field]), 'right_count': len(right[field]),
|
||||
'first_differing_row': first,
|
||||
'left_row': None if first is None else left[field][first],
|
||||
'right_row': None if first is None else right[field][first]}
|
||||
else:
|
||||
differences[field] = {'left': left[field], 'right': right[field]}
|
||||
if record_history:
|
||||
for stage in ('histories', 'pre'):
|
||||
for field, dtype in (('v', 'float64'), ('g', 'float64'), ('ref', 'int32')):
|
||||
_compare_array(stage+'.'+field, left[stage][field], right[stage][field], checks, differences, dtype, (steps, n))
|
||||
|
||||
|
||||
def _queue_digest(tick, ids, accepted):
|
||||
ids = np.ascontiguousarray(ids, dtype='<i8').tobytes()
|
||||
mask = np.ascontiguousarray(accepted, dtype=np.uint8).tobytes()
|
||||
return {'tick': tick, 'count': len(mask), 'accepted_count': int(np.count_nonzero(accepted)),
|
||||
'ordered_edge_sha256': hashlib.sha256(ids).hexdigest(),
|
||||
'accepted_mask_sha256': hashlib.sha256(mask).hexdigest(),
|
||||
'joint_sha256': hashlib.sha256(b'phase4af-queue-v1\0'+struct.pack('<qq', tick, len(mask))+ids+mask).hexdigest()}
|
||||
|
||||
|
||||
def _independent_event_accounting(run, role, graph, requested, steps, checks, differences):
|
||||
nodes, ptr = graph['nodes'], graph['ptr']
|
||||
n = len(nodes)
|
||||
state = run['state']
|
||||
local = np.searchsorted(nodes, state['neuron_idx'])
|
||||
require(np.all(local < n) and np.array_equal(nodes[local], state['neuron_idx']), 'Actual spike outside graph identity')
|
||||
_check(checks, role+'.events_per_neuron_counts', np.array_equal(np.bincount(local, minlength=n), state['counts']))
|
||||
actual_request_pairs = [(row['tick'], row['neuron_idx']) for row in run['requests']]
|
||||
_check(checks, role+'.requests_match_frozen_input', actual_request_pairs == requested)
|
||||
require(actual_request_pairs == requested, 'Actual request population/time differs from frozen input')
|
||||
bounds = np.searchsorted(state['step'], np.arange(steps+1))
|
||||
request_ticks = np.asarray([row['tick'] for row in run['requests']], np.int32)
|
||||
rbounds = np.searchsorted(request_ticks, np.arange(steps+1))
|
||||
request_nodes = np.asarray([row['neuron_idx'] for row in run['requests']], np.int32)
|
||||
request_local = np.searchsorted(nodes, request_nodes)
|
||||
require(np.all(request_local < n) and np.array_equal(nodes[request_local], request_nodes), 'Request outside graph')
|
||||
last = np.full(n, -1_000_000_000, np.int64)
|
||||
degrees = np.diff(ptr)
|
||||
expected_counts = np.zeros(steps, np.int64)
|
||||
arrival_tick = state['step'].astype(np.int64)+18
|
||||
valid = arrival_tick < steps
|
||||
np.add.at(expected_counts, arrival_tick[valid], degrees[local[valid]])
|
||||
_check(checks, role+'.delayed_all_edges_count', np.array_equal(expected_counts, [row['count'] for row in run['queues']]))
|
||||
request_ok, delivered_ok, ref_ok, queue_ok, reset_ok = True, True, True, True, True
|
||||
first = {}
|
||||
for tick in range(steps):
|
||||
fired = local[bounds[tick]:bounds[tick+1]]
|
||||
lo, hi = rbounds[tick:tick+2]
|
||||
targets = request_local[lo:hi]
|
||||
accepted = (tick-last[targets]) >= 22
|
||||
for index, ok in zip(range(lo, hi), accepted):
|
||||
row = run['requests'][index]
|
||||
if row['accepted'] != int(ok) or row['rejected'] != int(not ok):
|
||||
request_ok = False
|
||||
first.setdefault('request_ref_mismatch', {'tick': tick, 'neuron_idx': row['neuron_idx']})
|
||||
if not np.isin(targets[accepted], fired).all() or np.isin(targets[~accepted], fired).any():
|
||||
delivered_ok = False
|
||||
first.setdefault('request_delivery_mismatch', tick)
|
||||
last[fired] = tick
|
||||
ref = np.maximum(22-(tick-last), 0).astype(np.int32)
|
||||
if raw_hash(ref, '<i4') != run['hashes'][tick]['ref_sha256']:
|
||||
ref_ok = False
|
||||
first.setdefault('ref_hash_mismatch', tick)
|
||||
if run['pre']:
|
||||
if len(fired) and not np.array_equal(np.ascontiguousarray(run['pre']['g'][tick, fired]).view(np.uint64), np.zeros(len(fired), np.uint64)):
|
||||
reset_ok = False
|
||||
first.setdefault('pre_apl_spike_reset_mismatch', tick)
|
||||
due = local[bounds[tick-18]:bounds[tick-17]] if tick >= 18 else np.empty(0, np.int64)
|
||||
chunks = [np.arange(ptr[source], ptr[source+1], dtype=np.int64) for source in due]
|
||||
indices = np.concatenate(chunks) if chunks else np.empty(0, np.int64)
|
||||
expected_queue = _queue_digest(tick, graph['edge_id'][indices], (ref[graph['dst'][indices]] == 0).astype(np.uint8))
|
||||
if expected_queue != run['queues'][tick]:
|
||||
queue_ok = False
|
||||
first.setdefault('queue_reconstruction_mismatch', tick)
|
||||
_check(checks, role+'.request_refractory_gate', request_ok)
|
||||
_check(checks, role+'.accepted_spikes_rejected_absent', delivered_ok)
|
||||
_check(checks, role+'.all_ref_hashes_from_actual_events', ref_ok)
|
||||
if run['pre']:
|
||||
_check(checks, role+'.small_queue_independent_reconstruction', queue_ok)
|
||||
_check(checks, role+'.spike_g_reset_before_apl', reset_ok)
|
||||
if first:
|
||||
differences[role+'.independent_accounting'] = first
|
||||
|
||||
|
||||
def _check_apl_application(run, role, nodes, kc, checks, differences):
|
||||
if not run['pre']:
|
||||
return
|
||||
common, local_kc, bundle_kc = np.intersect1d(nodes, kc, assume_unique=True, return_indices=True)
|
||||
_compare_array(role+'.apl_keeps_voltage', run['pre']['v'], run['histories']['v'], checks, differences, 'float64')
|
||||
_compare_array(role+'.apl_keeps_ref', run['pre']['ref'], run['histories']['ref'], checks, differences, 'int32')
|
||||
expected = np.array(run['pre']['g'], copy=True)
|
||||
delta = run['apl']['inhibition'][:, bundle_kc].astype(np.float64)*np.float64(.001)
|
||||
expected[:, local_kc] = expected[:, local_kc]-delta
|
||||
_compare_array(role+'.apl_unconditional_g_write', expected, run['histories']['g'], checks, differences, 'float64')
|
||||
|
||||
|
||||
def _finish(report, checks, differences, used, out_json):
|
||||
report['checks'] = {name: bool(value) for name, value in checks.items()}
|
||||
report['differences'] = differences
|
||||
report['pass_all'] = bool(checks) and all(checks.values()) and 'error' not in report
|
||||
report['failed_checks'] = sorted(name for name, value in checks.items() if not value)
|
||||
report['evidence_sha256'] = {str(path): file_hash(path) for path in sorted(used) if path.is_file()}
|
||||
save(out_json, report)
|
||||
return report
|
||||
|
||||
|
||||
def compare(case_dir, gpu_dir, cpu_dir, out_json=None):
|
||||
"""Strict independent closed-loop comparison; out_json=None is read-only."""
|
||||
case_dir, gpu_dir, cpu_dir = Path(case_dir), Path(gpu_dir), Path(cpu_dir)
|
||||
if out_json is not None:
|
||||
require(not Path(out_json).exists(), 'Refusing existing comparison report')
|
||||
checks, differences, used = {}, {}, set()
|
||||
report = {'kind': 'shared-APL-component closed-loop byte-exact comparison', 'pass_all': False,
|
||||
'case_dir': str(case_dir), 'gpu_dir': str(gpu_dir), 'cpu_dir': str(cpu_dir),
|
||||
'tolerance': None, 'signed_zero_included': True, 'timing_fields_compared': False,
|
||||
'independent_CPU_all_components_claim': False}
|
||||
try:
|
||||
spec = _read_json(case_dir/'spec.json', used)
|
||||
steps, history = spec['steps'], spec['record_history']
|
||||
require(type(steps) is int and steps > 0 and type(history) is bool and spec['dt_ms'] == .1,
|
||||
'Invalid fixed case clock/history schema')
|
||||
graph_dir, bundle_dir = Path(spec['graph_dir']), Path(spec['apl_bundle_dir'])
|
||||
graph = {field: _read_array(graph_dir/(field+'.npy'), used) for field in
|
||||
('nodes', 'edge_id', 'src', 'dst', 'ptr', 'signed_raw', 'weights')}
|
||||
nodes = graph['nodes']
|
||||
n, edges = len(nodes), len(graph['edge_id'])
|
||||
require(n > 0 and edges > 0 and nodes.dtype == np.int32 and nodes.ndim == 1 and np.all((nodes >= 0) & (nodes < N_GLOBAL)) and
|
||||
np.all(np.diff(nodes) > 0), 'Invalid canonical node identities')
|
||||
require(graph['ptr'].dtype == np.int64 and graph['ptr'].shape == (n+1,) and graph['ptr'][0] == 0 and
|
||||
graph['ptr'][-1] == edges and np.all(np.diff(graph['ptr']) >= 0), 'Invalid CSR')
|
||||
for key, dtype in (('edge_id', 'int64'), ('src', 'int32'), ('dst', 'int32'), ('signed_raw', 'int64'), ('weights', 'float64')):
|
||||
require(graph[key].dtype == np.dtype(dtype) and graph[key].shape == (edges,), 'Invalid graph '+key)
|
||||
require(np.array_equal(graph['src'], np.repeat(np.arange(n, dtype=np.int32), np.diff(graph['ptr']))) and
|
||||
np.all((graph['dst'] >= 0) & (graph['dst'] < n)), 'Invalid source/destination row mapping')
|
||||
require(history == (n < 512) and (n < 512 or n == N_GLOBAL), 'Invalid small/full history declaration')
|
||||
require(np.all(graph['edge_id'] >= 0) and np.all(np.diff(graph['edge_id']) > 0),
|
||||
'Original canonical edge IDs must be unique and increasing')
|
||||
if n == N_GLOBAL:
|
||||
require(edges == 25582938 and np.array_equal(nodes, np.arange(n, dtype=np.int32)) and
|
||||
np.array_equal(graph['edge_id'], np.arange(edges, dtype=np.int64)), 'Full canonical graph identity changed')
|
||||
bundle = {name: _read_array(bundle_dir/name, used) for name in BUNDLE_FILES}
|
||||
kc, apl_ids = bundle['kc_globals.npy'], bundle['apl_globals.npy']
|
||||
require(kc.dtype == apl_ids.dtype == np.int32 and kc.shape == (4064,) and apl_ids.shape == (2,), 'Invalid APL/KC identities')
|
||||
require(np.all((kc >= 0) & (kc < N_GLOBAL)) and np.all(np.diff(kc) > 0) and
|
||||
np.all((apl_ids >= 0) & (apl_ids < N_GLOBAL)) and np.all(np.diff(apl_ids) > 0) and
|
||||
not np.intersect1d(kc, apl_ids).size, 'APL/KC identities overlap or are invalid')
|
||||
for key, shape in (('input_matrix.npy', (2, N_GLOBAL)), ('output_matrix.npy', (2, 4064))):
|
||||
require(bundle[key].dtype == np.float32 and bundle[key].shape == shape and
|
||||
np.isfinite(bundle[key]).all() and np.all(bundle[key] >= 0), 'Invalid normalized APL matrix')
|
||||
gpu = _load_run(gpu_dir, 'gpu', n, edges, steps, history, spec['name'], used, checks)
|
||||
cpu = _load_run(cpu_dir, 'cpu', n, edges, steps, history, spec['name'], used, checks)
|
||||
_pair_outputs(gpu, cpu, checks, differences, n, steps, history)
|
||||
used.add(case_dir/'requests.npz')
|
||||
with np.load(case_dir/'requests.npz', allow_pickle=False) as source:
|
||||
require(set(source.files) == {'tick', 'neuron_idx'} and source['tick'].dtype == source['neuron_idx'].dtype == np.int32,
|
||||
'Fixed request NPZ schema changed')
|
||||
require(source['tick'].shape == source['neuron_idx'].shape and source['tick'].ndim == 1, 'Invalid sparse request shape')
|
||||
requested = list(zip(map(int, source['tick']), map(int, source['neuron_idx'])))
|
||||
for role, run in (('gpu', gpu), ('cpu', cpu)):
|
||||
_independent_event_accounting(run, role, graph, requested, steps, checks, differences)
|
||||
_check_apl_application(run, role, nodes, kc, checks, differences)
|
||||
|
||||
# Independently derive the effective clone. Original graph weights stay read-only.
|
||||
cpu_weights = _read_array(cpu_dir/'weights.npy', used)
|
||||
require(cpu_weights.dtype == np.float64 and cpu_weights.shape == (edges,), 'CPU effective weight schema mismatch')
|
||||
expected_hash, base_hash, mask_hash = hashlib.sha256(), hashlib.sha256(), hashlib.sha256()
|
||||
weight_ok, mask_count = True, 0
|
||||
for start in range(0, edges, 1 << 20):
|
||||
stop = min(edges, start+(1 << 20))
|
||||
original = np.asarray(graph['weights'][start:stop])
|
||||
require(np.isfinite(original).all(), 'Nonfinite original weights')
|
||||
raw_expected = graph['signed_raw'][start:stop]*(.275*.001)
|
||||
require(not np.any(_bit_mask(original, raw_expected)), 'Original SI weights differ from unaltered signed contacts')
|
||||
mask = np.isin(nodes[graph['src'][start:stop]], apl_ids)
|
||||
expected = original.copy()
|
||||
expected[mask] = np.float64(0)
|
||||
mask_count += int(mask.sum())
|
||||
expected_hash.update(np.ascontiguousarray(expected, dtype='<f8').tobytes())
|
||||
base_hash.update(np.ascontiguousarray(original, dtype='<f8').tobytes())
|
||||
mask_hash.update(np.ascontiguousarray(graph['edge_id'][start:stop][mask], dtype='<i8').tobytes())
|
||||
weight_ok &= not np.any(_bit_mask(expected, cpu_weights[start:stop]))
|
||||
expected_sha = expected_hash.hexdigest()
|
||||
_check(checks, 'cpu.effective_weights_independent_bytes', weight_ok)
|
||||
for role, run in (('gpu', gpu), ('cpu', cpu)):
|
||||
_check(checks, role+'.effective_weights_raw_sha', run['summary'].get('effective_weights_raw_sha256') == expected_sha)
|
||||
_check(checks, role+'.apl_fast_edge_count', run['summary'].get('apl_fast_edges_zeroed') == mask_count)
|
||||
_check(checks, role+'.kc_target_count', run['summary'].get('kc_targets_present') == len(np.intersect1d(nodes, kc)))
|
||||
mask_audit = _read_json(cpu_dir/'apl-mask-audit.json', used)
|
||||
_check(checks, 'cpu.mask_provenance', mask_audit['count'] == mask_count and
|
||||
mask_audit['ordered_original_edge_id_sha256'] == mask_hash.hexdigest() and
|
||||
mask_audit['base_weight_raw_bytes_sha256'] == base_hash.hexdigest() and
|
||||
mask_audit['effective_weights_raw_sha256'] == expected_sha and mask_audit['base_graph_files_modified'] is False)
|
||||
|
||||
case_hashes = {name: file_hash(case_dir/name) for name in ('spec.json', 'requests.npz')}
|
||||
if (case_dir/'arrays.npz').exists():
|
||||
case_hashes['arrays.npz'] = file_hash(case_dir/'arrays.npz')
|
||||
used.add(case_dir/'arrays.npz')
|
||||
baseline_coeff = _read_json(Path(spec['coefficients_path']), used)
|
||||
actual_coeff = _read_json(cpu_dir/'coefficients.json', used)
|
||||
_check(checks, 'cpu.actual_author_coefficients_exact', all(float(actual_coeff[key]).hex() == float(baseline_coeff[key]).hex()
|
||||
for key in COEFFICIENTS))
|
||||
audit = _read_json(cpu_dir/'input-audit.json', used)
|
||||
expected_graph_hashes = {name: file_hash(graph_dir/name) for name in
|
||||
tuple(field+'.npy' for field in graph)+('completeness.csv', 'connectivity.parquet')}
|
||||
bundle_hashes = {name: file_hash(bundle_dir/name) for name in BUNDLE_FILES}
|
||||
_check(checks, 'cpu.read_actual_case', audit['spec'] == spec and audit['case_hashes'] == case_hashes)
|
||||
_check(checks, 'cpu.graph_hashes', audit['graph_hashes'] == expected_graph_hashes)
|
||||
_check(checks, 'cpu.bundle_hashes', audit['bundle_hashes'] == bundle_hashes)
|
||||
_check(checks, 'cpu.no_candidate_trace_input', audit.get('candidate_trace_used_as_input') is False and
|
||||
cpu['summary'].get('candidate_trace_used_as_input') is False)
|
||||
_check(checks, 'cpu.worker_source_hash', audit['legacy_apl_worker_sha256'] == file_hash(spec['legacy_apl_worker']))
|
||||
_check(checks, 'cpu.coefficient_input_hash', audit['coefficients_sha256'] == file_hash(spec['coefficients_path']))
|
||||
gpu_hashes = dict(case_hashes, **{'base_weights.npy': file_hash(graph_dir/'weights.npy'),
|
||||
'coefficients.json': file_hash(spec['coefficients_path'])})
|
||||
_check(checks, 'gpu.input_hashes', gpu['summary']['input_sha256'] == gpu_hashes)
|
||||
for role, run in (('gpu', gpu), ('cpu', cpu)):
|
||||
component = run['component']
|
||||
_check(checks, role+'.actual_component_bundle_bytes', component['files_sha256'] == bundle_hashes and
|
||||
component['array_sha256'] == {name: raw_hash(value) for name, value in bundle.items()})
|
||||
_check(checks, role+'.component_dimensions_clock', component.get('ok') is True and
|
||||
component['protocol'] == 'phase4ag-apl-v1' and component['n_neurons'] == N_GLOBAL and
|
||||
component['n_apl'] == 2 and component['n_kc'] == 4064 and component['dt_ms'] == .1)
|
||||
decay = np.float32(math.exp(-.1/20.0))
|
||||
gain = np.float32(.01)
|
||||
_check(checks, role+'.frozen_apl_constants', component['decay_f32_bits'] == f'{int(decay.view(np.uint32)):08x}' and
|
||||
component['gain_f32_bits'] == f'{int(gain.view(np.uint32)):08x}')
|
||||
env = component['environment']
|
||||
_check(checks, role+'.component_precision_environment', env['cuda_matmul_allow_tf32'] is False and
|
||||
env['float32_matmul_precision'] == 'highest' and env['precision_configuration_changed_by_component'] is False and
|
||||
'4090' in env['device_name'])
|
||||
environment = _read_json(cpu_dir/'environment.json', used)
|
||||
_check(checks, 'cpu.pinned_environment', environment['packages']['brian2'] == '2.5.1' and
|
||||
environment['packages']['numpy'] == '1.22.3' and environment['packages']['sympy'] == '1.12' and
|
||||
environment['Torch_imported_into_Brian'] is False and environment['author_precision'] == 'float64')
|
||||
queue_audit = _read_json(cpu_dir/'queue-audit.json', used)
|
||||
_check(checks, 'cpu.queue_observer_coverage', queue_audit['peek_calls'] == queue_audit['advance_calls'] == steps and
|
||||
queue_audit['same_peek_object_returned'] is True and queue_audit['zero_weight_and_refractory_rejected_edges_retained'] is True)
|
||||
_check(checks, 'cpu.sidecar_complete', cpu['summary']['APL_worker_exit_status'] == 0 and cpu['summary']['APL_worker_ticks'] == steps)
|
||||
report.update(case=spec['name'], steps=steps, nodes=n, edges=edges, record_history=history,
|
||||
total_spikes=len(gpu['state']['step']), active_neurons=int(np.count_nonzero(gpu['state']['counts'])),
|
||||
requested=len(requested), accepted_requests=sum(row['accepted'] for row in gpu['requests']),
|
||||
rejected_requests=sum(row['rejected'] for row in gpu['requests']),
|
||||
arriving_edges=sum(row['count'] for row in gpu['queues']),
|
||||
accepted_arriving_edges=sum(row['accepted_count'] for row in gpu['queues']),
|
||||
apl_fast_edges_zeroed=mask_count, effective_weights_raw_sha256=expected_sha,
|
||||
component_environment=gpu['component']['environment'],
|
||||
limits=['APL Torch primitives are shared; original AST component validation is a separate required gate.',
|
||||
'Small cases preserve and independently hash all pre/post states; full cases compare all-tick instrumented state hashes without saving dense whole-brain histories.',
|
||||
'For full cases intermediate finite checks are performed by the instrumented backends; this file independently inspects final states and all APL traces.',
|
||||
'This is exact agreement for the declared trajectory, not physiological validity or learning.'])
|
||||
except Exception as error:
|
||||
report['error'] = {'type': type(error).__name__, 'message': str(error)}
|
||||
return _finish(report, checks, differences, used, out_json)
|
||||
|
||||
|
||||
def compare_gpu_repeats(first, second, out_json=None):
|
||||
"""Repeat equality only; the first run's independent oracle gate is separate."""
|
||||
first, second = Path(first), Path(second)
|
||||
if out_json is not None:
|
||||
require(not Path(out_json).exists(), 'Refusing existing repeat report')
|
||||
checks, differences, used = {}, {}, set()
|
||||
report = {'kind': 'fresh-GPU-run exact repeat comparison', 'first': str(first), 'second': str(second),
|
||||
'pass_all': False, 'tolerance': None, 'signed_zero_included': True, 'timing_fields_compared': False,
|
||||
'scope': 'Scientific outputs, recording integrity and input hashes; independent case/graph audit belongs to compare()'}
|
||||
try:
|
||||
summary = _read_json(first/'summary.json', used)
|
||||
n, edges, steps, name, history = (summary[key] for key in ('nodes', 'edges', 'steps', 'case', 'record_history'))
|
||||
require(type(n) is int and type(edges) is int and type(steps) is int and n > 0 and edges > 0 and steps > 0 and
|
||||
type(history) is bool and history == (n < 512), 'Invalid repeat dimensions')
|
||||
left = _load_run(first, 'first_gpu', n, edges, steps, history, name, used, checks)
|
||||
right = _load_run(second, 'second_gpu', n, edges, steps, history, name, used, checks)
|
||||
_pair_outputs(left, right, checks, differences, n, steps, history)
|
||||
stable = ('case', 'complete', 'pass_all', 'steps', 'completed_ticks', 'nodes', 'edges', 'precision',
|
||||
'apl_precision', 'state_units', 'total_spikes', 'requested', 'accepted_requests', 'rejected_requests',
|
||||
'arriving_edges', 'accepted_arriving_edges', 'counts_match_sparse_events', 'all_finite', 'record_history',
|
||||
'apl_fast_edges_zeroed', 'kc_targets_present', 'effective_weights_raw_sha256', 'apl_schedule',
|
||||
'apl_si_conversion', 'build', 'sealed_ae_backend_sha256', 'learning', 'input_sha256')
|
||||
for key in stable:
|
||||
_check(checks, 'summary.'+key, key in left['summary'] and key in right['summary'] and
|
||||
left['summary'][key] == right['summary'][key])
|
||||
if not checks['summary.'+key]:
|
||||
differences['summary.'+key] = {'first': left['summary'].get(key), 'second': right['summary'].get(key)}
|
||||
report.update(case=name, steps=steps, nodes=n, edges=edges, record_history=history,
|
||||
total_spikes=len(left['state']['step']),
|
||||
explicit_exclusions=['wall_seconds', 'diagnostic_simulation_seconds', 'build_or_cache_seconds',
|
||||
'peak_allocated_vram_bytes', 'peak_reserved_vram_bytes'])
|
||||
except Exception as error:
|
||||
report['error'] = {'type': type(error).__name__, 'message': str(error)}
|
||||
return _finish(report, checks, differences, used, out_json)
|
||||
@@ -0,0 +1,23 @@
|
||||
{
|
||||
"phase": "4AG-r2",
|
||||
"name": "author-apl-lab-v2",
|
||||
"execution_revision": "Preserve declared venv Python entry rather than resolving its symlink to system Python. No dynamics or inputs changed.",
|
||||
"human_authorization": "允许独立新模型分支测试动力学假设",
|
||||
"gpu_uuid": "GPU-79f3a85d-adce-7833-7003-6488140ca596",
|
||||
"neurons": 166700,
|
||||
"edges": 25582938,
|
||||
"raw_weight_sum": 124177617,
|
||||
"precision": "float64",
|
||||
"apl_precision": "float32",
|
||||
"apl": {"tau_ms": 20.0, "gain": 0.01, "normalization": 1.0},
|
||||
"hypothesis": "Author LIF plus unchanged legacy graded APL operator at end, after reset, unconditional KC g write. Mixed precision: h32 promoted to float64, then multiplied by float64 .001 SI, then subtracted. Not original author's APL model.",
|
||||
"fixtures": ["zero", "pn_pulse", "kc_ref_boundary", "two_sources", "arrival_collision", "natural_request", "apl_only", "high_fanin"],
|
||||
"fixture_repeats": 3,
|
||||
"calibration": {"split": "calibration", "odor_count": 6, "modes": ["physiological", "matched_energy"], "seeds": [51101,51102,51103,51104,51105], "duration_ms": 100.0, "median_scope": "original frozen 12-odor panel", "full_oracle_probes": ["cal-physiological-00-s51101", "cal-matched_energy-00-s51101"], "repeat_first_seed_each_odor": true},
|
||||
"gates": {"sparse_min_fraction": 0.03, "sparse_max_fraction": 0.09, "max_fraction_strict": 0.5, "min_fraction_strict": 0.0, "minimum_separable_odor_pairs": 3, "same_jaccard_gt_between": true, "between_euclidean_gt_within": true, "both_modes_required": true, "finite_required": true, "repeat_exact_required": true},
|
||||
"training": false,
|
||||
"heldout_dynamic_access": false,
|
||||
"candidate_generation": false,
|
||||
"output_budget_bytes": 16106127360,
|
||||
"scope": "Semantic compatibility and fixed old CAL only. Shared-APL-component closed-loop Brian oracle plus independent arithmetic audit; never claim all-component independent CPU equivalence or learning success. No parameter/input/site/window selection from results."
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
import hashlib
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
ROOT=Path('/data/digital-drosophila')
|
||||
MODEL=Path(__file__).resolve().parent
|
||||
EXP=ROOT/'experiments/phase4ag-author-apl-r2'
|
||||
AF_MODEL=ROOT/'models/male-cns-lif/author-fullgraph-lab-v1'
|
||||
AF_EXP=ROOT/'experiments/phase4af-author-fullgraph'
|
||||
AD_EXP=ROOT/'experiments/phase4ad-author-semantics'
|
||||
CFG=json.loads((MODEL/'config.json').read_text(encoding='utf-8'))
|
||||
|
||||
def digest(path):
|
||||
h=hashlib.sha256()
|
||||
with Path(path).open('rb') as f:
|
||||
for b in iter(lambda:f.read(8<<20),b''):h.update(b)
|
||||
return h.hexdigest()
|
||||
|
||||
def save(path,value):
|
||||
path=Path(path);path.parent.mkdir(parents=True,exist_ok=True)
|
||||
with path.open('x',encoding='utf-8') as f:json.dump(value,f,indent=2,ensure_ascii=False,allow_nan=False);f.write('\n')
|
||||
|
||||
def sources():
|
||||
return {p.name:digest(p) for p in sorted(MODEL.iterdir()) if p.is_file() and p.name not in ('manifest.json','checksums.sha256')}
|
||||
|
||||
def integrity():
|
||||
spec=importlib.util.spec_from_file_location('frozen_af_context',AF_MODEL/'context.py')
|
||||
mod=importlib.util.module_from_spec(spec);spec.loader.exec_module(mod)
|
||||
result=mod.integrity()
|
||||
import sys
|
||||
sys.path.insert(0,str(ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'))
|
||||
from diagnostic import manifest_check
|
||||
for p in (AF_MODEL,AF_EXP,ROOT/'models/male-cns-lif/author-apl-lab-v1',ROOT/'experiments/phase4ag-author-apl'):
|
||||
result[str(p.relative_to(ROOT))]=manifest_check(p,digest)
|
||||
assert all(r['pass'] for r in result.values())
|
||||
return result
|
||||
|
||||
def check():
|
||||
p=json.loads((EXP/'protocol/preregistered.json').read_text())
|
||||
assert p['sources']==sources(),'Source changed after preregistration'
|
||||
for row in p['inputs']:assert digest(ROOT/row['path'])==row['sha256'],row['path']
|
||||
for row in json.loads((EXP/'protocol/prepared-hashes.json').read_text()):
|
||||
assert digest(EXP/row['path'])==row['sha256'],row['path']
|
||||
return p
|
||||
|
||||
def budget():
|
||||
size=sum(p.stat().st_size for p in EXP.rglob('*') if p.is_file())
|
||||
assert size<CFG['output_budget_bytes'],'New output resource budget exceeded'
|
||||
return size
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Seal a completed gate or preserved failure; never claim learning."""
|
||||
import json
|
||||
from context import ROOT,MODEL,EXP,CFG,check,save,digest,integrity,budget
|
||||
|
||||
|
||||
def seal(root):
|
||||
assert not (root/'manifest.json').exists() and not (root/'checksums.sha256').exists()
|
||||
rows=[dict(path=str(p.relative_to(root)),bytes=p.stat().st_size,sha256=digest(p))
|
||||
for p in sorted(root.rglob('*')) if p.is_file() and '__pycache__' not in p.parts]
|
||||
save(root/'manifest.json',dict(root=str(root),files=rows))
|
||||
with (root/'checksums.sha256').open('x') as f:
|
||||
for row in rows:f.write(row['sha256']+' '+row['path']+'\n')
|
||||
f.write(digest(root/'manifest.json')+' manifest.json\n')
|
||||
|
||||
|
||||
def main():
|
||||
from compare import compare,compare_gpu_repeats
|
||||
check();budget();comparisons=[];repeats={};components={}
|
||||
for p in sorted((EXP/'audit').glob('compare-*.json')):
|
||||
name=p.stem[len('compare-'):]
|
||||
actual=compare(EXP/'cases'/name,EXP/'gpu'/(name+'-r0'),EXP/'oracle'/name,None)
|
||||
assert actual==json.loads(p.read_text()),'Changed raw closed-loop evidence: '+name
|
||||
comparisons.append(dict(case=name,pass_all=actual['pass_all']))
|
||||
for p in sorted((EXP/'audit').glob('repeat-*.json')):
|
||||
name,repeat=p.stem[len('repeat-'):].rsplit('-r',1)
|
||||
actual=compare_gpu_repeats(EXP/'gpu'/(name+'-r0'),EXP/'gpu'/f'{name}-r{repeat}',None)
|
||||
assert actual==json.loads(p.read_text()),'Changed raw repeat evidence: '+name
|
||||
repeats[f'{name}-r{repeat}']=actual['pass_all']
|
||||
for p in (EXP/'component-audit').glob('*.json'):
|
||||
result=json.loads(p.read_text())
|
||||
components[p.stem]=result['pass_all']
|
||||
for filename,sha in result.get('input_sha256',{}).items():
|
||||
assert digest(EXP/'gpu'/(p.stem+'-r0')/filename)==sha
|
||||
gates={}
|
||||
for name in ('fixtures','probes','calibration'):
|
||||
path=EXP/'audit'/(name+'-gate.json')
|
||||
gates[name]=json.loads(path.read_text()) if path.exists() else None
|
||||
calpath=EXP/'calibration/summary.json'
|
||||
cal=json.loads(calpath.read_text()) if calpath.exists() else None
|
||||
if cal is not None:
|
||||
from calibration import summarize
|
||||
assert summarize(out_json=None)==cal,'CAL raw evidence changed'
|
||||
frozen=json.loads((EXP/'regression/frozen-five.json').read_text())
|
||||
after=integrity();save(EXP/'regression/old-integrity-after.json',after)
|
||||
stops=[dict(path=str(p.relative_to(EXP)),report=json.loads(p.read_text())) for p in sorted((EXP/'audit').glob('stopped-*.json'))]
|
||||
expected_cases=CFG['fixtures']+CFG['calibration']['full_oracle_probes']
|
||||
expected_repeats=[f'{name}-r{r}' for name in CFG['fixtures'] for r in range(1,CFG['fixture_repeats'])]
|
||||
compared={x['case']:x['pass_all'] for x in comparisons}
|
||||
missing=dict(comparisons=sorted(set(expected_cases)-set(compared)),
|
||||
components=sorted(set(expected_cases)-set(components)),repeats=sorted(set(expected_repeats)-set(repeats)))
|
||||
evidence_ok=(set(compared)==set(components)==set(expected_cases) and set(repeats)==set(expected_repeats) and
|
||||
all(compared.values()) and all(components.values()) and all(repeats.values()))
|
||||
numerical=evidence_ok and all(gates[x] is not None and gates[x]['pass_all'] for x in ('fixtures','probes'))
|
||||
representation=bool(cal is not None and cal['pass_all'])
|
||||
if numerical:status='PASS-FIXED-CAL-REPRESENTATION' if representation else 'FAIL-FIXED-CAL-REPRESENTATION' if cal is not None else 'INCOMPLETE-CAL-EXECUTION'
|
||||
else:status='FAIL-NUMERICAL-OR-AUDIT-GATE' if any(not p for group in (compared,components,repeats) for p in group.values()) else 'INCOMPLETE-NUMERICAL-EXECUTION'
|
||||
oldpass=bool(frozen['pass_all'] and all(r['pass'] for r in after.values()))
|
||||
if not oldpass:status='FAIL-FROZEN-INTEGRITY-OR-REGRESSION'
|
||||
measurements=[]
|
||||
for p in sorted((EXP/'gpu').glob('*/summary.json')):
|
||||
s=json.loads(p.read_text())
|
||||
measurements.append({k:s[k] for k in ('case','total_spikes','accepted_requests','nodes','edges','diagnostic_simulation_seconds','peak_allocated_vram_bytes')})
|
||||
result=dict(status=status,numerical_gate_pass=bool(numerical),fixed_CAL_representation_pass=representation,
|
||||
gates=gates,closed_loop_comparisons=comparisons,component_results=components,repeat_results=repeats,
|
||||
missing_numerical_evidence=missing,calibration=cal,stopped=stops,
|
||||
old_frozen_integrity_and_five_reference_pass=oldpass,measurements=measurements,
|
||||
unchanged_APL_parameters=CFG['apl'],author_LIF_precision='float64',legacy_APL_precision='float32',
|
||||
explicit_hypothesis=CFG['hypothesis'],APL_nonKC_fast_edges_unreplaced=1212,
|
||||
shared_APL_component_oracle_limitation=True,whole_brain_universal_equivalence_claim=False,
|
||||
heldout_generalization_tested=False,formal_AB_tested=False,learning_achieved=False,candidate_generated=False,
|
||||
payload_bytes_before_manifest=budget(),
|
||||
next_gate='Independent seeds/novel-odor representation, then credit specificity; no learning promotion from CAL' if numerical and representation else
|
||||
'Audit first failure in this explicitly declared hypothesis; no gain search, no heldout tuning, no conditioning')
|
||||
save(EXP/'audit/final-status.json',result)
|
||||
lines=['# Phase 4AG: author LIF + unchanged legacy graded APL','',f'Status: `{status}`.','',
|
||||
f'Numerical closed-loop gate: {numerical}; original complete fixed six-CAL gate: {representation}.',
|
||||
'The independent Brian LIF and candidate use independent state but share the APL arithmetic component. Original frozen APL AST replay and a float64 error audit are separately recorded.',
|
||||
'Author LIF uses float64; APL retains float32, tau20ms/gain0.01/normalization1, full original matrices and order. No learning/parameter search.',
|
||||
'The explicitly hypothesized unconditional end-APL write includes refractory KC. Original fast APL mask leaves1,212 non-KC edges unreplaced; this scope limitation was not silently fixed.',
|
||||
f'Old checksums and five frozen reference trials: {oldpass}.',
|
||||
f'Successfully compared cases: {comparisons}.',
|
||||
'Detailed CAL, exact comparisons, component errors, event/request traces and execution exceptions are linked from audit/final-status.json.',
|
||||
'Wall/VRAM here include per-tick CPU synchronization, IPC and hashes. They are diagnostic costs, not runtime-performance benchmarks.',
|
||||
'No heldout dynamic outputs, formal A/B, learning, game, reward, decoder or new candidate.',
|
||||
'Next gate: '+result['next_gate'],'']
|
||||
with (EXP/'REPORT.md').open('x',encoding='utf-8') as f:f.write('\n'.join(lines))
|
||||
check();seal(MODEL);seal(EXP);print(status,flush=True)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,346 @@
|
||||
"""Diagnostic author-float64 LIF plus frozen legacy-float32 APL overlay.
|
||||
|
||||
The sealed Phase 4AE CUDA operators are reused without modification. APL is
|
||||
an explicitly mixed-precision engineering overlay, not an author-model claim.
|
||||
All CUDA/CPU synchronization here is diagnostic; no performance claim is made.
|
||||
"""
|
||||
from functools import lru_cache
|
||||
import csv
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import time
|
||||
import traceback
|
||||
import types
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
from legacy_apl_worker import LegacyAPL, N_NEURONS, file_sha256, array_sha256
|
||||
|
||||
|
||||
SOURCE_ROOT = Path(__file__).resolve().parent
|
||||
AE_MODEL = SOURCE_ROOT.parent / "author-runtime-lab-v1"
|
||||
AF_MODEL = SOURCE_ROOT.parent / "author-fullgraph-lab-v1"
|
||||
COEFFICIENT_NAMES = ("gd", "c0", "cg", "cv", "threshold", "reset", "seed_volt")
|
||||
REQUEST_FIELDS = ("tick", "neuron_idx", "attempted", "accepted", "rejected")
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def write_json(path, value):
|
||||
with Path(path).open("x", encoding="utf-8") as output:
|
||||
json.dump(value, output, indent=2, allow_nan=False)
|
||||
output.write("\n")
|
||||
|
||||
|
||||
def raw_hash(value, dtype):
|
||||
return hashlib.sha256(np.ascontiguousarray(value, dtype=dtype).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def load_source(path, name):
|
||||
module = types.ModuleType(name)
|
||||
module.__file__ = str(path)
|
||||
exec(compile(Path(path).read_bytes(), str(path), "exec"), module.__dict__)
|
||||
return module
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def sealed_backend():
|
||||
return load_source(AE_MODEL / "gpu_backend.py", "phase4ag_sealed_ae_backend")
|
||||
|
||||
|
||||
@lru_cache(maxsize=1)
|
||||
def queue_function():
|
||||
return load_source(AF_MODEL / "oracle_full.py", "phase4ag_sealed_af_queue").queue_summary
|
||||
|
||||
|
||||
@lru_cache(maxsize=4)
|
||||
def load_graph(graph_directory):
|
||||
root = Path(graph_directory)
|
||||
require(root.is_absolute(), "graph_dir must be absolute")
|
||||
graph = {}
|
||||
for name, dtype in (("nodes", np.int32), ("edge_id", np.int64), ("src", np.int32),
|
||||
("dst", np.int32), ("signed_raw", np.int64), ("ptr", np.int64),
|
||||
("weights", np.float64)):
|
||||
value = np.load(root / (name + ".npy"), mmap_mode="r", allow_pickle=False)
|
||||
require(value.dtype == dtype and value.ndim == 1, "Graph dtype/rank mismatch: " + name)
|
||||
graph[name] = value
|
||||
nodes, edges = graph["nodes"], graph["edge_id"]
|
||||
n, e = len(nodes), len(edges)
|
||||
require(0 < n <= N_NEURONS and bool(np.all((nodes >= 0) & (nodes < N_NEURONS))) and
|
||||
bool(np.all(np.diff(nodes) > 0)), "Global node IDs must be sorted unique")
|
||||
require(n < 512 or (n == N_NEURONS and np.array_equal(nodes, np.arange(N_NEURONS, dtype=np.int32))),
|
||||
"Only small fixtures or the intact frozen full graph are permitted")
|
||||
require(n != N_NEURONS or e == 25582938, "Full graph edge count changed")
|
||||
for name in ("src", "dst", "signed_raw", "weights"):
|
||||
require(graph[name].shape == (e,), "Graph edge array length mismatch: " + name)
|
||||
ptr = graph["ptr"]
|
||||
require(ptr.shape == (n + 1,) and ptr[0] == 0 and ptr[-1] == e and
|
||||
bool(np.all(np.diff(ptr) >= 0)), "Invalid graph CSR")
|
||||
require(np.array_equal(graph["src"], np.repeat(np.arange(n, dtype=np.int32), np.diff(ptr))),
|
||||
"Source order does not match CSR")
|
||||
# Global edge IDs need not equal local row indices in a fixture.
|
||||
for start in range(0, e, 1 << 20):
|
||||
stop = min(start + (1 << 20), e)
|
||||
require(bool(np.all((graph["dst"][start:stop] >= 0) & (graph["dst"][start:stop] < n))),
|
||||
"Local destination out of range")
|
||||
require(bool(np.isfinite(graph["weights"][start:stop]).all()), "Nonfinite base SI weights")
|
||||
require(np.array_equal(np.sign(graph["weights"][start:stop]), np.sign(graph["signed_raw"][start:stop])),
|
||||
"Base weight sign/zero pattern differs from frozen raw contacts")
|
||||
lo = max(0, start - 1)
|
||||
require(bool(np.all(edges[start:stop] >= 0)) and bool(np.all(np.diff(edges[lo:stop]) > 0)),
|
||||
"Global edge IDs must retain canonical increasing source/edge order")
|
||||
return graph
|
||||
|
||||
|
||||
def load_inputs(case_dir):
|
||||
spec = json.loads((case_dir / "spec.json").read_text(encoding="utf-8"))
|
||||
for name in ("name", "steps", "dt_ms", "duration_ms", "model", "graph_dir", "apl",
|
||||
"apl_bundle_dir", "coefficients_path", "record_history"):
|
||||
require(name in spec, "Missing case field: " + name)
|
||||
steps = spec["steps"]
|
||||
require(type(steps) is int and 0 < steps <= 1000 and float(spec["dt_ms"]) == .1 and
|
||||
float(spec["duration_ms"]) == steps * .1, "Unexpected steps/dt/duration")
|
||||
require(type(spec["record_history"]) is bool, "record_history must be Boolean")
|
||||
require(spec["apl"] == {"tau_ms": 20, "gain": .01, "normalization": 1}, "Frozen APL parameters changed")
|
||||
for name in ("graph_dir", "apl_bundle_dir", "coefficients_path"):
|
||||
require(Path(spec[name]).is_absolute(), "Input path must be absolute: " + name)
|
||||
model = spec["model"]
|
||||
for name in ("resting_potential_mv", "threshold_mv", "reset_potential_mv", "synaptic_delay_ms",
|
||||
"refractory_ms", "w_syn_mv_per_anatomical_synapse"):
|
||||
require(name in model and math.isfinite(float(model[name])), "Invalid model field: " + name)
|
||||
require(float(model["synaptic_delay_ms"]) == 1.8 and float(model["refractory_ms"]) == 2.2 and
|
||||
float(model["w_syn_mv_per_anatomical_synapse"]) == .275, "Frozen delay/ref/W_syn changed")
|
||||
coeff = json.loads(Path(spec["coefficients_path"]).read_text(encoding="utf-8"))
|
||||
for name in COEFFICIENT_NAMES:
|
||||
require(name in coeff and math.isfinite(float(coeff[name])), "Invalid author coefficient: " + name)
|
||||
require(coeff["threshold"] == float(model["threshold_mv"]) * .001 and
|
||||
coeff["reset"] == float(model["reset_potential_mv"]) * .001, "Coefficient threshold/reset mismatch")
|
||||
graph = load_graph(str(Path(spec["graph_dir"]).resolve()))
|
||||
n = len(graph["nodes"])
|
||||
require(spec["record_history"] == (n < 512), "Fixtures need dense history; full graph must not duplicate dense histories")
|
||||
with np.load(case_dir / "requests.npz", allow_pickle=False) as source:
|
||||
require(set(source.files) == {"tick", "neuron_idx"}, "Request sparse schema changed")
|
||||
ticks, requested = source["tick"].copy(), source["neuron_idx"].copy()
|
||||
require(ticks.dtype == np.int32 and requested.dtype == np.int32 and ticks.ndim == 1 and
|
||||
ticks.shape == requested.shape, "Invalid sparse request dtype/shape")
|
||||
require(bool(np.all((ticks >= 0) & (ticks < steps))), "Request tick outside trial")
|
||||
if len(ticks) > 1:
|
||||
require(bool(np.all((ticks[1:] > ticks[:-1]) |
|
||||
((ticks[1:] == ticks[:-1]) & (requested[1:] > requested[:-1])))),
|
||||
"Requests must be sorted unique by tick/global neuron")
|
||||
local = np.searchsorted(graph["nodes"], requested)
|
||||
require(bool(np.all(local < n)) and np.array_equal(graph["nodes"][local], requested),
|
||||
"Requested neuron absent from graph")
|
||||
initial = {"initial_v_mv": np.full(n, float(model["resting_potential_mv"]), dtype=np.float64),
|
||||
"initial_g_mv": np.zeros(n, dtype=np.float64)}
|
||||
initial_path = case_dir / "arrays.npz"
|
||||
if initial_path.exists():
|
||||
with np.load(initial_path, allow_pickle=False) as source:
|
||||
require(set(source.files).issubset(initial), "Unexpected initial-state array")
|
||||
for name in source.files:
|
||||
value = source[name]
|
||||
require(value.dtype == np.float64 and value.shape == (n,) and bool(np.isfinite(value).all()),
|
||||
"Invalid initial state: " + name)
|
||||
initial[name] = value.copy()
|
||||
return spec, graph, coeff, ticks, requested, local, initial
|
||||
|
||||
|
||||
def new_array(path, dtype, shape):
|
||||
require(not Path(path).exists(), "Refusing existing output: " + str(path))
|
||||
return np.lib.format.open_memmap(path, mode="w+", dtype=dtype, shape=shape)
|
||||
|
||||
|
||||
def run_gpu(case_dir, out_dir, build_root):
|
||||
"""Run fresh candidate state; never consume oracle/candidate APL traces.
|
||||
|
||||
pass_all denotes internal recording/finite integrity only. Independent
|
||||
comparison and the original complete representation gate are separate.
|
||||
"""
|
||||
case_dir, out_dir = Path(case_dir), Path(out_dir)
|
||||
require(not out_dir.exists(), "Refusing existing output directory: " + str(out_dir))
|
||||
out_dir.mkdir(parents=True, exist_ok=False)
|
||||
start, tick, completed = time.perf_counter(), -1, 0
|
||||
maps = {}
|
||||
try:
|
||||
spec, graph, coeff, request_ticks, request_nodes, request_local, initial = load_inputs(case_dir)
|
||||
require(torch.cuda.is_available() and torch.cuda.device_count() == 1 and
|
||||
"4090" in torch.cuda.get_device_name(), "Only the declared single visible RTX4090 is permitted")
|
||||
ext, build = sealed_backend().build_extension(build_root)
|
||||
build_done = time.perf_counter()
|
||||
apl = LegacyAPL(spec["apl_bundle_dir"], spec["dt_ms"])
|
||||
write_json(out_dir / "apl-component.json", apl.metadata)
|
||||
device = apl.device
|
||||
torch.cuda.reset_peak_memory_stats(device)
|
||||
n, e, steps = len(graph["nodes"]), len(graph["edge_id"]), spec["steps"]
|
||||
|
||||
def upload(value, dtype):
|
||||
return torch.from_numpy(np.array(value, copy=True, order="C")).to(device=device, dtype=dtype).contiguous()
|
||||
|
||||
indptr, dst = upload(graph["ptr"], torch.int64), upload(graph["dst"], torch.int32)
|
||||
global_nodes_gpu = upload(graph["nodes"], torch.int64)
|
||||
base_weights = upload(graph["weights"], torch.float64)
|
||||
weights = base_weights.clone()
|
||||
del base_weights
|
||||
local_apl = np.flatnonzero(np.isin(graph["nodes"], apl.apl_globals)).astype(np.int64)
|
||||
apl_out_mask = np.isin(graph["src"], local_apl)
|
||||
if bool(np.any(apl_out_mask)):
|
||||
weights[upload(apl_out_mask, torch.bool)] = 0.0
|
||||
effective_hash = hashlib.sha256()
|
||||
for offset in range(0, e, 1 << 20):
|
||||
effective_hash.update(weights[offset:offset + (1 << 20)].cpu().numpy().astype("<f8", copy=False).tobytes())
|
||||
common_kc, local_kc, full_kc = np.intersect1d(graph["nodes"], apl.kc_globals, assume_unique=True, return_indices=True)
|
||||
kc_local_gpu, kc_full_gpu = upload(local_kc, torch.int64), upload(full_kc, torch.int64)
|
||||
v = upload(initial["initial_v_mv"] * .001, torch.float64)
|
||||
g = upload(initial["initial_g_mv"] * .001, torch.float64)
|
||||
ref = torch.zeros(n, device=device, dtype=torch.int32)
|
||||
counts = torch.zeros(n, device=device, dtype=torch.int32)
|
||||
requests = torch.zeros(n, device=device, dtype=torch.uint8)
|
||||
ring = torch.zeros((19, n), device=device, dtype=torch.uint8)
|
||||
for name, width in (("drive", 2), ("state", 2), ("inhibition", 4064)):
|
||||
maps["apl_" + name] = new_array(out_dir / ("apl-" + name + ".float32.npy"), np.float32, (steps, width))
|
||||
if spec["record_history"]:
|
||||
for name, dtype in (("v", np.float64), ("g", np.float64), ("ref", np.int32)):
|
||||
maps[name] = new_array(out_dir / (name + "-history." + np.dtype(dtype).name + ".npy"), dtype, (steps, n))
|
||||
maps["pre_" + name] = new_array(out_dir / ("preAPL-" + name + "." + np.dtype(dtype).name + ".npy"), dtype, (steps, n))
|
||||
event_steps, event_nodes = [], []
|
||||
accepted_requests_total, due_total, accepted_due_total = 0, 0, 0
|
||||
queue_summary = queue_function()
|
||||
simulation_start = time.perf_counter()
|
||||
with (out_dir / "queue.jsonl").open("x", encoding="utf-8", buffering=1) as queue_output, \
|
||||
(out_dir / "tick-state-hashes.jsonl").open("x", encoding="utf-8", buffering=1) as hash_output, \
|
||||
(out_dir / "requests.csv").open("x", encoding="utf-8", newline="", buffering=1) as request_output:
|
||||
writer = csv.DictWriter(request_output, fieldnames=REQUEST_FIELDS)
|
||||
writer.writeheader()
|
||||
for tick in range(steps):
|
||||
requests.zero_()
|
||||
begin, end = np.searchsorted(request_ticks, (tick, tick + 1))
|
||||
if end > begin:
|
||||
requested_local_gpu = upload(request_local[begin:end], torch.int64)
|
||||
requests[requested_local_gpu] = 1
|
||||
accepted = (ref[requested_local_gpu] <= 1).to(torch.uint8).cpu().numpy()
|
||||
accepted_requests_total += int(accepted.sum())
|
||||
writer.writerows({"tick": tick, "neuron_idx": int(node), "attempted": 1,
|
||||
"accepted": int(ok), "rejected": 1 - int(ok)}
|
||||
for node, ok in zip(request_nodes[begin:end], accepted))
|
||||
spikes = ring[tick % 19]
|
||||
ext.state_step(v, g, ref, requests, spikes, counts,
|
||||
*[float(coeff[name]) for name in COEFFICIENT_NAMES], 22)
|
||||
expanded_global = np.empty(0, dtype=np.int64)
|
||||
accepted_cpu = np.empty(0, dtype=np.uint8)
|
||||
if tick >= 18:
|
||||
due = ring[(tick - 18) % 19]
|
||||
sources = torch.nonzero(due, as_tuple=False).flatten().contiguous()
|
||||
if sources.numel():
|
||||
degrees = indptr[sources + 1] - indptr[sources]
|
||||
offsets = (torch.cumsum(degrees, dim=0) - degrees).contiguous()
|
||||
edge_count = int(degrees.sum().item())
|
||||
expanded = torch.empty(edge_count, device=device, dtype=torch.int64)
|
||||
ext.fill_edges(indptr, sources, offsets, expanded)
|
||||
if edge_count:
|
||||
destination = dst[expanded]
|
||||
# Actual active fanout order, BEFORE stable target grouping.
|
||||
expanded_global = graph["edge_id"][expanded.cpu().numpy()].copy()
|
||||
accepted_cpu = (ref[destination.long()] == 0).to(torch.uint8).cpu().numpy().copy()
|
||||
order = torch.argsort(destination, stable=True)
|
||||
ordered_edges, ordered_targets = expanded[order].contiguous(), destination[order]
|
||||
boundary = torch.ones(edge_count, device=device, dtype=torch.bool)
|
||||
boundary[1:] = ordered_targets[1:] != ordered_targets[:-1]
|
||||
starts = torch.nonzero(boundary, as_tuple=False).flatten().contiguous()
|
||||
targets = ordered_targets[starts].contiguous()
|
||||
ext.apply_arrivals(ordered_edges, starts, targets, weights, g, ref)
|
||||
queue = queue_summary(tick, expanded_global, accepted_cpu)
|
||||
queue_output.write(json.dumps(queue, allow_nan=False) + "\n")
|
||||
due_total += queue["count"]
|
||||
accepted_due_total += queue["accepted_count"]
|
||||
if spec["record_history"]:
|
||||
for name, value in (("v", v), ("g", g), ("ref", ref)):
|
||||
maps["pre_" + name][tick] = value.cpu().numpy()
|
||||
if n == N_NEURONS:
|
||||
full_spikes = spikes
|
||||
else:
|
||||
apl.spikes.zero_()
|
||||
apl.spikes[global_nodes_gpu] = spikes
|
||||
full_spikes = apl.spikes
|
||||
drive, apl_state, h = apl.advance(full_spikes)
|
||||
# Mixed precision contract: widen f32 h EXACTLY, THEN multiply
|
||||
# in f64, THEN subtract in f64. Do not guard refractory KCs.
|
||||
delta_si = h[kc_full_gpu].to(torch.float64) * 0.001
|
||||
g[kc_local_gpu] = g[kc_local_gpu] - delta_si
|
||||
apl_cpu = {"drive": drive.cpu().numpy().copy(), "state": apl_state.cpu().numpy().copy(),
|
||||
"inhibition": h.cpu().numpy().copy()}
|
||||
for name, value in apl_cpu.items():
|
||||
maps["apl_" + name][tick] = value
|
||||
actual = {"v": v.cpu().numpy().copy(), "g": g.cpu().numpy().copy(),
|
||||
"ref": ref.cpu().numpy().copy()}
|
||||
finite = all(bool(np.isfinite(value).all()) for value in list(actual.values()) + list(apl_cpu.values()))
|
||||
row = {"tick": tick, **{name + "_sha256": raw_hash(value, "<i4" if name == "ref" else "<f8")
|
||||
for name, value in actual.items()}}
|
||||
hash_output.write(json.dumps(row, allow_nan=False) + "\n")
|
||||
if spec["record_history"]:
|
||||
for name, value in actual.items():
|
||||
maps[name][tick] = value
|
||||
fired_local = torch.nonzero(spikes, as_tuple=False).flatten().cpu().numpy()
|
||||
if len(fired_local):
|
||||
event_steps.append(np.full(len(fired_local), tick, dtype=np.int32))
|
||||
event_nodes.append(graph["nodes"][fired_local].copy())
|
||||
completed += 1
|
||||
if not finite:
|
||||
with (out_dir / "first-failure-state.npz").open("xb") as target:
|
||||
np.savez_compressed(target, tick=np.array(tick, dtype=np.int32), **actual,
|
||||
**{"apl_" + name: value for name, value in apl_cpu.items()})
|
||||
require(False, "Nonfinite state; first failure saved")
|
||||
torch.cuda.synchronize(device)
|
||||
|
||||
def joined(values):
|
||||
return np.concatenate(values).astype(np.int32, copy=False) if values else np.empty(0, dtype=np.int32)
|
||||
|
||||
state = {"step": joined(event_steps), "neuron_idx": joined(event_nodes), "counts": counts.cpu().numpy(),
|
||||
"v": v.cpu().numpy(), "g": g.cpu().numpy(), "ref": ref.cpu().numpy()}
|
||||
with (out_dir / "state.npz").open("xb") as target:
|
||||
np.savez_compressed(target, **state)
|
||||
event_locals = np.searchsorted(graph["nodes"], state["neuron_idx"])
|
||||
counts_exact = np.array_equal(np.bincount(event_locals, minlength=n), state["counts"])
|
||||
require(counts_exact and completed == steps, "Sparse events/counts/completed ticks mismatch")
|
||||
for value in maps.values():
|
||||
value.flush()
|
||||
summary = {"case": spec["name"], "complete": True, "pass_all": True,
|
||||
"pass_all_scope": "internal finite/recording/event-count integrity only; independent equivalence not claimed",
|
||||
"steps": steps, "completed_ticks": completed, "nodes": n, "edges": e, "precision": "float64",
|
||||
"apl_precision": "float32", "state_units": "SI volts", "total_spikes": len(state["step"]),
|
||||
"requested": len(request_nodes), "accepted_requests": accepted_requests_total,
|
||||
"rejected_requests": len(request_nodes) - accepted_requests_total,
|
||||
"arriving_edges": due_total, "accepted_arriving_edges": accepted_due_total,
|
||||
"counts_match_sparse_events": bool(counts_exact), "all_finite": True,
|
||||
"record_history": spec["record_history"], "apl_fast_edges_zeroed": int(apl_out_mask.sum()),
|
||||
"kc_targets_present": len(common_kc), "effective_weights_raw_sha256": effective_hash.hexdigest(),
|
||||
"apl_component": apl.metadata, "apl_schedule": "end, after threshold/reset and guarded arrivals, unconditional all KC",
|
||||
"apl_si_conversion": "h32 -> float64 -> multiply float64 0.001 -> separate float64 subtraction",
|
||||
"build": build, "sealed_ae_backend_sha256": file_sha256(AE_MODEL / "gpu_backend.py"),
|
||||
"shared_apl_component_oracle_limitation": True, "oracle_validation_claim": False,
|
||||
"learning": False, "performance_runtime_claim": False, "cpu_sync_each_tick": True,
|
||||
"build_or_cache_seconds": build_done - start,
|
||||
"diagnostic_simulation_seconds": time.perf_counter() - simulation_start,
|
||||
"wall_seconds": time.perf_counter() - start,
|
||||
"peak_allocated_vram_bytes": int(torch.cuda.max_memory_allocated(device)),
|
||||
"peak_reserved_vram_bytes": int(torch.cuda.max_memory_reserved(device)),
|
||||
"input_sha256": {"spec.json": file_sha256(case_dir / "spec.json"),
|
||||
"requests.npz": file_sha256(case_dir / "requests.npz"),
|
||||
"base_weights.npy": file_sha256(Path(spec["graph_dir"]) / "weights.npy"),
|
||||
"coefficients.json": file_sha256(spec["coefficients_path"])}}
|
||||
if (case_dir / "arrays.npz").exists():
|
||||
summary["input_sha256"]["arrays.npz"] = file_sha256(case_dir / "arrays.npz")
|
||||
write_json(out_dir / "summary.json", summary)
|
||||
return summary
|
||||
except Exception as error:
|
||||
for value in maps.values():
|
||||
value.flush()
|
||||
write_json(out_dir / "error.json", {"complete": False, "pass_all": False,
|
||||
"tick_in_progress": tick, "completed_ticks": completed,
|
||||
"type": type(error).__name__, "message": str(error), "traceback": traceback.format_exc()})
|
||||
raise
|
||||
@@ -0,0 +1,175 @@
|
||||
"""Frozen eager Torch float32 APL component and binary oracle sidecar.
|
||||
|
||||
This is the Phase 4P operator sequence, not an author-neuron implementation.
|
||||
No precision/determinism flags are changed here. The full matrix dimensions
|
||||
are retained even when the caller simulates a small neuronal fixture.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import traceback
|
||||
|
||||
import numpy as np
|
||||
import torch
|
||||
|
||||
|
||||
PROTOCOL = "phase4ag-apl-v1"
|
||||
N_NEURONS, N_APL, N_KC = 166700, 2, 4064
|
||||
BUNDLE_FILES = ("input_matrix.npy", "output_matrix.npy", "kc_globals.npy", "apl_globals.npy")
|
||||
FRAME = struct.Struct("<qQ")
|
||||
PAYLOAD_BYTES = (N_APL + N_APL + N_KC) * 4
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def file_sha256(path):
|
||||
digest = hashlib.sha256()
|
||||
with Path(path).open("rb") as stream:
|
||||
for block in iter(lambda: stream.read(8 << 20), b""):
|
||||
digest.update(block)
|
||||
return digest.hexdigest()
|
||||
|
||||
|
||||
def array_sha256(value):
|
||||
return hashlib.sha256(np.ascontiguousarray(value).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def environment_metadata():
|
||||
device = torch.cuda.current_device()
|
||||
return {"torch": str(torch.__version__), "numpy": str(np.__version__),
|
||||
"cuda": torch.version.cuda, "cudnn": torch.backends.cudnn.version(),
|
||||
"device_name": torch.cuda.get_device_name(device), "device_index": device,
|
||||
"device_capability": list(torch.cuda.get_device_capability(device)),
|
||||
"deterministic_algorithms": torch.are_deterministic_algorithms_enabled(),
|
||||
"cudnn_deterministic": torch.backends.cudnn.deterministic,
|
||||
"cudnn_benchmark": torch.backends.cudnn.benchmark,
|
||||
"cuda_matmul_allow_tf32": torch.backends.cuda.matmul.allow_tf32,
|
||||
"cudnn_allow_tf32": torch.backends.cudnn.allow_tf32,
|
||||
"float32_matmul_precision": torch.get_float32_matmul_precision(),
|
||||
"CUBLAS_WORKSPACE_CONFIG": os.environ.get("CUBLAS_WORKSPACE_CONFIG"),
|
||||
"precision_configuration_changed_by_component": False}
|
||||
|
||||
|
||||
class LegacyAPL:
|
||||
"""Independent zero-initialized APL state with original full-width matrices."""
|
||||
|
||||
def __init__(self, bundle_dir, dt_ms):
|
||||
require(float(dt_ms) == 0.1, "Only frozen dt_ms=0.1 is allowed")
|
||||
require(torch.cuda.is_available(), "CUDA is required for the legacy Torch component")
|
||||
self.bundle_dir = Path(bundle_dir)
|
||||
expected = {"input_matrix.npy": (np.float32, (N_APL, N_NEURONS)),
|
||||
"output_matrix.npy": (np.float32, (N_APL, N_KC)),
|
||||
"kc_globals.npy": (np.int32, (N_KC,)),
|
||||
"apl_globals.npy": (np.int32, (N_APL,))}
|
||||
arrays = {}
|
||||
for name, (dtype, shape) in expected.items():
|
||||
value = np.load(self.bundle_dir / name, allow_pickle=False)
|
||||
require(value.dtype == dtype and value.shape == shape and value.flags.c_contiguous,
|
||||
"APL bundle dtype/shape/layout mismatch: " + name)
|
||||
require(bool(np.isfinite(value).all()), "Nonfinite APL bundle: " + name)
|
||||
arrays[name] = value
|
||||
for name in ("kc_globals.npy", "apl_globals.npy"):
|
||||
ids = arrays[name]
|
||||
require(bool(np.all((ids >= 0) & (ids < N_NEURONS))) and
|
||||
bool(np.all(np.diff(ids) > 0)), "APL/KC globals must be sorted unique: " + name)
|
||||
require(not np.intersect1d(arrays["kc_globals.npy"], arrays["apl_globals.npy"]).size,
|
||||
"APL and KC identities overlap")
|
||||
require(bool(np.all(arrays["input_matrix.npy"] >= 0)) and
|
||||
bool(np.all(arrays["output_matrix.npy"] >= 0)), "APL raw-contact matrices must be nonnegative")
|
||||
self.device = torch.device("cuda", torch.cuda.current_device())
|
||||
self.input_gpu = torch.from_numpy(arrays["input_matrix.npy"].copy()).to(self.device)
|
||||
self.output_gpu = torch.from_numpy(arrays["output_matrix.npy"].copy()).to(self.device)
|
||||
self.kc_globals = arrays["kc_globals.npy"].copy()
|
||||
self.apl_globals = arrays["apl_globals.npy"].copy()
|
||||
self.state = torch.zeros(N_APL, dtype=torch.float32, device=self.device)
|
||||
self.spikes = torch.zeros(N_NEURONS, dtype=torch.uint8, device=self.device)
|
||||
self.decay = np.float32(math.exp(-float(dt_ms) / 20.0))
|
||||
self.gain = np.float32(0.01)
|
||||
self.metadata = {"ok": True, "protocol": PROTOCOL, "n_neurons": N_NEURONS,
|
||||
"n_apl": N_APL, "n_kc": N_KC, "dt_ms": float(dt_ms),
|
||||
"files_sha256": {name: file_sha256(self.bundle_dir / name) for name in BUNDLE_FILES},
|
||||
"array_sha256": {name: array_sha256(value) for name, value in arrays.items()},
|
||||
"decay_f32_bits": f"{int(self.decay.view(np.uint32)):08x}",
|
||||
"gain_f32_bits": f"{int(self.gain.view(np.uint32)):08x}",
|
||||
"environment": environment_metadata(),
|
||||
"operator_sequence": "state.mul_(decay32); drive=mv(I32,spikes32); state.add_(drive); h=matmul(state,O32)*gain32",
|
||||
"h_units": "legacy mV", "initial_state": "float32 zeros"}
|
||||
|
||||
@torch.no_grad()
|
||||
def advance(self, spikes_u8):
|
||||
require(spikes_u8.device == self.device and spikes_u8.dtype == torch.uint8 and
|
||||
tuple(spikes_u8.shape) == (N_NEURONS,) and spikes_u8.is_contiguous(),
|
||||
"APL expects full-width contiguous CUDA uint8 spikes")
|
||||
# Preserve separate eager operations, including Python evaluation order
|
||||
# in the frozen state.mul_(...).add_(torch.mv(...)) expression.
|
||||
self.state.mul_(float(self.decay))
|
||||
drive = torch.mv(self.input_gpu, spikes_u8.to(torch.float32))
|
||||
self.state.add_(drive)
|
||||
h = torch.matmul(self.state, self.output_gpu) * self.gain
|
||||
return drive, self.state, h
|
||||
|
||||
def advance_global_fired(self, fired):
|
||||
require(fired.dtype == np.int32 and fired.ndim == 1, "Global spikes must be int32 vector")
|
||||
require(bool(np.all((fired >= 0) & (fired < N_NEURONS))) and
|
||||
bool(np.all(np.diff(fired) > 0)), "Global spikes must be sorted unique and in range")
|
||||
self.spikes.zero_()
|
||||
if fired.size:
|
||||
indices = torch.from_numpy(fired.copy()).to(device=self.device, dtype=torch.int64)
|
||||
self.spikes[indices] = 1
|
||||
return self.advance(self.spikes)
|
||||
|
||||
|
||||
def read_exact(stream, count, boundary_eof=False):
|
||||
result = bytearray()
|
||||
while len(result) < count:
|
||||
block = stream.read(count - len(result))
|
||||
if not block:
|
||||
if boundary_eof and not result:
|
||||
return None
|
||||
raise EOFError("Truncated APL request frame")
|
||||
result.extend(block)
|
||||
return bytes(result)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--bundle-dir", required=True)
|
||||
parser.add_argument("--dt-ms", required=True, type=float)
|
||||
args = parser.parse_args()
|
||||
# stdout is exclusively the handshake line and binary response frames.
|
||||
output, source = sys.stdout.buffer, sys.stdin.buffer
|
||||
component = LegacyAPL(args.bundle_dir, args.dt_ms)
|
||||
output.write((json.dumps(component.metadata, allow_nan=False) + "\n").encode("utf-8"))
|
||||
output.flush()
|
||||
expected_tick = 0
|
||||
while True:
|
||||
header = read_exact(source, FRAME.size, boundary_eof=True)
|
||||
if header is None:
|
||||
return
|
||||
tick, count = FRAME.unpack(header)
|
||||
require(tick == expected_tick and count <= N_NEURONS, "Invalid APL request tick/count")
|
||||
fired = np.frombuffer(read_exact(source, count * 4), dtype="<i4").copy()
|
||||
drive, state, h = component.advance_global_fired(fired)
|
||||
arrays = [value.detach().cpu().numpy().astype("<f4", copy=False) for value in (drive, state, h)]
|
||||
require(all(bool(np.isfinite(value).all()) for value in arrays), "Nonfinite APL output")
|
||||
output.write(FRAME.pack(tick, PAYLOAD_BYTES))
|
||||
for value in arrays:
|
||||
output.write(value.tobytes())
|
||||
output.flush()
|
||||
expected_tick += 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
try:
|
||||
main()
|
||||
except Exception:
|
||||
traceback.print_exc(file=sys.stderr)
|
||||
sys.exit(1)
|
||||
@@ -0,0 +1,90 @@
|
||||
{
|
||||
"root": "/data/digital-drosophila/models/male-cns-lif/author-apl-lab-v2",
|
||||
"files": [
|
||||
{
|
||||
"path": "PROTOCOL.md",
|
||||
"bytes": 5583,
|
||||
"sha256": "49f53924f758317bfcbd2b53cab9b8b785121eb99f3234a8343c9b483c5701d8"
|
||||
},
|
||||
{
|
||||
"path": "calibration.py",
|
||||
"bytes": 25414,
|
||||
"sha256": "edb9cb2d890471fd07afd88c959ef3540b0cd40af490fac3f26396b5e7f5bf9c"
|
||||
},
|
||||
{
|
||||
"path": "compare.py",
|
||||
"bytes": 33632,
|
||||
"sha256": "6e1616b086f667fa4e4b5087f500f8080281977ca9922e7b055acfddd2003bd5"
|
||||
},
|
||||
{
|
||||
"path": "config.json",
|
||||
"bytes": 1994,
|
||||
"sha256": "a8bafb9deec49d918976f2836d977ef9f13c26a6fb7f70e015879e29cf986c9a"
|
||||
},
|
||||
{
|
||||
"path": "context.py",
|
||||
"bytes": 2134,
|
||||
"sha256": "274eb13ccf38dbbc5c7f14d9d642130eb0f69072d91a090f208c30c27f87dd2b"
|
||||
},
|
||||
{
|
||||
"path": "finalize.py",
|
||||
"bytes": 6747,
|
||||
"sha256": "b6e3c67e668a971b544df32d76435ab5e537061b9fc2e8d9dedb5d2a726f5782"
|
||||
},
|
||||
{
|
||||
"path": "gpu_apl.py",
|
||||
"bytes": 21206,
|
||||
"sha256": "69f5d893f47dafff4ef7c3186c3df0b9014666b32c8f4e943b57016bec92b18a"
|
||||
},
|
||||
{
|
||||
"path": "legacy_apl_worker.py",
|
||||
"bytes": 8158,
|
||||
"sha256": "21ceec65fe42581dc55872010e0025d660e80358905f1cf4df1dadf341a3b291"
|
||||
},
|
||||
{
|
||||
"path": "oracle_apl.py",
|
||||
"bytes": 46236,
|
||||
"sha256": "357a7d3debb5abc34a8714c0233b456547a154d9d5855d28661b4f01019a80ea"
|
||||
},
|
||||
{
|
||||
"path": "prepare.py",
|
||||
"bytes": 15297,
|
||||
"sha256": "e9328567753e144fc315491970f7e02af13725b93f35e4d4b97f7c4c48ce3338"
|
||||
},
|
||||
{
|
||||
"path": "run_case.py",
|
||||
"bytes": 2116,
|
||||
"sha256": "afbe6d5a37c29bda66d38c470093584130d1852e67881c681662aec3c76b33c7"
|
||||
},
|
||||
{
|
||||
"path": "run_suite.py",
|
||||
"bytes": 3700,
|
||||
"sha256": "84c681030b0032119b5392d0c3f3c136a9c6b0d133eb548c40791a10bbbb8b63"
|
||||
},
|
||||
{
|
||||
"path": "test_calibration.py",
|
||||
"bytes": 8127,
|
||||
"sha256": "bab3136eecbc5bb09a985d715844ee845cb72b07930ce376bfb39be33bd1845a"
|
||||
},
|
||||
{
|
||||
"path": "test_protocol.py",
|
||||
"bytes": 11764,
|
||||
"sha256": "1a233f47943e01bbd845ac1dacdb1eec3f88c270c65a5f75bce1cf679b5de097"
|
||||
},
|
||||
{
|
||||
"path": "test_python_entry.py",
|
||||
"bytes": 1760,
|
||||
"sha256": "0ed665f686e3499f8543e48f1712ccb4523946971cb7a0e096cf4f6d10d3bcba"
|
||||
},
|
||||
{
|
||||
"path": "validate_components.py",
|
||||
"bytes": 24824,
|
||||
"sha256": "5cfc9a4505da6b8fb4e7d2617cdaedaa8475ed46a11ec9bdf0d4d25b1d22c2d3"
|
||||
},
|
||||
{
|
||||
"path": "verify_frozen.py",
|
||||
"bytes": 2160,
|
||||
"sha256": "0d0e72b2f1348f6648124c8c34f75094f76fc5e1a281fbcca130a5fad2f35293"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,747 @@
|
||||
"""Phase 4AG shared-APL-component closed-loop Brian oracle.
|
||||
|
||||
Brian has no Torch dependency. A separate, pinned science-Python process
|
||||
executes the unchanged legacy float32 APL operations from Brian's own spikes.
|
||||
This is independent LIF/scheduling validation with a shared APL numerical
|
||||
component, NOT an all-components-independent CPU oracle.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import csv
|
||||
import hashlib
|
||||
import importlib.metadata
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
import platform
|
||||
import queue
|
||||
import struct
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
import types
|
||||
from contextlib import ExitStack
|
||||
from pathlib import Path
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
BLOCK_BYTES = 8 << 20
|
||||
PROTOCOL = "phase4ag-apl-v1"
|
||||
AUTHOR_PATH = Path("/data/digital-drosophila/experiments/phase4ad-author-semantics/external-resources/model.py")
|
||||
AUTHOR_SHA256 = "fc45837d7122c6ce2a7f3f2f23c515992e4b232aadb919efabb72337fac88e4e"
|
||||
AE_PATH = Path("/data/digital-drosophila/models/male-cns-lif/author-runtime-lab-v1/oracle_stress.py")
|
||||
AE_SHA256 = "32eb1c42383068ce52701d2139a376ce155ad7072cde019949f8ca2d55492409"
|
||||
AF_PATH = Path("/data/digital-drosophila/models/male-cns-lif/author-fullgraph-lab-v1/oracle_full.py")
|
||||
AF_SHA256 = "d72003d98f0f1624b12126eff50bd361c43d888f4a64d90ba9247a58976f1961"
|
||||
EVENT_DTYPE = np.dtype([("step", "<i4"), ("neuron_idx", "<i4")])
|
||||
BUNDLE_FILES = ("input_matrix.npy", "output_matrix.npy", "kc_globals.npy", "apl_globals.npy")
|
||||
_PROCESS_USED = False
|
||||
|
||||
|
||||
def _check(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def _save(path, value):
|
||||
with Path(path).open("x", encoding="utf-8") as handle:
|
||||
json.dump(value, handle, indent=2, sort_keys=True, allow_nan=False)
|
||||
handle.write("\n")
|
||||
|
||||
|
||||
def _digest(path):
|
||||
result = hashlib.sha256()
|
||||
with Path(path).open("rb") as handle:
|
||||
for block in iter(lambda: handle.read(BLOCK_BYTES), b""):
|
||||
result.update(block)
|
||||
return result.hexdigest()
|
||||
|
||||
|
||||
def _array_hash(array, dtype):
|
||||
return hashlib.sha256(np.ascontiguousarray(array, dtype=dtype).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def _load_checked(path, expected, name):
|
||||
data = Path(path).read_bytes()
|
||||
_check(hashlib.sha256(data).hexdigest() == expected, f"Pinned source hash mismatch: {path}")
|
||||
module = types.ModuleType(name)
|
||||
module.__file__ = str(path)
|
||||
exec(compile(data, str(path), "exec"), module.__dict__)
|
||||
return module
|
||||
|
||||
|
||||
def _python_entry(path):
|
||||
"""A venv is selected by its invocation path, not the resolved ELF target."""
|
||||
entry = Path(path).absolute()
|
||||
_check(entry.is_file(), f"Missing declared Python entry: {entry}")
|
||||
return entry
|
||||
|
||||
|
||||
def _new_memmap(path, dtype, shape):
|
||||
_check(not Path(path).exists(), f"Refusing to overwrite output: {path}")
|
||||
return np.lib.format.open_memmap(path, mode="w+", dtype=dtype, shape=shape)
|
||||
|
||||
|
||||
def _budget(root, limit, reserve=0):
|
||||
total = 0
|
||||
for directory, _, filenames in os.walk(root, followlinks=False):
|
||||
for filename in filenames:
|
||||
total += (Path(directory) / filename).stat(follow_symlinks=False).st_size
|
||||
_check(total + int(reserve) <= limit,
|
||||
f"Output budget exceeded: {total}+{reserve}>{limit}; inputs and duration were not adjusted")
|
||||
return total
|
||||
|
||||
|
||||
class _PipeReader:
|
||||
"""One bounded reader thread, with timeout/EOF checks for every frame."""
|
||||
|
||||
def __init__(self, pipe, timeout):
|
||||
self.pipe, self.timeout = pipe, float(timeout)
|
||||
self.chunks = queue.Queue(maxsize=8)
|
||||
self.buffer = bytearray()
|
||||
self.thread = threading.Thread(target=self._pump, daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
def _pump(self):
|
||||
try:
|
||||
while True:
|
||||
chunk = os.read(self.pipe.fileno(), 65536)
|
||||
self.chunks.put(chunk)
|
||||
if not chunk:
|
||||
return
|
||||
except BaseException as error:
|
||||
self.chunks.put(error)
|
||||
|
||||
def _fill(self, deadline):
|
||||
remaining = deadline - time.monotonic()
|
||||
if remaining <= 0:
|
||||
raise TimeoutError("APL sidecar response timed out")
|
||||
try:
|
||||
chunk = self.chunks.get(timeout=remaining)
|
||||
except queue.Empty as error:
|
||||
raise TimeoutError("APL sidecar response timed out") from error
|
||||
if isinstance(chunk, BaseException):
|
||||
raise RuntimeError("APL sidecar stdout reader failed") from chunk
|
||||
if not chunk:
|
||||
raise EOFError("APL sidecar ended before a complete response")
|
||||
self.buffer.extend(chunk)
|
||||
|
||||
def read(self, size):
|
||||
deadline = time.monotonic() + self.timeout
|
||||
while len(self.buffer) < size:
|
||||
self._fill(deadline)
|
||||
result = bytes(self.buffer[:size])
|
||||
del self.buffer[:size]
|
||||
return result
|
||||
|
||||
def line(self):
|
||||
deadline = time.monotonic() + self.timeout
|
||||
while b"\n" not in self.buffer:
|
||||
_check(len(self.buffer) < 1 << 20, "APL sidecar handshake exceeds 1 MiB")
|
||||
self._fill(deadline)
|
||||
end = self.buffer.index(b"\n") + 1
|
||||
result = bytes(self.buffer[:end])
|
||||
del self.buffer[:end]
|
||||
return result
|
||||
|
||||
|
||||
class _APLWorker:
|
||||
def __init__(self, python, script, bundle_dir, stderr, timeout=180):
|
||||
self.command = [str(python), "-B", str(script), "--bundle-dir", str(bundle_dir), "--dt-ms", "0.1"]
|
||||
self.process = subprocess.Popen(self.command, stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE, stderr=stderr, bufsize=0, shell=False)
|
||||
self.reader = _PipeReader(self.process.stdout, timeout)
|
||||
self.tick = 0
|
||||
try:
|
||||
self.hello = json.loads(self.reader.line().decode("utf-8"))
|
||||
_check(self.hello.get("ok") is True, "APL sidecar did not acknowledge initialization")
|
||||
_check(self.hello.get("protocol") == PROTOCOL, "Unexpected APL sidecar protocol")
|
||||
for key, value in (("n_neurons", 166700), ("n_apl", 2), ("n_kc", 4064)):
|
||||
_check(self.hello.get(key) == value, f"APL sidecar changed {key}")
|
||||
_check(self.hello.get("dt_ms") == .1, "APL sidecar changed dt")
|
||||
expected_decay = f"{int(np.float32(math.exp(-.1 / 20.)).view(np.uint32)):08x}"
|
||||
expected_gain = f"{int(np.float32(.01).view(np.uint32)):08x}"
|
||||
_check(self.hello.get("decay_f32_bits") == expected_decay and self.hello.get("gain_f32_bits") == expected_gain,
|
||||
"APL sidecar decay/gain differs from the frozen float32 scalars")
|
||||
_check(self.hello.get("files_sha256") == {name: _digest(bundle_dir / name) for name in BUNDLE_FILES},
|
||||
"APL sidecar bundle hashes differ from the independently read bundle")
|
||||
except BaseException:
|
||||
self.abort()
|
||||
raise
|
||||
|
||||
def step(self, tick, global_spikes):
|
||||
_check(tick == self.tick, "APL sidecar request ticks must be consecutive")
|
||||
nodes = np.asarray(global_spikes)
|
||||
_check(nodes.ndim == 1 and nodes.dtype == np.dtype("int32"), "APL input spikes must be int32 globals")
|
||||
_check(not len(nodes) or (nodes[0] >= 0 and nodes[-1] < 166700 and np.all(np.diff(nodes) > 0)),
|
||||
"APL input spikes must be sorted, unique, and in the frozen global range")
|
||||
packet = struct.pack("<qQ", int(tick), len(nodes)) + np.ascontiguousarray(nodes, dtype="<i4").tobytes()
|
||||
view = memoryview(packet)
|
||||
while view:
|
||||
written = self.process.stdin.write(view)
|
||||
if not written:
|
||||
raise BrokenPipeError("APL sidecar stdin failed")
|
||||
view = view[written:]
|
||||
self.process.stdin.flush()
|
||||
response_tick, byte_count = struct.unpack("<qQ", self.reader.read(16))
|
||||
_check(response_tick == tick and byte_count == (2 + 2 + 4064) * 4,
|
||||
"APL sidecar frame tick/length mismatch")
|
||||
values = np.frombuffer(self.reader.read(byte_count), dtype="<f4")
|
||||
_check(np.isfinite(values).all() and np.all(values >= 0), "APL sidecar returned invalid nonnegative f32 states")
|
||||
self.tick += 1
|
||||
return values[:2], values[2:4], values[4:]
|
||||
|
||||
def finish(self, steps):
|
||||
_check(self.tick == steps, "APL sidecar did not process every tick")
|
||||
self.process.stdin.close()
|
||||
status = self.process.wait(timeout=30)
|
||||
_check(status == 0, f"APL sidecar exited unsuccessfully: {status}")
|
||||
self.reader.thread.join(timeout=5)
|
||||
_check(not self.reader.thread.is_alive(), "APL stdout did not reach EOF")
|
||||
extra = bytes(self.reader.buffer)
|
||||
while not self.reader.chunks.empty():
|
||||
chunk = self.reader.chunks.get_nowait()
|
||||
_check(not isinstance(chunk, BaseException), "APL stdout reader failed after the last frame")
|
||||
extra += chunk
|
||||
_check(not extra, "APL sidecar emitted unrequested trailing stdout bytes")
|
||||
self.process.stdout.close()
|
||||
return status
|
||||
|
||||
def abort(self):
|
||||
if self.process.poll() is None:
|
||||
self.process.terminate()
|
||||
try:
|
||||
self.process.wait(timeout=5)
|
||||
except subprocess.TimeoutExpired:
|
||||
self.process.kill()
|
||||
self.process.wait(timeout=5)
|
||||
for handle in (self.process.stdin, self.process.stdout):
|
||||
if handle is not None and not handle.closed:
|
||||
handle.close()
|
||||
|
||||
|
||||
def _graph_inputs(directory):
|
||||
arrays = {}
|
||||
definitions = {"nodes": "int32", "edge_id": "int64", "src": "int32",
|
||||
"dst": "int32", "signed_raw": "int64", "ptr": "int64", "weights": "float64"}
|
||||
for name, dtype in definitions.items():
|
||||
value = np.load(directory / (name + ".npy"), mmap_mode="r", allow_pickle=False)
|
||||
_check(value.ndim == 1 and value.dtype == np.dtype(dtype), f"Invalid graph array {name}")
|
||||
arrays[name] = value
|
||||
nodes, edge_ids = arrays["nodes"], arrays["edge_id"]
|
||||
n, edges = len(nodes), len(edge_ids)
|
||||
_check(0 < n <= 166700 and edges > 0, "A case needs real global nodes and explicit nonempty connectivity")
|
||||
_check(nodes[0] >= 0 and nodes[-1] < 166700 and np.all(np.diff(nodes) > 0), "Invalid global node order")
|
||||
for name in ("src", "dst", "signed_raw", "weights"):
|
||||
_check(arrays[name].shape == (edges,), f"Graph array {name} has the wrong shape")
|
||||
pointer = arrays["ptr"]
|
||||
_check(pointer.shape == (n + 1,) and pointer[0] == 0 and pointer[-1] == edges and
|
||||
np.all(np.diff(pointer) >= 0), "Invalid graph CSR pointer")
|
||||
source_counts = np.zeros(n, np.int64)
|
||||
identity_edges = True
|
||||
previous_source, previous_edge = -1, -1
|
||||
width = BLOCK_BYTES // 8
|
||||
for begin in range(0, edges, width):
|
||||
stop = min(edges, begin + width)
|
||||
source, target, ids = arrays["src"][begin:stop], arrays["dst"][begin:stop], edge_ids[begin:stop]
|
||||
_check(np.all((source >= 0) & (source < n)) and np.all((target >= 0) & (target < n)),
|
||||
"Graph edge endpoint is outside the local node array")
|
||||
_check(np.all(ids >= 0), "Graph edge IDs must be nonnegative")
|
||||
_check(source[0] > previous_source or (source[0] == previous_source and ids[0] > previous_edge),
|
||||
"Graph edge blocks are not in canonical source/edge order")
|
||||
_check(np.all((source[1:] > source[:-1]) |
|
||||
((source[1:] == source[:-1]) & (ids[1:] > ids[:-1]))),
|
||||
"Graph edges are not in canonical source/edge order")
|
||||
previous_source, previous_edge = int(source[-1]), int(ids[-1])
|
||||
identity_edges &= np.array_equal(ids, np.arange(begin, stop, dtype=np.int64))
|
||||
source_counts += np.bincount(source, minlength=n)
|
||||
_check(np.isfinite(arrays["weights"][begin:stop]).all(), "Nonfinite graph weights")
|
||||
_check(np.array_equal(source_counts, np.diff(pointer)), "CSR pointer and source rows disagree")
|
||||
if not identity_edges:
|
||||
_check(len(np.unique(edge_ids)) == edges, "Original edge IDs must be unique")
|
||||
for name in ("completeness.csv", "connectivity.parquet"):
|
||||
_check((directory / name).is_file(), f"Missing unchanged author input: {name}")
|
||||
if n == 166700:
|
||||
_check(edges == 25582938 and identity_edges and np.array_equal(nodes, np.arange(n, dtype=np.int32)),
|
||||
"A full probe must use the unchanged complete canonical graph")
|
||||
return arrays, identity_edges
|
||||
|
||||
|
||||
def _requests(case_dir, nodes, steps):
|
||||
with np.load(case_dir / "requests.npz", allow_pickle=False) as archive:
|
||||
_check(set(archive.files) == {"tick", "neuron_idx"}, "Unexpected sparse request schema")
|
||||
ticks, requested = archive["tick"].copy(), archive["neuron_idx"].copy()
|
||||
_check(ticks.dtype == requested.dtype == np.dtype("int32") and ticks.ndim == requested.ndim == 1 and
|
||||
ticks.shape == requested.shape, "Requests must be equal-length int32 vectors")
|
||||
_check(np.all((ticks >= 0) & (ticks < steps)), "Request time outside the case")
|
||||
_check(not len(ticks) or np.all((ticks[1:] > ticks[:-1]) |
|
||||
((ticks[1:] == ticks[:-1]) & (requested[1:] > requested[:-1]))),
|
||||
"Requests must be sorted and unique by tick/global node")
|
||||
local = np.searchsorted(nodes, requested)
|
||||
_check(np.all(local < len(nodes)) and np.array_equal(nodes[local], requested),
|
||||
"Request references a global neuron absent from this case")
|
||||
counts = np.bincount(ticks, minlength=steps).astype(np.int32)
|
||||
pointer = np.concatenate((np.zeros(1, np.int64), np.cumsum(counts, dtype=np.int64)))
|
||||
return requested, local.astype(np.int32), pointer, counts
|
||||
|
||||
|
||||
class _QueueProxy:
|
||||
def __init__(self, original, get_tick, available, arrays, identity_edges,
|
||||
due_counts, spike_ring, delay_steps, handle, records, summarize):
|
||||
self._original_queue, self._get_tick = original, get_tick
|
||||
self._available, self._arrays, self._identity_edges = available, arrays, identity_edges
|
||||
self._due_counts, self._spike_ring, self._delay_steps = due_counts, spike_ring, delay_steps
|
||||
self._handle, self._records, self._summarize = handle, records, summarize
|
||||
|
||||
def peek(self, *args, **kwargs):
|
||||
original = self._original_queue.peek(*args, **kwargs)
|
||||
local_edges = np.asarray(original)
|
||||
tick = self._get_tick()
|
||||
_check(tick == len(self._records["rows"]), "Queue peek ticks must be consecutive")
|
||||
_check(local_edges.ndim == 1 and np.all((local_edges >= 0) & (local_edges < len(self._arrays["edge_id"]))),
|
||||
"Original queue contains an invalid local edge index")
|
||||
_check(len(local_edges) == self._due_counts[tick % len(self._due_counts)],
|
||||
"Queue arrivals differ from the fixed-delay outgoing degree of actual spikes")
|
||||
fired_then = np.zeros(len(self._arrays["nodes"]), dtype=bool)
|
||||
if tick >= self._delay_steps:
|
||||
fired_then[self._spike_ring[(tick - self._delay_steps) % len(self._spike_ring)]] = True
|
||||
|
||||
def accepted(begin, stop):
|
||||
part = local_edges[begin:stop]
|
||||
_check(np.all(fired_then[self._arrays["src"][part]]), "Arrival lacks a presynaptic spike at author delay")
|
||||
return self._available[self._arrays["dst"][part]]
|
||||
|
||||
# Only the observed ID view changes; the queue is returned verbatim.
|
||||
edge_ids = local_edges if self._identity_edges else self._arrays["edge_id"][local_edges]
|
||||
summary = self._summarize(tick, edge_ids, accepted)
|
||||
self._handle.write(json.dumps(summary, sort_keys=True) + "\n")
|
||||
self._records["rows"].append(summary)
|
||||
self._records["peek_calls"] += 1
|
||||
return original
|
||||
|
||||
def advance(self, *args, **kwargs):
|
||||
self._records["advance_calls"] += 1
|
||||
return self._original_queue.advance(*args, **kwargs)
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self._original_queue, name)
|
||||
|
||||
|
||||
def run_case(case_dir: Path, out_dir: Path) -> dict:
|
||||
"""Run one continuous case, with exclusive outputs and no candidate trace input.
|
||||
|
||||
Graph inputs are the AF array/author-table schema plus original weights.npy.
|
||||
requests.npz contains sparse tick/neuron_idx int32 vectors. Optional
|
||||
arrays.npz contains initial_v_mv/initial_g_mv float64 vectors only.
|
||||
Full probes write state hashes, sparse events and APL traces, not dense
|
||||
whole-brain histories. Each case requires a fresh process.
|
||||
"""
|
||||
global _PROCESS_USED
|
||||
_check(not _PROCESS_USED, "Every oracle case requires a fresh process")
|
||||
_PROCESS_USED = True
|
||||
case_dir, out_dir = Path(case_dir).resolve(strict=True), Path(out_dir).resolve()
|
||||
spec = json.loads((case_dir / "spec.json").read_text(encoding="utf-8"))
|
||||
graph_dir = Path(spec["graph_dir"]).resolve(strict=True)
|
||||
bundle_dir = Path(spec["apl_bundle_dir"]).resolve(strict=True)
|
||||
worker_path = Path(spec["legacy_apl_worker"]).resolve(strict=True)
|
||||
science_python = _python_entry(spec["science_python"])
|
||||
author_path = Path(spec.get("author_path", AUTHOR_PATH)).resolve(strict=True)
|
||||
ae_path = Path(spec.get("ae_oracle_path", AE_PATH)).resolve(strict=True)
|
||||
af_path = Path(spec.get("af_oracle_path", AF_PATH)).resolve(strict=True)
|
||||
coefficients_path = Path(spec["coefficients_path"]).resolve(strict=True)
|
||||
steps = int(spec["steps"])
|
||||
_check(steps == spec["steps"] and 0 < steps <= np.iinfo(np.int32).max, "Invalid number of steps")
|
||||
_check(float(spec["dt_ms"]) == .1 and np.isclose(float(spec["duration_ms"]), steps * .1, rtol=0, atol=1e-12),
|
||||
"The frozen APL protocol requires dt_ms=0.1 and a matching duration")
|
||||
_check(spec.get("precision", "float64") == "float64", "Author state must be float64")
|
||||
_check(spec.get("training", False) is False and spec.get("poi_used", False) is False,
|
||||
"Training and original poi are outside this gate")
|
||||
_check(spec["apl"] == {"tau_ms": 20, "gain": .01, "normalization": 1}, "Frozen APL parameters changed")
|
||||
record_history = spec["record_history"]
|
||||
_check(isinstance(record_history, bool), "record_history must be a boolean")
|
||||
budget_root = Path(spec.get("budget_root", out_dir.parent)).resolve()
|
||||
budget_limit = int(spec.get("output_budget_bytes", 15 * 1024 ** 3))
|
||||
_check(budget_limit > 0 and out_dir.is_relative_to(budget_root), "Invalid output budget/root")
|
||||
out_dir.mkdir(parents=True, exist_ok=False)
|
||||
mapped_files = []
|
||||
saved_preferences, old_dt, worker = None, None, None
|
||||
last_complete_tick = -1
|
||||
try:
|
||||
arrays, identity_edges = _graph_inputs(graph_dir)
|
||||
nodes, n, edges = arrays["nodes"], len(arrays["nodes"]), len(arrays["edge_id"])
|
||||
_check(n < 166700 or not record_history, "Full probes must not save dense V/g/ref histories")
|
||||
bundle = {name: np.load(bundle_dir / (name + ".npy"), mmap_mode="r", allow_pickle=False)
|
||||
for name in ("input_matrix", "output_matrix", "kc_globals", "apl_globals")}
|
||||
for name, shape in (("input_matrix", (2, 166700)), ("output_matrix", (2, 4064))):
|
||||
value = bundle[name]
|
||||
_check(value.dtype == np.dtype("float32") and value.shape == shape and value.flags.c_contiguous,
|
||||
f"Frozen APL {name} shape/dtype/layout changed")
|
||||
_check(np.isfinite(value).all() and np.all(value >= 0), f"Invalid positive raw-normalized APL {name}")
|
||||
for name, count in (("kc_globals", 4064), ("apl_globals", 2)):
|
||||
value = bundle[name]
|
||||
_check(value.dtype == np.dtype("int32") and value.shape == (count,) and value[0] >= 0 and
|
||||
value[-1] < 166700 and np.all(np.diff(value) > 0), f"Frozen APL identity {name} changed")
|
||||
_check(not np.intersect1d(bundle["kc_globals"], bundle["apl_globals"]).size, "APL and KC identities overlap")
|
||||
kc_positions = np.searchsorted(nodes, bundle["kc_globals"])
|
||||
present = kc_positions < n
|
||||
present[present] &= nodes[kc_positions[present]] == bundle["kc_globals"][present]
|
||||
kc_local, kc_in_bundle = kc_positions[present], np.flatnonzero(present)
|
||||
requested_globals, request_local, request_ptr, request_count = _requests(case_dir, nodes, steps)
|
||||
initial = None
|
||||
if (case_dir / "arrays.npz").exists():
|
||||
with np.load(case_dir / "arrays.npz", allow_pickle=False) as archive:
|
||||
_check(set(archive.files) == {"initial_v_mv", "initial_g_mv"}, "Unexpected initial-state schema")
|
||||
initial = {name: archive[name].copy() for name in archive.files}
|
||||
for name, value in initial.items():
|
||||
_check(value.dtype == np.dtype("float64") and value.shape == (n,) and np.isfinite(value).all(),
|
||||
f"Invalid diagnostic initial state: {name}")
|
||||
history_bytes = steps * n * 40 if record_history else 0
|
||||
_budget(budget_root, budget_limit, history_bytes + steps * (2 + 2 + 4064) * 4 + edges * 8 + (64 << 20))
|
||||
versions = {name: importlib.metadata.version(name) for name in
|
||||
("brian2", "numpy", "pandas", "pyarrow", "sympy", "scipy", "cython")}
|
||||
_save(out_dir / "environment.json", {"python": sys.version, "platform": platform.platform(),
|
||||
"packages": versions, "codegen": "numpy", "author_precision": "float64",
|
||||
"APL_precision": "float32 in separate science-Python process", "Torch_imported_into_Brian": False})
|
||||
for name, required in (("brian2", "2.5.1"), ("numpy", "1.22.3"), ("sympy", "1.12")):
|
||||
_check(versions[name] == required, f"{name} must remain pinned to {required}")
|
||||
import brian2 as b2
|
||||
import pandas as pd
|
||||
from brian2.core.base import BrianObject
|
||||
|
||||
ae = _load_checked(ae_path, AE_SHA256, "phase4ag_sealed_ae_helpers")
|
||||
af = _load_checked(af_path, AF_SHA256, "phase4ag_sealed_af_helpers")
|
||||
author = _load_checked(author_path, AUTHOR_SHA256, "phase4ag_unchanged_author")
|
||||
_check("torch" not in sys.modules, "Torch must not be imported into the pinned Brian process")
|
||||
comp = pd.read_csv(graph_dir / "completeness.csv", index_col=0)
|
||||
_check(np.array_equal(comp.index.to_numpy(), nodes) and
|
||||
np.array_equal(comp["local_index"].to_numpy(), np.arange(n)), "Author CSV neuron identity differs")
|
||||
del comp
|
||||
graph_hashes = {name: _digest(graph_dir / name) for name in
|
||||
tuple(key + ".npy" for key in arrays) + ("completeness.csv", "connectivity.parquet")}
|
||||
bundle_hashes = {name: _digest(bundle_dir / name) for name in BUNDLE_FILES}
|
||||
case_hashes = {name: _digest(case_dir / name) for name in ("spec.json", "requests.npz")}
|
||||
if initial is not None:
|
||||
case_hashes["arrays.npz"] = _digest(case_dir / "arrays.npz")
|
||||
worker_hash = _digest(worker_path)
|
||||
coefficient_hash = _digest(coefficients_path)
|
||||
baseline_coefficients = json.loads(coefficients_path.read_text(encoding="utf-8"))
|
||||
if "legacy_apl_worker_sha256" in spec:
|
||||
_check(worker_hash == spec["legacy_apl_worker_sha256"], "APL worker differs from preregistered source")
|
||||
_save(out_dir / "input-audit.json", {"spec": spec, "case_hashes": case_hashes,
|
||||
"graph_hashes": graph_hashes, "bundle_hashes": bundle_hashes,
|
||||
"author_sha256": AUTHOR_SHA256, "ae_sha256": AE_SHA256, "af_sha256": AF_SHA256,
|
||||
"legacy_apl_worker_sha256": worker_hash, "coefficients_sha256": coefficient_hash,
|
||||
"candidate_trace_used_as_input": False,
|
||||
"scope": "shared-APL-component closed-loop oracle, not all-components-independent CPU equivalence"})
|
||||
saved_preferences = {key: b2.prefs[key] for key in
|
||||
("codegen.target", "core.default_float_dtype", "legacy.refractory_timing")}
|
||||
old_dt = b2.defaultclock.dt
|
||||
b2.start_scope()
|
||||
b2.prefs["codegen.target"] = "numpy"
|
||||
b2.prefs["core.default_float_dtype"] = np.float64
|
||||
b2.prefs["legacy.refractory_timing"] = False
|
||||
b2.defaultclock.dt = .1 * b2.ms
|
||||
parameters = dict(author.default_params)
|
||||
_save(out_dir / "parameter-checks.json", ae._parameters_match(parameters, spec["model"], b2))
|
||||
neu, syn, spikes = author.create_model(graph_dir / "completeness.csv", graph_dir / "connectivity.parquet", parameters)
|
||||
_check(len(neu) == n and len(syn) == edges, "Author loaded an unexpected graph size")
|
||||
_check(all(np.dtype(neu.variables[key].dtype) == np.float64 for key in ("v", "g")) and
|
||||
np.dtype(syn.variables["w"].dtype) == np.float64, "Author states/weights are not float64")
|
||||
effective = _new_memmap(out_dir / "weights.npy", "<f8", (edges,))
|
||||
mapped_files.append(effective)
|
||||
width, w_unit, mask_count = BLOCK_BYTES // 8, float(parameters["w_syn"] / b2.volt), 0
|
||||
mask_hasher = hashlib.sha256()
|
||||
base_hasher = hashlib.sha256()
|
||||
effective_hasher = hashlib.sha256()
|
||||
for begin in range(0, edges, width):
|
||||
stop = min(edges, begin + width)
|
||||
np.testing.assert_array_equal(np.asarray(syn.i[begin:stop]), arrays["src"][begin:stop])
|
||||
np.testing.assert_array_equal(np.asarray(syn.j[begin:stop]), arrays["dst"][begin:stop])
|
||||
original = np.asarray(syn.w_[begin:stop], dtype=np.float64).copy()
|
||||
raw_expected = arrays["signed_raw"][begin:stop] * w_unit
|
||||
_check(np.array_equal(original.view(np.uint64), raw_expected.view(np.uint64)) and
|
||||
np.array_equal(original.view(np.uint64), arrays["weights"][begin:stop].view(np.uint64)),
|
||||
"Actual original Brian weights differ from unchanged signed raw or AF base weights")
|
||||
base_hasher.update(np.ascontiguousarray(original, dtype="<f8").tobytes())
|
||||
mask = np.isin(nodes[arrays["src"][begin:stop]], bundle["apl_globals"])
|
||||
mask_count += int(mask.sum())
|
||||
mask_hasher.update(np.ascontiguousarray(arrays["edge_id"][begin:stop][mask], dtype="<i8").tobytes())
|
||||
original[mask] = np.float64(0.0)
|
||||
effective_hasher.update(np.ascontiguousarray(original, dtype="<f8").tobytes())
|
||||
effective[begin:stop] = original
|
||||
syn.w_[begin:stop] = original
|
||||
if n == 166700:
|
||||
_check(mask_count == 5845, "Full legacy APL fast-edge mask must contain exactly 5845 existing edges")
|
||||
effective.flush()
|
||||
_save(out_dir / "apl-mask-audit.json", {"count": mask_count, "ordered_original_edge_id_sha256": mask_hasher.hexdigest(),
|
||||
"base_weight_raw_bytes_sha256": base_hasher.hexdigest(), "effective_weights_file_sha256": _digest(out_dir / "weights.npy"),
|
||||
"effective_weights_raw_sha256": effective_hasher.hexdigest(),
|
||||
"rule": "zero +0 all existing fast edges whose source GLOBAL neuron is in apl_globals; no graph edge removal",
|
||||
"graded_targets": "only KC identities present in this LIF case", "KC_targets_in_case": len(kc_local),
|
||||
"base_graph_files_modified": False, "fullgraph_required_mask_count": 5845})
|
||||
if initial is not None:
|
||||
neu.v = initial["initial_v_mv"] * b2.mV
|
||||
neu.g = initial["initial_g_mv"] * b2.mV
|
||||
v_raw = neu.variables["v"].get_value()
|
||||
g_raw = neu.variables["g"].get_value()
|
||||
last_raw = neu.variables["lastspike"].get_value()
|
||||
available_raw = neu.variables["not_refractory"].get_value()
|
||||
histories = {}
|
||||
pre_apl_histories = {}
|
||||
if record_history:
|
||||
for name, dtype in (("v", "float64"), ("g", "float64"), ("ref", "int32")):
|
||||
histories[name] = _new_memmap(out_dir / f"{name}-history.{dtype}.npy", np.dtype(dtype), (steps, n))
|
||||
mapped_files.append(histories[name])
|
||||
pre_apl_histories[name] = _new_memmap(out_dir / f"preAPL-{name}.{dtype}.npy", np.dtype(dtype), (steps, n))
|
||||
mapped_files.append(pre_apl_histories[name])
|
||||
apl_history = {}
|
||||
for name, count in (("drive", 2), ("state", 2), ("inhibition", 4064)):
|
||||
apl_history[name] = _new_memmap(out_dir / f"apl-{name}.float32.npy", "<f4", (steps, count))
|
||||
mapped_files.append(apl_history[name])
|
||||
_save(out_dir / "output-schema.json", {"history_recorded": record_history, "history_shape": [steps, n],
|
||||
"state_hash_file": "tick-state-hashes.jsonl", "state_hash_encoding": "SHA256 of contiguous little-endian raw f64 V/g and i32 ref",
|
||||
"APL_dtype": "float32", "APL_h_units": "mV; gain already applied in float32",
|
||||
"APL_SI_conversion": "h.astype(float64) * float64(0.001), then separate float64 g subtraction",
|
||||
"APL_write_schedule": "end, order=0, after original reset, unconditional raw g write; recorder end/order=1",
|
||||
"APL_input": "Brian's own actual global spikes, never candidate spikes or candidate inhibition",
|
||||
"APL_shapes": {"drive": [steps, 2], "state": [steps, 2], "inhibition": [steps, 4064]},
|
||||
"request_columns": ["tick", "neuron_idx", "attempted", "accepted", "rejected"],
|
||||
"queue_protocol": "sealed Phase4AF queue_summary; original global edge IDs, including zero-weight/ref-rejected arrivals",
|
||||
"state_npz": ["step", "neuron_idx", "counts", "v", "g", "ref"],
|
||||
"complete_marker": "summary.json complete=true only after worker exit and all checks"})
|
||||
delay_steps = int(round(float(parameters["t_dly"] / b2.ms) / .1))
|
||||
ref_steps = int(round(float(parameters["t_rfc"] / b2.ms) / .1))
|
||||
_check(delay_steps == 18 and ref_steps == 22, "Author delay/refractory changed")
|
||||
queue_len = delay_steps + 1
|
||||
due_counts = np.zeros(queue_len, np.int64)
|
||||
spike_ring = [np.empty(0, np.int32) for _ in range(queue_len)]
|
||||
outdegree = np.diff(arrays["ptr"])
|
||||
request_seen = np.zeros(steps, np.uint8)
|
||||
apl_seen = np.zeros(steps, np.uint8)
|
||||
applicable_count = np.zeros(steps, np.int32)
|
||||
rejected_count = np.zeros(steps, np.int32)
|
||||
spike_count = np.zeros(steps, np.int32)
|
||||
total_counts = np.zeros(n, np.int32)
|
||||
records = {"rows": [], "peek_calls": 0, "advance_calls": 0, "installations": []}
|
||||
last_ticks = np.zeros(n, np.int64)
|
||||
final_ref = np.zeros(n, np.int32)
|
||||
permitted = np.empty(0, np.int32)
|
||||
total_events = 0
|
||||
|
||||
def tick_now():
|
||||
return int(round(float(neu.clock.t / b2.second) / .0001))
|
||||
|
||||
def current_ref(tick):
|
||||
seen = np.isfinite(last_raw) & (last_raw >= 0)
|
||||
last_ticks.fill(0)
|
||||
last_ticks[seen] = np.rint(last_raw[seen] / .0001).astype(np.int64)
|
||||
result = np.where(seen, np.maximum(ref_steps - (tick - last_ticks), 0), 0).astype(np.int32)
|
||||
_check(np.array_equal(result == 0, available_raw), "Ref history disagrees with author not_refractory")
|
||||
return result
|
||||
|
||||
with ExitStack() as stack:
|
||||
stderr = stack.enter_context((out_dir / "apl-worker.stderr.log").open("xb"))
|
||||
worker = _APLWorker(science_python, worker_path, bundle_dir, stderr, float(spec.get("worker_timeout_seconds", 180)))
|
||||
_save(out_dir / "apl-worker-handshake.json", {"command": worker.command, "handshake": worker.hello})
|
||||
queue_file = stack.enter_context((out_dir / "queue.jsonl").open("x", encoding="utf-8"))
|
||||
event_file = stack.enter_context((out_dir / "events.bin").open("xb"))
|
||||
hash_file = stack.enter_context((out_dir / "tick-state-hashes.jsonl").open("x", encoding="utf-8"))
|
||||
tick_file = stack.enter_context((out_dir / "ticks.jsonl").open("x", encoding="utf-8"))
|
||||
requests_file = stack.enter_context((out_dir / "requests.csv").open("x", encoding="utf-8", newline=""))
|
||||
request_writer = csv.DictWriter(requests_file, fieldnames=["tick", "neuron_idx", "attempted", "accepted", "rejected"])
|
||||
request_writer.writeheader()
|
||||
|
||||
def request_operation():
|
||||
nonlocal permitted
|
||||
tick = tick_now()
|
||||
_check(0 <= tick < steps and not request_seen[tick], "Invalid/duplicate common request tick")
|
||||
request_seen[tick] = 1
|
||||
begin, stop = request_ptr[tick:tick + 2]
|
||||
local = request_local[begin:stop]
|
||||
available = available_raw[local].copy()
|
||||
permitted = local[available]
|
||||
applicable_count[tick] = len(permitted)
|
||||
rejected_count[tick] = len(local) - len(permitted)
|
||||
for node, accepted in zip(requested_globals[begin:stop], available):
|
||||
request_writer.writerow({"tick": tick, "neuron_idx": int(node), "attempted": 1,
|
||||
"accepted": int(accepted), "rejected": int(not accepted)})
|
||||
if len(permitted):
|
||||
neu.v[permitted] = parameters["v_th"] + 1 * b2.mV
|
||||
|
||||
def apl_operation():
|
||||
tick = tick_now()
|
||||
_check(0 <= tick < steps and not apl_seen[tick] and request_seen[tick], "Invalid/duplicate APL tick")
|
||||
active = np.asarray(neu.spikes, dtype=np.int32)
|
||||
_check(not len(active) or np.all(np.diff(active) > 0), "Brian actual spikes are not canonical/unique")
|
||||
_check(np.all(g_raw[active] == 0.0), "APL ran before the original author g reset")
|
||||
if record_history:
|
||||
pre_apl_histories["v"][tick] = v_raw
|
||||
pre_apl_histories["g"][tick] = g_raw
|
||||
pre_apl_histories["ref"][tick] = current_ref(tick)
|
||||
drive, state, inhibition = worker.step(tick, np.asarray(nodes[active], dtype=np.int32))
|
||||
apl_history["drive"][tick] = drive
|
||||
apl_history["state"][tick] = state
|
||||
apl_history["inhibition"][tick] = inhibition
|
||||
# These are separate array operations. Do not move the unit
|
||||
# conversion into a fused subtract; do not guard by refractory.
|
||||
delta_si = inhibition[kc_in_bundle].astype(np.float64) * np.float64(.001)
|
||||
g_raw[kc_local] = g_raw[kc_local] - delta_si
|
||||
apl_seen[tick] = 1
|
||||
|
||||
def record_end():
|
||||
nonlocal last_complete_tick, total_events, final_ref
|
||||
tick = tick_now()
|
||||
_check(tick == last_complete_tick + 1 and tick < steps and apl_seen[tick], "End tick is incomplete/out of order")
|
||||
_check(len(records["rows"]) == tick + 1, "End state lacks queue observation")
|
||||
_check(np.isfinite(v_raw).all() and np.isfinite(g_raw).all(), "Nonfinite author/APL state")
|
||||
final_ref = current_ref(tick)
|
||||
if record_history:
|
||||
histories["v"][tick], histories["g"][tick], histories["ref"][tick] = v_raw, g_raw, final_ref
|
||||
active = np.asarray(neu.spikes, dtype=np.int32)
|
||||
_check(np.isin(permitted, active).all(), "An applicable request did not produce a real spike")
|
||||
spike_count[tick] = len(active)
|
||||
total_counts[active] += 1
|
||||
if len(active):
|
||||
events = np.empty(len(active), EVENT_DTYPE)
|
||||
events["step"], events["neuron_idx"] = tick, nodes[active]
|
||||
event_file.write(memoryview(events).cast("B"))
|
||||
total_events += len(active)
|
||||
spike_ring[tick % queue_len] = active.copy()
|
||||
due_counts[(tick + delay_steps) % queue_len] = int(outdegree[active].sum(dtype=np.int64))
|
||||
hash_file.write(json.dumps({"tick": tick, "v_sha256": _array_hash(v_raw, "<f8"),
|
||||
"g_sha256": _array_hash(g_raw, "<f8"), "ref_sha256": _array_hash(final_ref, "<i4")}, sort_keys=True) + "\n")
|
||||
tick_file.write(json.dumps({"tick": tick, "spikes": len(active), "requests": int(request_count[tick]),
|
||||
"applicable_requests": int(applicable_count[tick]), "ref_rejected_requests": int(rejected_count[tick]),
|
||||
"event_sha256": _array_hash(nodes[active], "<i4"),
|
||||
"due_arrivals": records["rows"][tick]["count"], "accepted_arrivals": records["rows"][tick]["accepted_count"],
|
||||
"APL_drive_sha256": _array_hash(apl_history["drive"][tick], "<f4"),
|
||||
"APL_state_sha256": _array_hash(apl_history["state"][tick], "<f4"),
|
||||
"APL_inhibition_sha256": _array_hash(apl_history["inhibition"][tick], "<f4")}, sort_keys=True) + "\n")
|
||||
_budget(budget_root, budget_limit, reserve=1 << 20)
|
||||
last_complete_tick = tick
|
||||
if (tick + 1) % 100 == 0 or tick + 1 == steps:
|
||||
for mapped in mapped_files:
|
||||
mapped.flush()
|
||||
for handle in (queue_file, event_file, hash_file, tick_file, requests_file):
|
||||
handle.flush()
|
||||
print(json.dumps({"oracle": spec["name"], "completed_ticks": tick + 1,
|
||||
"spikes": total_events, "APL_worker_ticks": worker.tick}), flush=True)
|
||||
|
||||
class QueueInstaller(BrianObject):
|
||||
def __init__(self):
|
||||
super().__init__(clock=neu.clock, when="end", order=1_000_000, name="zz_apl_queue_installer")
|
||||
|
||||
def before_run(self, run_namespace):
|
||||
installed = []
|
||||
for code in syn.pre.code_objects:
|
||||
if "_queue" not in code.namespace or "_queue.peek()" not in str(code.code):
|
||||
continue
|
||||
original = code.namespace["_queue"]
|
||||
if isinstance(original, _QueueProxy):
|
||||
original = original._original_queue
|
||||
code.namespace["_queue"] = _QueueProxy(original, tick_now, available_raw, arrays, identity_edges,
|
||||
due_counts, spike_ring, delay_steps, queue_file, records, af.queue_summary)
|
||||
installed.append(code.name)
|
||||
_check(len(installed) == 1, "Expected one original anatomical queue consumer")
|
||||
records["installations"].append(installed[0])
|
||||
|
||||
objects = [neu, syn, spikes,
|
||||
b2.NetworkOperation(request_operation, clock=neu.clock, when="before_thresholds", order=0, name="oracle_common_requests"),
|
||||
b2.NetworkOperation(apl_operation, clock=neu.clock, when="end", order=0, name="oracle_legacy_apl_unconditional"),
|
||||
b2.NetworkOperation(record_end, clock=neu.clock, when="end", order=1, name="oracle_after_apl_record"), QueueInstaller()]
|
||||
network = b2.Network(*objects)
|
||||
_save(out_dir / "schedule.json", {"schedule": list(network.schedule), "summary": str(network.scheduling_summary())})
|
||||
network.run(0 * b2.ms, namespace={})
|
||||
_save(out_dir / "compiled-code.json", ae._code_records(network))
|
||||
compiled_coefficients = ae._coefficients(neu, parameters, b2, .0001)
|
||||
coefficient_keys = ("gd", "c0", "cg", "cv", "threshold", "reset", "seed_volt")
|
||||
_check(all(float(compiled_coefficients[key]).hex() == float(baseline_coefficients[key]).hex()
|
||||
for key in coefficient_keys), "Compiled Brian coefficients differ from sealed AF")
|
||||
_check(last_complete_tick == -1 and worker.tick == 0 and not records["rows"] and not request_seen.any(),
|
||||
"Zero-duration compilation unexpectedly advanced state")
|
||||
network.run(steps * .1 * b2.ms, namespace={})
|
||||
_save(out_dir / "generated-code.json", ae._code_records(network))
|
||||
worker_status = worker.finish(steps)
|
||||
worker = None
|
||||
|
||||
_check(last_complete_tick == steps - 1 and np.all(request_seen == 1) and np.all(apl_seen == 1), "Incomplete trial")
|
||||
_check(records["peek_calls"] == records["advance_calls"] == steps, "Incomplete queue audit")
|
||||
_check(total_events == int(spikes.num_spikes) and np.array_equal(total_counts, np.asarray(spikes.count[:])),
|
||||
"Sparse events disagree with original SpikeMonitor")
|
||||
_check(np.array_equal(request_count, applicable_count + rejected_count), "Incomplete request accounting")
|
||||
_check(np.all(np.asarray(neu.rfc[:] / b2.ms) == float(parameters["t_rfc"] / b2.ms)), "Author refractory duration changed")
|
||||
for begin in range(0, edges, width):
|
||||
stop = min(edges, begin + width)
|
||||
_check(np.array_equal(np.asarray(syn.w_[begin:stop]).view(np.uint64), effective[begin:stop].view(np.uint64)),
|
||||
"Effective synaptic weights changed during the trial")
|
||||
event_values = np.memmap(out_dir / "events.bin", dtype=EVENT_DTYPE, mode="r", shape=(total_events,)) if total_events else np.empty(0, EVENT_DTYPE)
|
||||
_check((out_dir / "events.bin").stat().st_size == total_events * 8, "Sparse event byte count mismatch")
|
||||
for begin in range(0, total_events, width):
|
||||
stop = min(total_events, begin + width)
|
||||
spike_local = np.asarray(spikes.i[begin:stop], dtype=np.int32)
|
||||
np.testing.assert_array_equal(event_values["neuron_idx"][begin:stop], nodes[spike_local])
|
||||
times = np.rint(np.asarray(spikes.t[begin:stop] / b2.second) / .0001).astype(np.int32)
|
||||
np.testing.assert_array_equal(event_values["step"][begin:stop], times)
|
||||
_budget(budget_root, budget_limit, total_events * 8 + n * 24 + (8 << 20))
|
||||
with (out_dir / "state.npz").open("xb") as handle:
|
||||
np.savez_compressed(handle, step=event_values["step"], neuron_idx=event_values["neuron_idx"],
|
||||
counts=total_counts, v=v_raw, g=g_raw, ref=final_ref)
|
||||
with (out_dir / "tick-summary.npz").open("xb") as handle:
|
||||
np.savez_compressed(handle, request_count=request_count, applicable_count=applicable_count,
|
||||
rejected_count=rejected_count, spike_count=spike_count,
|
||||
arrival_count=np.asarray([row["count"] for row in records["rows"]], np.int64),
|
||||
accepted_arrival_count=np.asarray([row["accepted_count"] for row in records["rows"]], np.int64))
|
||||
coefficients = ae._coefficients(neu, parameters, b2, .0001)
|
||||
keys = ("gd", "c0", "cg", "cv", "threshold", "reset", "seed_volt")
|
||||
_check(all(float(coefficients[key]).hex() == float(baseline_coefficients[key]).hex() for key in keys),
|
||||
"Actual Brian coefficients differ from sealed AF coefficients")
|
||||
_save(out_dir / "coefficients.json", coefficients)
|
||||
_save(out_dir / "queue-audit.json", {"peek_calls": records["peek_calls"], "advance_calls": records["advance_calls"],
|
||||
"installations": records["installations"], "same_peek_object_returned": True,
|
||||
"original_edge_ids_not_sorted_or_filtered": True, "zero_weight_and_refractory_rejected_edges_retained": True,
|
||||
"post_threshold_pre_synaptic_guard_observed": True, "forwarding_contract": "sealed AE/AF design",
|
||||
"new_unwrapped_repeat_performed": False, "limit": "Current wrapper was not independently rerun without the queue proxy"})
|
||||
for path, expected in ((author_path, AUTHOR_SHA256), (ae_path, AE_SHA256), (af_path, AF_SHA256), (worker_path, worker_hash)):
|
||||
_check(_digest(path) == expected, f"Source changed during the run: {path}")
|
||||
_check(_digest(coefficients_path) == coefficient_hash, "Sealed coefficient reference changed during the run")
|
||||
for directory, hashes in ((graph_dir, graph_hashes), (bundle_dir, bundle_hashes), (case_dir, case_hashes)):
|
||||
for name, expected in hashes.items():
|
||||
_check(_digest(directory / name) == expected, f"Read-only input changed during the run: {name}")
|
||||
for mapped in mapped_files:
|
||||
mapped.flush()
|
||||
_check("torch" not in sys.modules, "Pinned Brian process unexpectedly imported Torch")
|
||||
output_bytes = _budget(budget_root, budget_limit, 1 << 20)
|
||||
summary = {"name": spec["name"], "complete": True, "precision": "float64", "APL_precision": "float32",
|
||||
"backend": "shared-APL-component closed-loop Brian2 oracle", "steps": steps, "completed_ticks": last_complete_tick + 1,
|
||||
"nodes": n, "edges": edges, "spikes": total_events, "requests": int(request_count.sum()),
|
||||
"applicable_requests": int(applicable_count.sum()), "ref_rejected_requests": int(rejected_count.sum()),
|
||||
"APL_fast_edges_zeroed": mask_count, "APL_worker_exit_status": worker_status, "APL_worker_ticks": steps,
|
||||
"apl_fast_edges_zeroed": mask_count, "effective_weights_raw_sha256": effective_hasher.hexdigest(),
|
||||
"kc_targets_present": len(kc_local),
|
||||
"candidate_trace_used_as_input": False, "independent_CPU_all_components_claim": False,
|
||||
"history_recorded": record_history, "single_continuous_run": True, "no_training": True, "no_poi": True,
|
||||
"source_and_inputs_unchanged": True, "pass_all": True,
|
||||
"pass_all_scope": "Oracle construction, protocol and internal self-checks only; candidate comparison is a separate gate",
|
||||
"scope": "Only this fixed trajectory; legacy Torch APL is a shared numerical component, not an independent CPU APL implementation",
|
||||
"output_budget_bytes": budget_limit, "output_bytes_before_summary": output_bytes,
|
||||
"state_hashes_sha256": _digest(out_dir / "tick-state-hashes.jsonl"), "queue_sha256": _digest(out_dir / "queue.jsonl"),
|
||||
"events_sha256": _digest(out_dir / "events.bin"), "weights_sha256": _digest(out_dir / "weights.npy")}
|
||||
_save(out_dir / "summary.json", summary)
|
||||
return summary
|
||||
except Exception as error:
|
||||
for mapped in mapped_files:
|
||||
mapped.flush()
|
||||
_save(out_dir / "error.json", {"complete": False, "last_complete_tick": last_complete_tick,
|
||||
"type": type(error).__name__, "message": str(error), "partial_files_preserved": True,
|
||||
"inputs_or_duration_adjusted": False})
|
||||
raise
|
||||
finally:
|
||||
if worker is not None:
|
||||
worker.abort()
|
||||
if saved_preferences is not None:
|
||||
for key, value in saved_preferences.items():
|
||||
b2.prefs[key] = value
|
||||
b2.defaultclock.dt = old_dt
|
||||
|
||||
@@ -0,0 +1,172 @@
|
||||
"""Preregister exact legacy APL data and unmodified fixed CAL requests."""
|
||||
from datetime import datetime,timezone
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import subprocess
|
||||
import sys
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
from context import ROOT,MODEL,EXP,AF_MODEL,AF_EXP,AD_EXP,CFG,save,digest,sources,integrity,budget
|
||||
|
||||
def arr(path,value):
|
||||
with Path(path).open('xb') as f:np.save(f,value,allow_pickle=False)
|
||||
|
||||
def graph(folder,nodes,full):
|
||||
folder.mkdir();nodes=np.asarray(nodes,np.int32);ptr,dst,src,raw,base=full
|
||||
if len(nodes)==CFG['neurons']:ids=np.arange(len(dst),dtype=np.int64)
|
||||
else:
|
||||
ids=np.concatenate([np.arange(ptr[s],ptr[s+1],dtype=np.int64) for s in nodes])
|
||||
ids=ids[np.isin(dst[ids],nodes)]
|
||||
lookup=np.full(CFG['neurons'],-1,np.int32);lookup[nodes]=np.arange(len(nodes),dtype=np.int32)
|
||||
ss,dd=lookup[src[ids]],lookup[dst[ids]];pp=np.r_[np.int64(0),np.bincount(ss,minlength=len(nodes)).cumsum(dtype=np.int64)]
|
||||
for name,value in dict(nodes=nodes,edge_id=ids,src=ss,dst=dd,ptr=pp,signed_raw=raw[ids],weights=base[ids]).items():arr(folder/(name+'.npy'),value)
|
||||
pd.DataFrame({'local_index':np.arange(len(nodes),dtype=np.int32)},index=pd.Index(nodes,name='global_neuron_idx')).to_csv(folder/'completeness.csv')
|
||||
pd.DataFrame({'Presynaptic_Index':ss,'Postsynaptic_Index':dd,'Excitatory x Connectivity':raw[ids],'edge_id':ids},copy=False).to_parquet(folder/'connectivity.parquet',index=False)
|
||||
save(folder/'audit.json',dict(nodes=len(nodes),edges=len(ids),canonical_original_edge_ids=True,anatomical_weights_not_modified=True,
|
||||
effective_APL_zeroing='Only in new runtime clone, never these original weight files'))
|
||||
return nodes
|
||||
|
||||
def case(name,kind,steps,nodes,graphdir,requests,initial=None,extra=None):
|
||||
folder=EXP/'cases'/name;folder.mkdir()
|
||||
model=json.loads((AD_EXP/'protocol/preregistered.json').read_text())['config']['model']
|
||||
spec=dict(name=name,kind=kind,steps=steps,dt_ms=.1,duration_ms=steps*.1,model=model,
|
||||
graph_dir=str(graphdir),apl_bundle_dir=str(EXP/'apl-bundle'),apl=CFG['apl'],record_history=kind=='fixture',
|
||||
coefficients_path=str(AF_EXP/'oracle/no_input/coefficients.json'),
|
||||
science_python=str(ROOT/'models/male-cns-lif/.venv/bin/python'),legacy_apl_worker=str(MODEL/'legacy_apl_worker.py'),
|
||||
budget_root=str(EXP),output_budget_bytes=CFG['output_budget_bytes'],precision='float64',apl_precision='float32',training=False)
|
||||
if extra:spec.update(extra)
|
||||
requests=np.asarray(requests,dtype=np.int32).reshape(-1,2)
|
||||
if len(requests):
|
||||
requests=requests[np.lexsort((requests[:,1],requests[:,0]))]
|
||||
assert len(np.unique(requests.astype(np.int64)[:,0]*CFG['neurons']+requests[:,1]))==len(requests)
|
||||
assert requests[:,0].min()>=0 and requests[:,0].max()<steps and np.isin(requests[:,1],nodes).all()
|
||||
with (folder/'requests.npz').open('xb') as f:np.savez_compressed(f,tick=requests[:,0],neuron_idx=requests[:,1])
|
||||
if kind=='fixture':
|
||||
v=np.full(len(nodes),model['resting_potential_mv'],np.float64);g=np.zeros(len(nodes),np.float64)
|
||||
if initial:
|
||||
for node,value in initial.get('v',{}).items():v[np.flatnonzero(nodes==node)[0]]=value
|
||||
for node,value in initial.get('g',{}).items():g[np.flatnonzero(nodes==node)[0]]=value
|
||||
with (folder/'arrays.npz').open('xb') as f:np.savez_compressed(f,initial_v_mv=v,initial_g_mv=g)
|
||||
save(folder/'spec.json',spec)
|
||||
|
||||
def main():
|
||||
assert not (EXP/'protocol/preregistered.json').exists()
|
||||
assert all(not (EXP/d).exists() or not any((EXP/d).iterdir()) for d in ('protocol','graphs','apl-bundle','cases','gpu','oracle'))
|
||||
assert shutil.disk_usage(EXP.parent).free>30*1024**3
|
||||
for d in ('protocol','apl-bundle','graphs','cases','gpu','oracle','component-audit','regression','calibration','audit','extension-build','logs'):(EXP/d).mkdir(parents=True,exist_ok=True)
|
||||
can=ROOT/'datasets/male-cns-v1.0/processed/neuron-only';legacy=ROOT/'models/male-cns-lif';lp=legacy/'olfactory-grounding-v1/phase4l.yaml'
|
||||
lock=ROOT/'experiments/phase4j-mb-physiology-repair/calibration/locked-parameters.json'
|
||||
files=[AF_MODEL/'manifest.json',AF_EXP/'manifest.json',AF_EXP/'oracle/no_input/coefficients.json',
|
||||
legacy/'deterministic-runtime-v2/phase4p_revalidate.py',lp,lock,can/'neuron-index/unified-metadata.parquet',can/'gpu/csr-weights.int64.npy',
|
||||
legacy/'author-runtime-lab-v1/author_ext.cpp',legacy/'author-runtime-lab-v1/author_ext.cu',legacy/'author-runtime-lab-v1/gpu_backend.py',
|
||||
legacy/'author-runtime-lab-v1/oracle_stress.py',AD_EXP/'external-resources/model.py',AF_MODEL/'oracle_full.py']
|
||||
lroot=ROOT/'experiments/phase4l-olfactory-grounding'
|
||||
files += [lroot/'odor-encoder'/x for x in ('frozen-odor-panel.parquet','mapped-glomerular-vectors.npz','selection.json')]
|
||||
files += [lroot/'receptor-mapping/olfactory-mapping.parquet']
|
||||
prior=ROOT/'experiments/phase4ag-author-apl'
|
||||
files += [prior/'manifest.json',ROOT/'models/male-cns-lif/author-apl-lab-v1/manifest.json']
|
||||
save(EXP/'protocol/preregistered.json',dict(timestamp_utc=datetime.now(timezone.utc).isoformat(),sources=sources(),config=CFG,
|
||||
inputs=[dict(path=str(p.relative_to(ROOT)),sha256=digest(p)) for p in files],before_any_trajectory=True,
|
||||
parent_commit=subprocess.check_output(['git','-C',str(ROOT),'rev-parse','HEAD'],text=True).strip()))
|
||||
save(EXP/'regression/old-integrity-before.json',integrity())
|
||||
science_entry=ROOT/'models/male-cns-lif/.venv/bin/python'
|
||||
command=[str(science_entry),'-B','-c',
|
||||
'import sys,json,importlib.util; print(json.dumps(dict(executable=sys.executable,prefix=sys.prefix,base_prefix=sys.base_prefix,numpy_visible=importlib.util.find_spec("numpy") is not None,torch_visible=importlib.util.find_spec("torch") is not None)))']
|
||||
environment=json.loads(subprocess.check_output(command,text=True))
|
||||
assert environment['executable']==str(science_entry) and environment['prefix']==str(science_entry.parent.parent)
|
||||
assert environment['prefix']!=environment['base_prefix'] and environment['numpy_visible'] and environment['torch_visible']
|
||||
save(EXP/'protocol/execution-preflight.json',dict(command=command,observed=environment,
|
||||
before_any_trajectory=True,torch_imported=False,CUDA_initialized=False))
|
||||
locked=json.loads(lock.read_text());assert locked['parameters_locked'] and locked['calibration_only']
|
||||
assert {k:locked['selected'][k] for k in ('gain','normalization','tau_ms')}==CFG['apl']
|
||||
meta=pd.read_parquet(can/'neuron-index/unified-metadata.parquet').sort_values('neuron_idx').reset_index(drop=True)
|
||||
assert len(meta)==CFG['neurons'] and np.array_equal(meta.neuron_idx,np.arange(CFG['neurons']))
|
||||
cls=meta.annotation_class.fillna('').astype(str).to_numpy();typ=meta.annotation_type.fillna('').astype(str).to_numpy()
|
||||
kc=np.flatnonzero(cls=='Kenyon_Cell').astype(np.int32);pn=np.flatnonzero(cls=='ALPN').astype(np.int32);apl=np.flatnonzero(typ=='APL').astype(np.int32)
|
||||
assert len(kc)==4064 and len(apl)==2
|
||||
fg=AF_EXP/'graph';ptr=np.load(fg/'ptr.npy',mmap_mode='r');dst=np.load(fg/'dst.npy',mmap_mode='r');src=np.load(fg/'src.npy',mmap_mode='r')
|
||||
signed=np.load(fg/'signed_raw.npy',mmap_mode='r');base=np.load(AF_EXP/'oracle/no_input/weights.npy',mmap_mode='r');raw=np.load(can/'gpu/csr-weights.int64.npy',mmap_mode='r')
|
||||
assert len(dst)==CFG['edges'] and int(raw.sum())==CFG['raw_weight_sum']
|
||||
aplout=np.isin(src,apl);aplids=np.flatnonzero(aplout).astype(np.int64);assert len(aplids)==5845
|
||||
I=np.zeros((2,len(meta)),np.float32);O=np.zeros((2,len(kc)),np.float32)
|
||||
alook={int(v):i for i,v in enumerate(apl)};kl=np.full(len(meta),-1,np.int32);kl[kc]=np.arange(len(kc),dtype=np.int32)
|
||||
allowed=np.isin(src,np.r_[kc,pn])&np.isin(dst,apl)
|
||||
for eid in np.flatnonzero(allowed):I[alook[int(dst[eid])],int(src[eid])]+=float(raw[eid])
|
||||
for eid in np.flatnonzero(aplout&np.isin(dst,kc)):O[alook[int(src[eid])],kl[dst[eid]]]+=float(raw[eid])
|
||||
iscale=float(I[I>0].sum()/np.count_nonzero(np.any(I>0,axis=0)));oscale=float(O[O>0].mean());bundle=EXP/'apl-bundle'
|
||||
for name,value in dict(input_matrix=I/iscale,output_matrix=O/oscale,input_raw=I,output_raw=O,kc_globals=kc,pn_globals=pn,apl_globals=apl,apl_out_ids=aplids).items():arr(bundle/(name+'.npy'),value)
|
||||
save(bundle/'scales.json',dict(input_scale=iscale,output_scale=oscale))
|
||||
save(bundle/'provenance.json',dict(canonical_root=str(can),source=str(legacy/'deterministic-runtime-v2/phase4p_revalidate.py'),source_sha256=digest(legacy/'deterministic-runtime-v2/phase4p_revalidate.py'),
|
||||
parameters=CFG['apl'],lock_file=str(lock),lock_sha256=digest(lock),full_dimensions=[list(I.shape),list(O.shape)],
|
||||
APL=len(apl),KC=len(kc),ALPN=len(pn),APL_fast_edges=len(aplids),APL_nonKC_edges=int(np.count_nonzero(aplout&~np.isin(dst,kc))),
|
||||
APL_nonKC_contacts=int(raw[aplout&~np.isin(dst,kc)].sum()),normalization_recomputed_in_original_order=True,
|
||||
raw_files_modified=False))
|
||||
full=(ptr,dst,src,signed,base);fullnodes=graph(EXP/'graphs/full',np.arange(len(meta),dtype=np.int32),full)
|
||||
pscore=I.sum(axis=0);pnrank=pn[np.lexsort((pn,-pscore[pn]))];kscore=O.sum(axis=0);kcrank=kc[np.lexsort((kc,-kscore))]
|
||||
candidates=np.flatnonzero(np.isin(src,pn)&np.isin(dst,kc));best=int(candidates[np.argmax(raw[candidates])]);pairpn=int(src[best])
|
||||
pe=np.arange(ptr[pairpn],ptr[pairpn+1],dtype=np.int64);pe=pe[np.isin(dst[pe],kc)];pe=pe[np.lexsort((dst[pe],-raw[pe]))]
|
||||
collision_kc=np.unique(dst[pe[:3]]);assert len(collision_kc)==3
|
||||
nonkc=aplids[~np.isin(dst[aplids],kc)];nonkc_target=int(dst[nonkc[np.argmax(raw[nonkc])]])
|
||||
nodes=np.unique(np.r_[apl,pnrank[:16],kcrank[:16],pairpn,collision_kc,nonkc_target,[1197,1381,1511,82312,133666]]).astype(np.int32)
|
||||
fixturegraph=EXP/'graphs/fixtures';graph(fixturegraph,nodes,full)
|
||||
p=int(pnrank[0]);k=int(kcrank[0]);kc1,kc2,kc3=map(int,collision_kc)
|
||||
designs=[('zero',100,[],None),('pn_pulse',300,[(0,p)],None),
|
||||
('kc_ref_boundary',100,[(t,k) for t in (0,1,21,22,23,44)],None),
|
||||
('two_sources',300,[(t,n) for t in (0,22,44,66,88) for n in (p,k)],None),
|
||||
('arrival_collision',120,[(0,pairpn),(0,kc1),(18,kc2),(21,kc1)],None),
|
||||
('natural_request',100,[(0,k)],dict(v={k:-44.0})),
|
||||
('apl_only',100,[(0,int(a)) for a in apl],None),
|
||||
('high_fanin',400,[(t,int(n)) for t in (0,22,44) for n in nodes if pscore[n]>0],dict(g={k:.123}))]
|
||||
assert [v[0] for v in designs]==CFG['fixtures']
|
||||
for name,steps,requests,initial in designs:case(name,'fixture',steps,nodes,fixturegraph,requests,initial)
|
||||
meta.iloc[nodes][['neuron_idx','body_id','annotation_class','annotation_type','neurotransmitter']].to_csv(EXP/'protocol/fixture-identities.csv',index=False)
|
||||
save(EXP/'protocol/fixture-selection.json',dict(pn=p,kc=k,pair_PN=pairpn,arrival_KCs=collision_kc.tolist(),real_PN_KC_edge=best,
|
||||
nonKC_APL_target=nonkc_target,method='Static raw-count ranks and exact metadata; retain ALL existing induced edges and full APL matrices/norms',runtime_selection=False))
|
||||
os.environ['CUDA_VISIBLE_DEVICES']=CFG['gpu_uuid'];sys.path.insert(0,str(legacy));sys.path.insert(0,str(legacy/'deterministic-runtime-v2'))
|
||||
sys.path.insert(0,str(legacy/'terminal-scope-diagnostic-v1'));from diagnostic import queues
|
||||
queues()
|
||||
import torch
|
||||
from phase4p_revalidate import door_inputs,make_drive
|
||||
oc,panel,vectors,selection,mapping=door_inputs();cal=panel[panel.split=='calibration'].copy();assert len(panel)==12 and len(cal)==6
|
||||
assert oc['odor_encoder']['evaluation_seeds']==CFG['calibration']['seeds']
|
||||
assert float(oc['odor_encoder']['duration_ms'])==CFG['calibration']['duration_ms']
|
||||
cal.to_parquet(EXP/'protocol/calibration-panel.parquet',index=False);calrecords=[]
|
||||
for mode in CFG['calibration']['modes']:
|
||||
for oi,(_,odor) in enumerate(cal.iterrows()):
|
||||
targets,rates=make_drive(oc,panel,vectors,selection,mapping,odor.odor_id,mode)
|
||||
assert len(np.unique(targets))==len(targets),'Original encoder emits duplicate PN targets; audit instead of silently merging'
|
||||
rates=np.asarray(rates,np.float32);steps=round(CFG['calibration']['duration_ms']/.1)
|
||||
for seed in CFG['calibration']['seeds']:
|
||||
gen=torch.Generator(device='cuda');gen.manual_seed(seed)
|
||||
original_probability=rates*.1/1000.0
|
||||
explicit_probability=rates*np.float32(.1)/np.float32(1000.)
|
||||
assert original_probability.dtype==np.float32 and np.array_equal(original_probability.view(np.uint32),explicit_probability.view(np.uint32))
|
||||
probs=torch.as_tensor(original_probability,dtype=torch.float32,device='cuda')
|
||||
forced=(torch.rand((steps,len(targets)),generator=gen,device='cuda')<probs[None,:]).cpu().numpy()
|
||||
tt,cc=np.nonzero(forced);req=np.column_stack([tt,targets[cc]]).astype(np.int32)
|
||||
name=f'cal-{mode}-{oi:02d}-s{seed}'
|
||||
case(name,'calibration',steps,fullnodes,EXP/'graphs/full',req,extra=dict(odor_id=str(odor.odor_id),odor_name=str(odor.odor_name),
|
||||
mode=mode,seed=seed,odor_index=oi,expected_external_events=float(rates.sum())*steps*.1/1000.,target_count=len(targets),rate_sum_hz=float(rates.sum())))
|
||||
with (EXP/'cases'/name/'pn-drive.npz').open('xb') as f:np.savez_compressed(f,targets=targets,rates_hz=rates)
|
||||
calrecords.append(dict(name=name,mode=mode,odor_index=oi,odor_id=str(odor.odor_id),seed=seed,requests=len(req),rates_sum=float(rates.sum())))
|
||||
save(EXP/'protocol/calibration-cases.json',calrecords)
|
||||
save(EXP/'protocol/encoder-audit.json',dict(panel_count=12,cal_count=6,median_mapped_response_sum=float(np.median(panel.mapped_response_sum)),
|
||||
target_rng_column_order='Original pn_drive, no sorting before torch.rand',seeds=CFG['calibration']['seeds'],
|
||||
requested_before_any_dynamics=True,formal_AB_run=False,heldout_run=False,torch=torch.__version__,numpy=np.__version__,
|
||||
generation='Original rates*.1/1000 float32 GPU Bernoulli, independent per-case original seed'))
|
||||
revision=[]
|
||||
for folder in sorted((EXP/'cases').iterdir()):
|
||||
with np.load(folder/'requests.npz',allow_pickle=False) as new,np.load(prior/'cases'/folder.name/'requests.npz',allow_pickle=False) as old:
|
||||
assert new.files==old.files and all(new[key].dtype==old[key].dtype and np.array_equal(new[key],old[key]) for key in old.files)
|
||||
revision.append(dict(case=folder.name,sparse_request_arrays_exact=True))
|
||||
save(EXP/'protocol/revision-input-equivalence.json',dict(prior_experiment=str(prior),cases=revision,
|
||||
input_arrays_unchanged=True,reason='Execution-only venv invocation fix; no trajectory outcome used for input choice'))
|
||||
save(EXP/'protocol/prepared-hashes.json',[dict(path=str(p.relative_to(EXP)),bytes=p.stat().st_size,sha256=digest(p))
|
||||
for d in ('graphs','apl-bundle','cases','protocol') for p in sorted((EXP/d).rglob('*')) if p.is_file() and p.name!='prepared-hashes.json'])
|
||||
budget();print('PREPARED8 fixtures and60 fixed CAL inputs; no trajectories; frozen integrity PASS',flush=True)
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,48 @@
|
||||
"""A fresh process per declared case; no concurrent scientific GPU jobs."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
from context import ROOT,MODEL,EXP,CFG,check,budget
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser()
|
||||
parser.add_argument('backend',choices=['gpu','oracle','component'])
|
||||
parser.add_argument('case')
|
||||
parser.add_argument('--repeat',type=int,default=0)
|
||||
args=parser.parse_args()
|
||||
assert '/' not in args.case and '..' not in args.case and 0<=args.repeat<3
|
||||
check();budget()
|
||||
os.environ['CUDA_VISIBLE_DEVICES']=CFG['gpu_uuid']
|
||||
os.environ['CUDA_HOME']='/usr/local/cuda'
|
||||
os.environ['PATH']=str(ROOT/'models/male-cns-lif/.venv/bin')+':/usr/local/cuda/bin:'+os.environ['PATH']
|
||||
os.environ['MAX_JOBS']='2';os.environ['TORCH_CUDA_ARCH_LIST']='8.9'
|
||||
# The pinned Brian environment intentionally has neither Torch nor PyYAML.
|
||||
# Keep this preflight standard-library-only; never install into that env.
|
||||
for port in (8188,8189,8190):
|
||||
with urllib.request.urlopen(f'http://192.168.10.51:{port}/queue',timeout=5) as response:
|
||||
queue=json.load(response)
|
||||
assert not queue['queue_running'] and not queue['queue_pending'],'ComfyUI busy: defer, never evict'
|
||||
case=EXP/'cases'/args.case
|
||||
assert case.is_dir()
|
||||
out=EXP/'gpu'/f'{args.case}-r{args.repeat}'
|
||||
if args.backend=='gpu':
|
||||
from gpu_apl import run_gpu
|
||||
result=run_gpu(case,out,EXP/'extension-build')
|
||||
elif args.backend=='oracle':
|
||||
assert args.repeat==0
|
||||
from oracle_apl import run_case
|
||||
result=run_case(case,EXP/'oracle'/args.case)
|
||||
else:
|
||||
assert args.repeat==0
|
||||
from validate_components import validate
|
||||
result=validate(out,EXP/'apl-bundle',EXP/'component-audit'/(args.case+'.json'))
|
||||
budget()
|
||||
print(json.dumps(dict(backend=args.backend,case=args.case,repeat=args.repeat,
|
||||
complete=result.get('complete'),pass_all=result.get('pass_all')),allow_nan=False),flush=True)
|
||||
if not result.get('pass_all',False):raise SystemExit(1)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,68 @@
|
||||
"""Ordered numerical gates before any full CAL; never tune from outcomes."""
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
import traceback
|
||||
from context import ROOT,MODEL,EXP,CFG,check,save,budget
|
||||
|
||||
SCIENCE=ROOT/'models/male-cns-lif/.venv/bin/python'
|
||||
ORACLE=ROOT/'experiments/phase4ad-author-semantics/environment/.venv/bin/python'
|
||||
|
||||
|
||||
def child(backend,name,repeat=0):
|
||||
log=EXP/'logs'/f'{backend}-{name}-r{repeat}.log'
|
||||
cmd=[str(ORACLE if backend=='oracle' else SCIENCE),'-B',str(MODEL/'run_case.py'),backend,name,'--repeat',str(repeat)]
|
||||
print('START',backend,name,repeat,flush=True);t0=time.perf_counter()
|
||||
with log.open('x',encoding='utf-8') as f:
|
||||
result=subprocess.run(cmd,stdout=f,stderr=subprocess.STDOUT,timeout=1800,check=False)
|
||||
print('FINISH',backend,name,repeat,'exit',result.returncode,'wall',round(time.perf_counter()-t0,2),flush=True)
|
||||
if result.returncode:raise RuntimeError(f'{backend} {name} failed; inspect {log}')
|
||||
|
||||
|
||||
def gate_case(name,repeats):
|
||||
from compare import compare,compare_gpu_repeats
|
||||
child('gpu',name)
|
||||
for r in range(1,repeats):child('gpu',name,r)
|
||||
child('oracle',name)
|
||||
result=compare(EXP/'cases'/name,EXP/'gpu'/(name+'-r0'),EXP/'oracle'/name,EXP/'audit'/('compare-'+name+'.json'))
|
||||
assert result['pass_all'],f'Closed-loop exact comparison failed: {name}; no later CAL permitted'
|
||||
for r in range(1,repeats):
|
||||
report=compare_gpu_repeats(EXP/'gpu'/(name+'-r0'),EXP/'gpu'/f'{name}-r{r}',EXP/'audit'/f'repeat-{name}-r{r}.json')
|
||||
assert report['pass_all'],f'Exact GPU repeat failed: {name}'
|
||||
child('component',name)
|
||||
return dict(case=name,pass_all=True,gpu_repeats=repeats)
|
||||
|
||||
|
||||
def main():
|
||||
parser=argparse.ArgumentParser();parser.add_argument('stage',choices=['fixtures','probes','calibration']);args=parser.parse_args()
|
||||
check();budget();receipt=EXP/'audit'/(args.stage+'-gate.json');assert not receipt.exists()
|
||||
try:
|
||||
if args.stage=='fixtures':
|
||||
rows=[gate_case(n,CFG['fixture_repeats']) for n in CFG['fixtures']]
|
||||
save(receipt,dict(pass_all=True,cases=rows,scope='Fixed real-edge induced fixtures only'))
|
||||
elif args.stage=='probes':
|
||||
assert json.loads((EXP/'audit/fixtures-gate.json').read_text())['pass_all']
|
||||
rows=[gate_case(n,1) for n in CFG['calibration']['full_oracle_probes']]
|
||||
save(receipt,dict(pass_all=True,cases=rows,scope='Only two original first-CAL fixed fullgraph trajectories'))
|
||||
else:
|
||||
assert json.loads((EXP/'audit/fixtures-gate.json').read_text())['pass_all']
|
||||
assert json.loads((EXP/'audit/probes-gate.json').read_text())['pass_all']
|
||||
records=json.loads((EXP/'protocol/calibration-cases.json').read_text());assert len(records)==60
|
||||
probes=CFG['calibration']['full_oracle_probes']
|
||||
for rec in records:
|
||||
if rec['name'] not in probes:child('gpu',rec['name'])
|
||||
if rec['seed']==CFG['calibration']['seeds'][0]:child('gpu',rec['name'],1)
|
||||
from calibration import summarize
|
||||
result=summarize()
|
||||
save(receipt,dict(pass_all=result['pass_all'],scope='Complete original six-CAL representation gate, not heldout or learning',summary='calibration/summary.json'))
|
||||
if not result['pass_all']:raise SystemExit(2)
|
||||
except Exception as e:
|
||||
save(EXP/'audit'/('stopped-'+args.stage+'.json'),dict(stage=args.stage,type=type(e).__name__,message=str(e),traceback=traceback.format_exc(),
|
||||
later_stages_not_permitted=True,partial_outputs_preserved=True,parameters_changed=False))
|
||||
raise
|
||||
budget()
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,142 @@
|
||||
"""Local, synthetic tests only: no CUDA, scientific trajectories, or heldout data."""
|
||||
import copy
|
||||
import unittest
|
||||
|
||||
import numpy as np
|
||||
|
||||
from calibration import (ORIGINAL_MODES, ORIGINAL_SEEDS, ORIGINAL_CAL_IDS,
|
||||
APL_FILES, STATE_DTYPES, arrays_bitwise_equal, evaluate_gates, exact_repeat, pair_metrics,
|
||||
sample_cv, summarize_mode, validate_inventory, validate_state)
|
||||
|
||||
|
||||
GATES = {"sparse_min_fraction": .03, "sparse_max_fraction": .09,
|
||||
"minimum_separable_odor_pairs": 3, "max_fraction_strict": .5, "min_fraction_strict": 0.,
|
||||
"same_jaccard_gt_between": True, "between_euclidean_gt_within": True,
|
||||
"both_modes_required": True, "finite_required": True, "repeat_exact_required": True}
|
||||
|
||||
|
||||
def synthetic_runs(silent=False):
|
||||
result = {}
|
||||
for mode in ORIGINAL_MODES:
|
||||
for odor in range(6):
|
||||
for seed in ORIGINAL_SEEDS:
|
||||
counts = np.zeros(1000, dtype=np.int32)
|
||||
if not silent:
|
||||
counts[odor * 50:(odor + 1) * 50] = 1
|
||||
result[(mode, str(odor), seed)] = {"kc_counts": counts,
|
||||
"expected_external_events": 10., "external_events": 0 if silent else 10}
|
||||
return result
|
||||
|
||||
|
||||
class CalibrationTests(unittest.TestCase):
|
||||
def test_complete_gate_and_original_pair_counts(self):
|
||||
summary = summarize_mode(ORIGINAL_MODES[0], list(map(str, range(6))), ORIGINAL_SEEDS, synthetic_runs(), GATES)
|
||||
self.assertTrue(summary["gate"])
|
||||
self.assertEqual(summary["within_distance_count"], 60)
|
||||
self.assertEqual(summary["between_distance_count"], 75)
|
||||
self.assertEqual(summary["pair_count"], 15)
|
||||
self.assertEqual(summary["pairwise_separation_pass_count"], 15)
|
||||
self.assertTrue(all(row["within_distance_count"] == 20 and row["between_distance_count"] == 5 for row in summary["pairs"]))
|
||||
|
||||
def test_silent_is_not_pass(self):
|
||||
summary = summarize_mode(ORIGINAL_MODES[0], list(map(str, range(6))), ORIGINAL_SEEDS, synthetic_runs(True), GATES)
|
||||
self.assertFalse(summary["gate"])
|
||||
self.assertFalse(summary["conditions"]["minimum_strictly_positive"])
|
||||
self.assertFalse(summary["conditions"]["same_jaccard_gt_different"])
|
||||
self.assertFalse(summary["conditions"]["different_euclidean_gt_same"])
|
||||
self.assertEqual(summary["pairwise_separation_pass_count"], 0)
|
||||
self.assertIsNone(summary["actual_event_count_cv"])
|
||||
|
||||
def test_every_original_conjunct_is_necessary(self):
|
||||
baseline = summarize_mode(ORIGINAL_MODES[0], list(map(str, range(6))), ORIGINAL_SEEDS, synthetic_runs(), GATES)
|
||||
changes = (("KC_active_fraction_mean", .029), ("KC_active_fraction_mean", .091),
|
||||
("KC_active_fraction_max", .5), ("KC_active_fraction_min", 0.),
|
||||
("same_odor_jaccard_mean", baseline["different_odor_jaccard_mean"]),
|
||||
("different_odor_euclidean_mean", baseline["same_odor_euclidean_mean"]),
|
||||
("pairwise_separation_pass_count", 2))
|
||||
for key, value in changes:
|
||||
with self.subTest(key=key, value=value):
|
||||
modified = dict(baseline, **{key: value})
|
||||
self.assertFalse(all(evaluate_gates(modified, GATES).values()))
|
||||
for value in (.03, .09):
|
||||
self.assertTrue(evaluate_gates(dict(baseline, KC_active_fraction_mean=value), GATES)["mean_sparsity_in_closed_interval"])
|
||||
|
||||
def test_pair_specific_not_global_baseline(self):
|
||||
runs = synthetic_runs()
|
||||
mode = ORIGINAL_MODES[0]
|
||||
for odor in range(6):
|
||||
for index, seed in enumerate(ORIGINAL_SEEDS):
|
||||
counts = np.array([index + odor * 100] if odor < 2 else [10000 * odor + index * 1000], dtype=np.int32)
|
||||
runs[(mode, str(odor), seed)]["kc_counts"] = counts
|
||||
summary = summarize_mode(mode, list(map(str, range(6))), ORIGINAL_SEEDS, runs, GATES)
|
||||
first_pair = summary["pairs"][0]
|
||||
self.assertEqual(first_pair["within_this_pair_mean"], 2.)
|
||||
self.assertEqual(first_pair["between_same_seed_mean"], 100.)
|
||||
self.assertTrue(first_pair["gate"])
|
||||
self.assertGreater(summary["same_odor_euclidean_mean"], 100.)
|
||||
|
||||
def test_original_metric_zero_and_count_semantics(self):
|
||||
zero = np.zeros(3, dtype=np.int32)
|
||||
self.assertEqual(pair_metrics(zero, zero)["active_jaccard"], 1.)
|
||||
self.assertEqual(pair_metrics(zero, zero)["correlation"], 1.)
|
||||
result = pair_metrics(np.array([2, 0, 0]), np.array([1, 0, 0]))
|
||||
self.assertEqual(result["euclidean_distance"], 1.)
|
||||
self.assertEqual(result["active_jaccard"], 1.)
|
||||
self.assertAlmostEqual(sample_cv([1., 2., 3.]), .5)
|
||||
|
||||
def test_repeat_comparison_is_bits_and_dtype(self):
|
||||
self.assertFalse(arrays_bitwise_equal(np.array([0.], np.float64), np.array([-0.], np.float64)))
|
||||
self.assertFalse(arrays_bitwise_equal(np.array([1], np.int32), np.array([1], np.int64)))
|
||||
self.assertTrue(arrays_bitwise_equal(np.array([1.], np.float32), np.array([1.], np.float32)))
|
||||
|
||||
def test_exact_repeat_covers_all_recorded_semantics(self):
|
||||
first = {"state": {name: np.zeros(1, dtype) for name, dtype in STATE_DTYPES.items()},
|
||||
"apl": {name: np.zeros((1, width), np.float32) for name, width in APL_FILES.items()},
|
||||
"hashes": [{"tick": 0}], "queues": [{"tick": 0}],
|
||||
"requests": [], "component": {"environment": {"tf32": False}}}
|
||||
self.assertTrue(exact_repeat(first, copy.deepcopy(first))["pass_all"])
|
||||
for category in ("state", "apl"):
|
||||
for name in first[category]:
|
||||
bad = copy.deepcopy(first)
|
||||
bad[category][name].flat[0] = 1
|
||||
with self.subTest(category=category, name=name):
|
||||
self.assertFalse(exact_repeat(first, bad)["pass_all"])
|
||||
for key in ("hashes", "queues", "requests", "component"):
|
||||
bad = copy.deepcopy(first)
|
||||
bad[key] = {} if isinstance(first[key], list) else []
|
||||
with self.subTest(key=key):
|
||||
self.assertFalse(exact_repeat(first, bad)["pass_all"])
|
||||
|
||||
def test_raw_state_rejects_nonfinite_duplicate_and_wrong_counts(self):
|
||||
state = {"step": np.array([0], np.int32), "neuron_idx": np.array([1], np.int32),
|
||||
"counts": np.array([0, 1], np.int32), "v": np.zeros(2, np.float64),
|
||||
"g": np.zeros(2, np.float64), "ref": np.zeros(2, np.int32)}
|
||||
validate_state(state, 2, 10)
|
||||
for mutation in ("nan", "counts", "duplicate", "wrong_dtype"):
|
||||
bad = copy.deepcopy(state)
|
||||
if mutation == "nan": bad["g"][0] = np.nan
|
||||
if mutation == "counts": bad["counts"][0] = 1
|
||||
if mutation == "duplicate":
|
||||
bad["step"] = np.array([0, 0], np.int32)
|
||||
bad["neuron_idx"] = np.array([1, 1], np.int32)
|
||||
bad["counts"][1] = 2
|
||||
if mutation == "wrong_dtype": bad["v"] = bad["v"].astype(np.float32)
|
||||
with self.subTest(mutation=mutation), self.assertRaises(ValueError):
|
||||
validate_state(bad, 2, 10)
|
||||
|
||||
def test_inventory_rejects_missing_duplicate_or_heldout_identity(self):
|
||||
odors = sorted(ORIGINAL_CAL_IDS)
|
||||
records = [{"name": f"cal-{mode}-{index:02d}-s{seed}", "mode": mode, "odor_index": index,
|
||||
"odor_id": odors[index], "seed": seed} for mode in ORIGINAL_MODES
|
||||
for index in range(6) for seed in ORIGINAL_SEEDS]
|
||||
cfg = {"calibration": {"modes": list(ORIGINAL_MODES), "seeds": list(ORIGINAL_SEEDS),
|
||||
"duration_ms": 100., "odor_count": 6, "split": "calibration", "repeat_first_seed_each_odor": True}, "gates": GATES}
|
||||
self.assertEqual(validate_inventory(records, cfg), odors)
|
||||
for malformed in (records[:-1], records[:-1] + [records[0]],
|
||||
[dict(row, odor_id="heldout") if row["odor_index"] == 0 else row for row in records]):
|
||||
with self.assertRaises(ValueError):
|
||||
validate_inventory(malformed, cfg)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
@@ -0,0 +1,266 @@
|
||||
"""Local/remote pure protocol tests; no Brian, Torch, GPU or network execution.
|
||||
|
||||
Run with the already-provisioned Python and ``-B test_protocol.py``. The
|
||||
sidecar process is mocked with in-memory stdin and an anonymous stdout pipe.
|
||||
These tests validate transport and declared arithmetic/input contracts, not
|
||||
the scientific trajectory or independent correctness of the Torch component.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import math
|
||||
import os
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import threading
|
||||
import types
|
||||
import unittest
|
||||
from unittest import mock
|
||||
|
||||
import numpy as np
|
||||
|
||||
|
||||
HERE = Path(__file__).resolve().parent
|
||||
|
||||
|
||||
def load_no_cache(path, name):
|
||||
module = types.ModuleType(name)
|
||||
module.__file__ = str(path)
|
||||
exec(compile(path.read_bytes(), str(path), "exec"), module.__dict__)
|
||||
return module
|
||||
|
||||
|
||||
ORACLE = load_no_cache(HERE / "oracle_apl.py", "phase4ag_protocol_oracle")
|
||||
|
||||
|
||||
def handshake(**updates):
|
||||
result = {"ok": True, "protocol": "phase4ag-apl-v1", "n_neurons": 166700,
|
||||
"n_apl": 2, "n_kc": 4064, "dt_ms": .1,
|
||||
"files_sha256": {name: "frozen-test-hash" for name in ORACLE.BUNDLE_FILES},
|
||||
"decay_f32_bits": f"{int(np.float32(math.exp(-.1 / 20.)).view(np.uint32)):08x}",
|
||||
"gain_f32_bits": f"{int(np.float32(.01).view(np.uint32)):08x}"}
|
||||
result.update(updates)
|
||||
return (json.dumps(result) + "\n").encode("utf-8")
|
||||
|
||||
|
||||
def frame(tick=0, values=None, byte_count=None):
|
||||
values = np.arange(4068, dtype="<f4") if values is None else np.asarray(values, dtype="<f4")
|
||||
count = values.nbytes if byte_count is None else byte_count
|
||||
return struct.pack("<qQ", tick, count) + values.tobytes()
|
||||
|
||||
|
||||
class MemoryInput(io.BytesIO):
|
||||
def close(self):
|
||||
if not self.closed:
|
||||
self.closed_data = self.getvalue()
|
||||
super().close()
|
||||
|
||||
|
||||
class FakeProcess:
|
||||
"""The only use of OS descriptors is an anonymous read-only response pipe."""
|
||||
def __init__(self, response, status=0, chunk_size=97):
|
||||
read_fd, write_fd = os.pipe()
|
||||
self.stdin = MemoryInput()
|
||||
self.stdout = os.fdopen(read_fd, "rb", buffering=0)
|
||||
self.status = status
|
||||
self.terminated = False
|
||||
|
||||
def write_response():
|
||||
try:
|
||||
for begin in range(0, len(response), chunk_size):
|
||||
view = memoryview(response[begin:begin + chunk_size])
|
||||
while view:
|
||||
view = view[os.write(write_fd, view):]
|
||||
except (BrokenPipeError, OSError):
|
||||
# Invalid-response tests deliberately close the reader early.
|
||||
pass
|
||||
finally:
|
||||
os.close(write_fd)
|
||||
|
||||
self.thread = threading.Thread(target=write_response, daemon=True)
|
||||
self.thread.start()
|
||||
|
||||
def poll(self):
|
||||
return self.status
|
||||
|
||||
def wait(self, timeout=None):
|
||||
return self.status
|
||||
|
||||
def terminate(self):
|
||||
self.terminated = True
|
||||
|
||||
def kill(self):
|
||||
self.terminated = True
|
||||
|
||||
def close(self):
|
||||
if not self.stdin.closed:
|
||||
self.stdin.close()
|
||||
if not self.stdout.closed:
|
||||
self.stdout.close()
|
||||
self.thread.join(timeout=2)
|
||||
|
||||
|
||||
class ProtocolTests(unittest.TestCase):
|
||||
def create_worker(self, response, status=0):
|
||||
process = FakeProcess(response, status=status)
|
||||
self.addCleanup(process.close)
|
||||
source_patch = mock.patch.object(ORACLE, "_digest", return_value="frozen-test-hash")
|
||||
process_patch = mock.patch.object(ORACLE.subprocess, "Popen", return_value=process)
|
||||
source_patch.start()
|
||||
process_mock = process_patch.start()
|
||||
self.addCleanup(source_patch.stop)
|
||||
self.addCleanup(process_patch.stop)
|
||||
worker = ORACLE._APLWorker(Path("science-python"), Path("legacy_apl_worker.py"), Path("bundle"), None, timeout=2)
|
||||
self.addCleanup(worker.abort)
|
||||
return worker, process, process_mock
|
||||
|
||||
def test_exact_handshake_frame_and_clean_eof(self):
|
||||
worker, process, process_mock = self.create_worker(handshake() + frame())
|
||||
globals_ = np.array([1, 40, 166699], dtype=np.int32)
|
||||
drive, state, inhibition = worker.step(0, globals_)
|
||||
values = np.arange(4068, dtype=np.float32)
|
||||
np.testing.assert_array_equal(drive, values[:2])
|
||||
np.testing.assert_array_equal(state, values[2:4])
|
||||
np.testing.assert_array_equal(inhibition, values[4:])
|
||||
request = process.stdin.getvalue()
|
||||
self.assertEqual(struct.unpack("<qQ", request[:16]), (0, 3))
|
||||
np.testing.assert_array_equal(np.frombuffer(request[16:], dtype="<i4"), globals_)
|
||||
self.assertEqual(worker.finish(1), 0)
|
||||
command = process_mock.call_args.args[0]
|
||||
self.assertEqual(command, ["science-python", "-B", "legacy_apl_worker.py", "--bundle-dir", "bundle", "--dt-ms", "0.1"])
|
||||
self.assertFalse(process_mock.call_args.kwargs["shell"])
|
||||
|
||||
def test_empty_spike_frame_still_advances_one_tick(self):
|
||||
worker, process, _ = self.create_worker(handshake() + frame(values=np.zeros(4068, np.float32)))
|
||||
arrays = worker.step(0, np.empty(0, np.int32))
|
||||
self.assertTrue(all(np.all(value == 0) for value in arrays))
|
||||
self.assertEqual(process.stdin.getvalue(), struct.pack("<qQ", 0, 0))
|
||||
self.assertEqual(worker.finish(1), 0)
|
||||
|
||||
def test_two_consecutive_ticks(self):
|
||||
worker, _, _ = self.create_worker(handshake() + frame(0) + frame(1))
|
||||
worker.step(0, np.array([4], np.int32))
|
||||
worker.step(1, np.array([8, 10], np.int32))
|
||||
self.assertEqual(worker.finish(2), 0)
|
||||
|
||||
def test_rejects_wrong_protocol_or_dimensions(self):
|
||||
for update in ({"protocol": "other"}, {"ok": False}, {"n_neurons": 42},
|
||||
{"n_kc": 3}, {"dt_ms": .2}, {"gain_f32_bits": "00000000"},
|
||||
{"files_sha256": {}}):
|
||||
with self.subTest(update=update):
|
||||
with self.assertRaises(ValueError):
|
||||
self.create_worker(handshake(**update))
|
||||
|
||||
def test_rejects_wrong_tick_or_payload_length(self):
|
||||
for response in (frame(tick=1), frame(byte_count=8)):
|
||||
with self.subTest(response_header=response[:16]):
|
||||
worker, _, _ = self.create_worker(handshake() + response)
|
||||
with self.assertRaises(ValueError):
|
||||
worker.step(0, np.empty(0, np.int32))
|
||||
|
||||
def test_rejects_truncated_response(self):
|
||||
worker, _, _ = self.create_worker(handshake() + frame()[:-1])
|
||||
with self.assertRaises(EOFError):
|
||||
worker.step(0, np.empty(0, np.int32))
|
||||
|
||||
def test_rejects_nonfinite_or_negative_apl_payload(self):
|
||||
for bad_value in (np.nan, np.inf, -1):
|
||||
with self.subTest(value=bad_value):
|
||||
values = np.zeros(4068, np.float32)
|
||||
values[-1] = bad_value
|
||||
worker, _, _ = self.create_worker(handshake() + frame(values=values))
|
||||
with self.assertRaises(ValueError):
|
||||
worker.step(0, np.empty(0, np.int32))
|
||||
|
||||
def test_rejects_invalid_spike_dtype_order_and_range(self):
|
||||
for values in (np.array([1], np.int64), np.array([2, 1], np.int32),
|
||||
np.array([1, 1], np.int32), np.array([-1], np.int32),
|
||||
np.array([166700], np.int32)):
|
||||
with self.subTest(values=values):
|
||||
worker, process, _ = self.create_worker(handshake() + frame())
|
||||
with self.assertRaises(ValueError):
|
||||
worker.step(0, values)
|
||||
self.assertEqual(process.stdin.getvalue(), b"")
|
||||
|
||||
def test_rejects_skipped_request_tick(self):
|
||||
worker, process, _ = self.create_worker(handshake() + frame())
|
||||
with self.assertRaises(ValueError):
|
||||
worker.step(1, np.empty(0, np.int32))
|
||||
self.assertEqual(process.stdin.getvalue(), b"")
|
||||
|
||||
def test_rejects_failed_exit_and_trailing_stdout(self):
|
||||
for suffix, status in ((b"", 7), (b"unexpected", 0)):
|
||||
with self.subTest(status=status, suffix=suffix):
|
||||
worker, _, _ = self.create_worker(handshake() + frame() + suffix, status=status)
|
||||
worker.step(0, np.empty(0, np.int32))
|
||||
with self.assertRaises(ValueError):
|
||||
worker.finish(1)
|
||||
|
||||
def test_pipe_reader_timeout(self):
|
||||
read_fd, write_fd = os.pipe()
|
||||
stream = os.fdopen(read_fd, "rb", buffering=0)
|
||||
reader = ORACLE._PipeReader(stream, .05)
|
||||
try:
|
||||
with self.assertRaises(TimeoutError):
|
||||
reader.read(1)
|
||||
finally:
|
||||
os.close(write_fd)
|
||||
reader.thread.join(timeout=2)
|
||||
stream.close()
|
||||
self.assertFalse(reader.thread.is_alive())
|
||||
|
||||
|
||||
class ArithmeticAndIdentityContractTests(unittest.TestCase):
|
||||
def test_si_conversion_must_follow_float64_promotion(self):
|
||||
h32 = np.array([.01, 1 / 7, 123.456], dtype=np.float32)
|
||||
correct = h32.astype(np.float64) * np.float64(.001)
|
||||
early_conversion = (h32 * np.float32(.001)).astype(np.float64)
|
||||
self.assertFalse(np.array_equal(correct.view(np.uint64), early_conversion.view(np.uint64)))
|
||||
for actual, original in zip(correct, h32):
|
||||
self.assertEqual(actual.hex(), (float(original) * float(.001)).hex())
|
||||
|
||||
def test_declared_overlay_does_not_guard_refractory_targets(self):
|
||||
before = np.array([0, -.002, .001], dtype=np.float64)
|
||||
refractory = np.array([22, 21, 0], dtype=np.int32)
|
||||
h32 = np.array([1, 2, 3], dtype=np.float32)
|
||||
delta = h32.astype(np.float64) * np.float64(.001)
|
||||
after = before - delta
|
||||
self.assertTrue(np.all(after != before))
|
||||
np.testing.assert_array_equal(refractory, [22, 21, 0])
|
||||
self.assertEqual(after[0], -.001)
|
||||
|
||||
def test_af_queue_hash_keeps_original_noncontiguous_ids_and_rejections(self):
|
||||
path = HERE.parent / "author-fullgraph-lab-v1" / "oracle_full.py"
|
||||
self.assertEqual(hashlib.sha256(path.read_bytes()).hexdigest(), ORACLE.AF_SHA256)
|
||||
af = load_no_cache(path, "phase4ag_protocol_af")
|
||||
edges = np.array([81, 2, 500], dtype=np.int64)
|
||||
accepted = np.array([1, 0, 1], dtype=np.uint8)
|
||||
original_edges, original_mask = edges.copy(), accepted.copy()
|
||||
result = af.queue_summary(18, edges, accepted, chunk_bytes=8)
|
||||
packet = b"phase4af-queue-v1\0" + struct.pack("<qq", 18, 3) + edges.astype("<i8").tobytes() + accepted.tobytes()
|
||||
self.assertEqual(result["joint_sha256"], hashlib.sha256(packet).hexdigest())
|
||||
self.assertEqual((result["count"], result["accepted_count"]), (3, 2))
|
||||
self.assertEqual(result, af.queue_summary(18, edges, accepted, chunk_bytes=8192))
|
||||
np.testing.assert_array_equal(edges, original_edges)
|
||||
np.testing.assert_array_equal(accepted, original_mask)
|
||||
|
||||
def test_empty_queue_hash_includes_tick(self):
|
||||
af = load_no_cache(HERE.parent / "author-fullgraph-lab-v1" / "oracle_full.py", "phase4ag_empty_af")
|
||||
first = af.queue_summary(0, np.empty(0, np.int64), np.empty(0, np.uint8))
|
||||
second = af.queue_summary(1, np.empty(0, np.int64), np.empty(0, np.uint8))
|
||||
self.assertEqual(first["ordered_edge_sha256"], hashlib.sha256(b"").hexdigest())
|
||||
self.assertNotEqual(first["joint_sha256"], second["joint_sha256"])
|
||||
|
||||
def test_oracle_import_has_no_brian_or_torch_dependency(self):
|
||||
before = set(sys.modules)
|
||||
load_no_cache(HERE / "oracle_apl.py", "phase4ag_second_import")
|
||||
added = set(sys.modules) - before
|
||||
self.assertFalse(any(name == "brian2" or name.startswith("brian2.") or
|
||||
name == "torch" or name.startswith("torch.") for name in added))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main(verbosity=2)
|
||||
@@ -0,0 +1,38 @@
|
||||
"""Regression for an execution-path error, unrelated to brain dynamics."""
|
||||
from pathlib import Path
|
||||
from unittest import TestCase,main,mock,skipIf
|
||||
import os
|
||||
import sys
|
||||
import tempfile
|
||||
from oracle_apl import _python_entry
|
||||
|
||||
|
||||
class PythonEntryTests(TestCase):
|
||||
@skipIf(os.name=='nt','Real POSIX interpreter symlink is validated on the task server')
|
||||
def test_real_symlink_remains_invocation_path(self):
|
||||
with tempfile.TemporaryDirectory(prefix='phase4ag-entry-') as folder:
|
||||
path=Path(folder)/'python'
|
||||
path.symlink_to(sys.executable)
|
||||
self.assertEqual(_python_entry(path),path)
|
||||
self.assertNotEqual(_python_entry(path),path.resolve())
|
||||
|
||||
@skipIf(os.name=='nt','POSIX dangling symlink test is validated on the task server')
|
||||
def test_real_dangling_link_and_directory_rejected(self):
|
||||
with tempfile.TemporaryDirectory(prefix='phase4ag-entry-') as folder:
|
||||
path=Path(folder)/'python'
|
||||
path.symlink_to(Path(folder)/'absent')
|
||||
with self.assertRaises(ValueError):_python_entry(path)
|
||||
with self.assertRaises(ValueError):_python_entry(Path(folder))
|
||||
|
||||
def test_preserves_declared_venv_path_without_resolving_symlink(self):
|
||||
entry=Path.cwd()/'.fixture-venv/bin/python'
|
||||
with mock.patch.object(Path,'resolve',side_effect=AssertionError('Must not resolve interpreter symlink')):
|
||||
with mock.patch.object(Path,'is_file',return_value=True):
|
||||
self.assertEqual(_python_entry(entry),entry)
|
||||
|
||||
def test_missing_python_entry_is_rejected(self):
|
||||
with mock.patch.object(Path,'is_file',return_value=False):
|
||||
with self.assertRaises(ValueError):_python_entry('/missing/python')
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,431 @@
|
||||
"""Independent extraction/replay of the frozen legacy APL component.
|
||||
|
||||
Only the initializer prefix before ``self.base`` and two original APL update
|
||||
statements are executed. No old simulator is instantiated, no learning code is
|
||||
executed, and no frozen artifact is written. The Torch replay is a component
|
||||
oracle, not an independent CPU whole-brain closed-loop oracle.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import argparse
|
||||
import ast
|
||||
import copy
|
||||
from functools import lru_cache
|
||||
import hashlib
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import types
|
||||
|
||||
import numpy as np
|
||||
import pyarrow.parquet as pq
|
||||
import torch
|
||||
|
||||
|
||||
ROOT = Path('/data/digital-drosophila')
|
||||
CANONICAL = ROOT/'datasets/male-cns-v1.0/processed/neuron-only'
|
||||
SOURCE = ROOT/'models/male-cns-lif/deterministic-runtime-v2/phase4p_revalidate.py'
|
||||
SOURCE_SHA256 = 'c7e693a2da70acdce4f53af5c2853e52e8333acc93f88a46a5e8083eac639346'
|
||||
N, A, K = 166700, 2, 4064
|
||||
BUNDLE_ARRAYS = {
|
||||
'input_matrix': (np.float32, (A, N)), 'output_matrix': (np.float32, (A, K)),
|
||||
'input_raw': (np.float32, (A, N)), 'output_raw': (np.float32, (A, K)),
|
||||
'kc_globals': (np.int32, (K,)), 'apl_globals': (np.int32, (A,)),
|
||||
'pn_globals': (np.int32, None), 'apl_out_ids': (np.int64, (5845,)),
|
||||
}
|
||||
|
||||
|
||||
def require(condition, message):
|
||||
if not condition:
|
||||
raise ValueError(message)
|
||||
|
||||
|
||||
def digest(path):
|
||||
h = hashlib.sha256()
|
||||
with Path(path).open('rb') as source:
|
||||
for block in iter(lambda: source.read(8 << 20), b''):
|
||||
h.update(block)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def array_hash(value):
|
||||
return hashlib.sha256(np.ascontiguousarray(value).tobytes()).hexdigest()
|
||||
|
||||
|
||||
def bit_equal(a, b):
|
||||
a, b = np.asarray(a), np.asarray(b)
|
||||
return bool(a.dtype == b.dtype and a.shape == b.shape and
|
||||
np.array_equal(np.ascontiguousarray(a).view(np.uint8),
|
||||
np.ascontiguousarray(b).view(np.uint8)))
|
||||
|
||||
|
||||
def save(path, value):
|
||||
path = Path(path)
|
||||
path.parent.mkdir(parents=True, exist_ok=True)
|
||||
with path.open('x', encoding='utf-8') as target:
|
||||
json.dump(value, target, indent=2, sort_keys=True, allow_nan=False)
|
||||
target.write('\n')
|
||||
|
||||
|
||||
class ReadOnlyNamespace:
|
||||
"""The extracted prefix can read rt, but cannot mutate or add rt fields."""
|
||||
def __init__(self, **values):
|
||||
object.__setattr__(self, '_values', types.MappingProxyType(values))
|
||||
|
||||
def __getattr__(self, name):
|
||||
return self._values[name]
|
||||
|
||||
def __setattr__(self, name, value):
|
||||
raise AttributeError('Read-only canonical runtime view')
|
||||
|
||||
|
||||
def _code(nodes, filename):
|
||||
module = ast.Module(body=copy.deepcopy(nodes), type_ignores=[])
|
||||
return compile(ast.fix_missing_locations(module), str(filename), 'exec')
|
||||
|
||||
|
||||
def _self_attribute(target, field):
|
||||
return isinstance(target, ast.Attribute) and isinstance(target.value, ast.Name) and \
|
||||
target.value.id == 'self' and target.attr == field
|
||||
|
||||
|
||||
def extract_statements():
|
||||
data = SOURCE.read_bytes()
|
||||
require(hashlib.sha256(data).hexdigest() == SOURCE_SHA256, 'Frozen legacy source SHA-256 mismatch')
|
||||
text = data.decode('utf-8')
|
||||
tree = ast.parse(text, filename=str(SOURCE))
|
||||
classes = [node for node in tree.body if isinstance(node, ast.ClassDef) and node.name == 'DeterministicAPLSim']
|
||||
require(len(classes) == 1, 'Expected exactly one frozen DeterministicAPLSim')
|
||||
methods = {node.name: node for node in classes[0].body if isinstance(node, ast.FunctionDef)}
|
||||
init, run = methods['__init__'], methods['run']
|
||||
cuts = [i for i, node in enumerate(init.body) if isinstance(node, ast.Assign) and
|
||||
any(_self_attribute(target, 'base') for target in node.targets)]
|
||||
require(len(cuts) == 1 and cuts[0] > 0, 'Missing/ambiguous self.base cutoff')
|
||||
prefix = init.body[:cuts[0]]
|
||||
# Executing no import or file-writing call is allowed in the extracted prefix.
|
||||
require(not any(isinstance(node, (ast.Import, ast.ImportFrom, ast.With, ast.AsyncWith))
|
||||
for stmt in prefix for node in ast.walk(stmt)), 'Unexpected side-effect construct in initializer prefix')
|
||||
expressions = [node for node in ast.walk(run) if isinstance(node, ast.Expr) and
|
||||
isinstance(node.value, ast.Call) and isinstance(node.value.func, ast.Attribute) and
|
||||
node.value.func.attr == 'add_' and isinstance(node.value.func.value, ast.Call) and
|
||||
isinstance(node.value.func.value.func, ast.Attribute) and
|
||||
isinstance(node.value.func.value.func.value, ast.Name) and
|
||||
node.value.func.value.func.value.id == 'apl_state' and node.value.func.value.func.attr == 'mul_']
|
||||
inhibition = [node for node in ast.walk(run) if isinstance(node, ast.Assign) and
|
||||
any(isinstance(target, ast.Name) and target.id == 'inhibition' for target in node.targets)]
|
||||
decay = [node for node in ast.walk(run) if isinstance(node, ast.Assign) and
|
||||
any(isinstance(target, ast.Name) and target.id == 'apl_decay' for target in node.targets)]
|
||||
require(len(expressions) == len(inhibition) == len(decay) == 1, 'Ambiguous frozen APL operator extraction')
|
||||
require(expressions[0].lineno < inhibition[0].lineno, 'Unexpected frozen APL update order')
|
||||
rhs = inhibition[0].value
|
||||
require(isinstance(rhs, ast.BinOp) and isinstance(rhs.op, ast.Mult), 'Unexpected inhibition gain expression')
|
||||
gain_expression = ast.Expression(copy.deepcopy(rhs.right))
|
||||
provenance = {'source': str(SOURCE), 'sha256': SOURCE_SHA256,
|
||||
'initializer_line_start': prefix[0].lineno, 'initializer_line_end': prefix[-1].end_lineno,
|
||||
'excluded_self_base_line': init.body[cuts[0]].lineno,
|
||||
'decay_line': decay[0].lineno, 'state_update_line': expressions[0].lineno,
|
||||
'inhibition_line': inhibition[0].lineno,
|
||||
'initializer_source': '\n'.join(ast.get_source_segment(text, node) for node in prefix),
|
||||
'decay_source': ast.get_source_segment(text, decay[0]),
|
||||
'state_update_source': ast.get_source_segment(text, expressions[0]),
|
||||
'inhibition_source': ast.get_source_segment(text, inhibition[0]),
|
||||
'learning_initializer_suffix_executed': False, 'original_run_loop_executed': False}
|
||||
return (_code(prefix, SOURCE), _code(decay, SOURCE),
|
||||
_code([expressions[0], inhibition[0]], SOURCE),
|
||||
compile(ast.fix_missing_locations(gain_expression), str(SOURCE), 'eval'), provenance)
|
||||
|
||||
|
||||
def _canonical_files():
|
||||
return {name: CANONICAL/path for name, path in {
|
||||
'metadata': 'neuron-index/unified-metadata.parquet',
|
||||
'indptr': 'gpu/csr-indptr.int64.npy', 'indices': 'gpu/csr-indices.int32.npy',
|
||||
'raw': 'gpu/csr-weights.int64.npy'}.items()}
|
||||
|
||||
|
||||
@lru_cache(maxsize=2)
|
||||
def reconstruct(canonical_hashes, device_name):
|
||||
"""Cache only reconstructions tied to exact canonical bytes and source hash."""
|
||||
files = _canonical_files()
|
||||
require(dict(canonical_hashes) == {name: digest(path) for name, path in files.items()},
|
||||
'Canonical bytes changed before matrix reconstruction')
|
||||
arrays = {name: np.load(files[name], mmap_mode='r', allow_pickle=False)
|
||||
for name in ('indptr', 'indices', 'raw')}
|
||||
require(arrays['indptr'].dtype == np.int64 and arrays['indices'].dtype == np.int32 and
|
||||
arrays['raw'].dtype == np.int64, 'Canonical CSR dtypes changed')
|
||||
require(arrays['indptr'].shape == (N + 1,) and arrays['indptr'][-1] == 25582938 and
|
||||
arrays['indices'].shape == arrays['raw'].shape == (25582938,), 'Canonical graph shape changed')
|
||||
require(int(arrays['raw'].sum()) == 124177617 and bool(np.all(arrays['raw'] > 0)), 'Canonical raw weights changed')
|
||||
table = pq.read_table(files['metadata'])
|
||||
require(table.num_rows == N, 'Canonical metadata count changed')
|
||||
rt = ReadOnlyNamespace(n=N, dt=.1, device=torch.device(device_name), table=table,
|
||||
indptr_np=arrays['indptr'], indices_np=arrays['indices'], raw_np=arrays['raw'])
|
||||
view = types.SimpleNamespace()
|
||||
init_code, decay_code, update_code, gain_code, provenance = extract_statements()
|
||||
scope = {'self': view, 'rt': rt, 'np': np, 'torch': torch}
|
||||
exec(init_code, scope)
|
||||
require(not hasattr(view, 'base') and not hasattr(view, 'identity'), 'Initializer crossed the learning cutoff')
|
||||
expected = {'input_matrix': view.input_gpu.detach().cpu().numpy().copy(),
|
||||
'output_matrix': view.output_gpu.detach().cpu().numpy().copy(),
|
||||
'input_raw': scope['input_matrix'], 'output_raw': scope['output_matrix'],
|
||||
'kc_globals': view.kc, 'apl_globals': view.apl, 'pn_globals': view.pn,
|
||||
'apl_out_ids': view.apl_out_ids}
|
||||
for value in expected.values():
|
||||
value.flags.writeable = False
|
||||
return view, expected, provenance, (decay_code, update_code, gain_code)
|
||||
|
||||
|
||||
class TorchObservationProxy:
|
||||
"""Observe the result of the actual extracted torch.mv call unchanged."""
|
||||
def __init__(self):
|
||||
self.drive = None
|
||||
self.calls = 0
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(torch, name)
|
||||
|
||||
def mv(self, *args, **kwargs):
|
||||
value = torch.mv(*args, **kwargs)
|
||||
self.drive = value
|
||||
self.calls += 1
|
||||
return value
|
||||
|
||||
|
||||
def _up(value):
|
||||
return np.nextafter(np.asarray(value, dtype=np.float64), np.inf)
|
||||
|
||||
|
||||
def _add(a, b):
|
||||
return _up(np.asarray(a, dtype=np.float64) + np.asarray(b, dtype=np.float64))
|
||||
|
||||
|
||||
def _mul(a, b):
|
||||
return _up(np.asarray(a, dtype=np.float64) * np.asarray(b, dtype=np.float64))
|
||||
|
||||
|
||||
def _gamma(count, unit_roundoff):
|
||||
require(count * unit_roundoff < 1, 'Gamma bound is undefined')
|
||||
# Outward rounding on the division; the denominator is rounded downward.
|
||||
numerator = _mul(count, unit_roundoff)
|
||||
denominator = np.nextafter(1.0 - numerator, -np.inf)
|
||||
return _up(numerator / denominator)
|
||||
|
||||
|
||||
def _positive_dot_upper(state, matrix):
|
||||
result = np.zeros(matrix.shape[1], np.float64)
|
||||
for index in range(len(state)):
|
||||
result = _add(result, _mul(state[index], matrix[index]))
|
||||
return result
|
||||
|
||||
|
||||
def _normal_or_zero(value, tiny):
|
||||
value = np.asarray(value)
|
||||
return bool(np.all((value == 0) | (value >= tiny)))
|
||||
|
||||
|
||||
def _new_metric():
|
||||
return {'max_abs_error': 0.0, 'sum_abs_error': 0.0, 'elements': 0,
|
||||
'max_relative_error_nonzero_reference': 0.0, 'zero_reference_nonzero_actual': 0,
|
||||
'max_error_bound': 0.0, 'max_bound_utilization': 0.0, 'bound_violations': 0}
|
||||
|
||||
|
||||
def _accumulate(metric, actual, reference, bound):
|
||||
error = np.abs(np.asarray(actual, np.float64) - reference)
|
||||
error_upper = np.where(error == 0, 0.0, _up(error))
|
||||
nonzero = reference != 0
|
||||
ratio = np.divide(error, np.abs(reference), out=np.zeros_like(error), where=nonzero)
|
||||
usage = np.divide(error_upper, bound, out=np.zeros_like(error), where=bound > 0)
|
||||
metric['max_abs_error'] = max(metric['max_abs_error'], float(error.max(initial=0)))
|
||||
metric['sum_abs_error'] += float(error.sum(dtype=np.float64))
|
||||
metric['elements'] += error.size
|
||||
metric['max_relative_error_nonzero_reference'] = max(metric['max_relative_error_nonzero_reference'], float(ratio.max(initial=0)))
|
||||
metric['zero_reference_nonzero_actual'] += int(np.count_nonzero((~nonzero) & (error != 0)))
|
||||
metric['max_error_bound'] = max(metric['max_error_bound'], float(np.max(bound, initial=0)))
|
||||
metric['max_bound_utilization'] = max(metric['max_bound_utilization'], float(usage.max(initial=0)))
|
||||
metric['bound_violations'] += int(np.count_nonzero(error_upper > bound))
|
||||
|
||||
|
||||
def validate(gpu_case_dir, bundle_dir, out_json):
|
||||
"""Validate one completed fixture; write one exclusive pure-JSON audit."""
|
||||
gpu_case_dir, bundle_dir, out_json = Path(gpu_case_dir), Path(bundle_dir), Path(out_json)
|
||||
require(not out_json.exists(), 'Refusing to overwrite an existing component audit')
|
||||
report = {'pass_all': False, 'scope': 'Frozen APL component extraction and recorded-spike replay only',
|
||||
'independent_cpu_closed_loop_oracle': False, 'brain_or_learning_parameters_changed': False,
|
||||
'candidate_function_called': False, 'gpu_case_dir': str(gpu_case_dir), 'bundle_dir': str(bundle_dir)}
|
||||
try:
|
||||
require(digest(SOURCE) == SOURCE_SHA256, 'Frozen source changed before component validation')
|
||||
require(torch.cuda.is_available() and torch.cuda.device_count() == 1 and
|
||||
'4090' in torch.cuda.get_device_name(), 'Require the one declared RTX4090')
|
||||
environment = {'torch': str(torch.__version__), 'numpy': np.__version__,
|
||||
'gpu': torch.cuda.get_device_name(), 'cuda': torch.version.cuda,
|
||||
'cuda_matmul_allow_tf32': torch.backends.cuda.matmul.allow_tf32,
|
||||
'float32_matmul_precision': torch.get_float32_matmul_precision(),
|
||||
'deterministic_algorithms': torch.are_deterministic_algorithms_enabled(),
|
||||
'precision_flags_changed': False}
|
||||
report['environment'] = environment
|
||||
canonical_hashes = tuple(sorted((name, digest(path)) for name, path in _canonical_files().items()))
|
||||
view, reconstructed, provenance, code = reconstruct(canonical_hashes, 'cuda:0')
|
||||
bundle = {}
|
||||
matrix_checks = {}
|
||||
for name, (dtype, shape) in BUNDLE_ARRAYS.items():
|
||||
value = np.load(bundle_dir/(name+'.npy'), mmap_mode='r', allow_pickle=False)
|
||||
require(value.dtype == dtype and value.ndim == len(shape or (0,)) and
|
||||
(shape is None or value.shape == shape), 'Bundle schema mismatch: '+name)
|
||||
bundle[name] = value
|
||||
matrix_checks[name] = bit_equal(value, reconstructed[name])
|
||||
scales = json.loads((bundle_dir/'scales.json').read_text())
|
||||
for name in ('input_scale', 'output_scale'):
|
||||
matrix_checks[name] = struct.pack('<d', float(scales[name])) == struct.pack('<d', float(getattr(view, name)))
|
||||
report['matrix_checks'] = matrix_checks
|
||||
report['extracted_source'] = provenance
|
||||
report['canonical_sha256'] = dict(canonical_hashes)
|
||||
report['bundle_sha256'] = {name+'.npy': digest(bundle_dir/(name+'.npy')) for name in BUNDLE_ARRAYS}
|
||||
report['bundle_sha256']['scales.json'] = digest(bundle_dir/'scales.json')
|
||||
report['reconstructed_array_sha256'] = {name: array_hash(value) for name, value in reconstructed.items()}
|
||||
bundle_provenance = json.loads((bundle_dir/'provenance.json').read_text())
|
||||
require(Path(bundle_provenance['canonical_root']).resolve() == CANONICAL.resolve(),
|
||||
'Bundle provenance names another canonical dataset')
|
||||
report['bundle_sha256']['provenance.json'] = digest(bundle_dir/'provenance.json')
|
||||
require(all(matrix_checks.values()), 'Bundle differs from the actual frozen initializer prefix')
|
||||
|
||||
traces = {name: np.load(gpu_case_dir/('apl-'+name+'.float32.npy'), mmap_mode='r', allow_pickle=False)
|
||||
for name in ('drive', 'state', 'inhibition')}
|
||||
steps = len(traces['state'])
|
||||
require(steps > 0, 'Empty APL trajectory cannot pass')
|
||||
for name, width in (('drive', A), ('state', A), ('inhibition', K)):
|
||||
require(traces[name].dtype == np.float32 and traces[name].shape == (steps, width) and
|
||||
bool(np.isfinite(traces[name]).all()) and bool(np.all(traces[name] >= 0)), 'Invalid APL trace: '+name)
|
||||
summary = json.loads((gpu_case_dir/'summary.json').read_text())
|
||||
require(summary.get('complete') is True and summary.get('pass_all') is True and
|
||||
summary.get('steps') == steps and summary.get('completed_ticks') == steps,
|
||||
'GPU fixture did not complete its internal execution checks')
|
||||
with np.load(gpu_case_dir/'state.npz', allow_pickle=False) as archive:
|
||||
event_step, event_node = archive['step'].copy(), archive['neuron_idx'].copy()
|
||||
require(event_step.dtype == event_node.dtype == np.int32 and event_step.ndim == 1 and
|
||||
event_step.shape == event_node.shape, 'Invalid sparse actual spike event schema')
|
||||
require(bool(np.all((event_step >= 0) & (event_step < steps))) and
|
||||
bool(np.all((event_node >= 0) & (event_node < N))), 'Sparse event outside its clock/identity range')
|
||||
require(np.array_equal(np.lexsort((event_node, event_step)), np.arange(len(event_step))), 'Actual events not sorted')
|
||||
keys = event_step.astype(np.int64)*N + event_node
|
||||
require(len(keys) < 2 or bool(np.all(np.diff(keys) > 0)), 'Duplicate actual spikes')
|
||||
boundaries = np.searchsorted(event_step, np.arange(steps + 1))
|
||||
|
||||
proxy = TorchObservationProxy()
|
||||
# Reconstructed original matrices are already on the original GPU/dtype.
|
||||
replay_rt = types.SimpleNamespace(spike_u8=torch.zeros(N, dtype=torch.uint8, device=view.device))
|
||||
shadow = types.SimpleNamespace(input_gpu=view.input_gpu, output_gpu=view.output_gpu, rt=replay_rt, dt=.1)
|
||||
scope = {'self': shadow, 'np': np, 'math': math, 'torch': proxy,
|
||||
'apl_state': torch.zeros(A, dtype=torch.float32, device=view.device)}
|
||||
decay_code, update_code, gain_code = code
|
||||
exec(decay_code, scope)
|
||||
decay32 = np.float32(scope['apl_decay'])
|
||||
gain32 = np.float32(eval(gain_code, {'np': np}))
|
||||
require(0 < decay32 < 1 and gain32 > 0, 'Invalid extracted positive APL constants')
|
||||
exact = {name: True for name in traces}
|
||||
first_difference = None
|
||||
I64 = np.asarray(bundle['input_matrix'], np.float64)
|
||||
O64 = np.asarray(bundle['output_matrix'], np.float64)
|
||||
require(bool(np.all(I64 >= 0)) and bool(np.all(O64 >= 0)), 'Positive-sum theory requires nonnegative matrices')
|
||||
d, q = float(decay32), float(gain32)
|
||||
u32, u64 = 2.0**-24, 2.0**-53
|
||||
gd32, gd64 = _gamma(N-1, u32), _gamma(N-1, u64)
|
||||
g232, g264 = _gamma(2, u32), _gamma(2, u64)
|
||||
gh32, gh64 = _gamma(2*A+1, u32), _gamma(2*A+1, u64)
|
||||
state64 = np.zeros(A, np.float64)
|
||||
e32 = np.zeros(A, np.float64)
|
||||
e64 = np.zeros(A, np.float64)
|
||||
previous32 = np.zeros(A, np.float64)
|
||||
statistics = {name: _new_metric() for name in traces}
|
||||
normal_arithmetic = _normal_or_zero(I64, np.finfo(np.float32).tiny) and _normal_or_zero(O64, np.finfo(np.float32).tiny)
|
||||
max_active_terms = 0
|
||||
with torch.no_grad():
|
||||
for tick in range(steps):
|
||||
fired = event_node[boundaries[tick]:boundaries[tick+1]]
|
||||
replay_rt.spike_u8.zero_()
|
||||
if len(fired):
|
||||
replay_rt.spike_u8[torch.as_tensor(fired.astype(np.int64), device=view.device)] = 1
|
||||
before_calls = proxy.calls
|
||||
exec(update_code, scope)
|
||||
require(proxy.calls == before_calls + 1, 'Original update did not issue exactly one mv')
|
||||
actual = {'drive': proxy.drive.detach().cpu().numpy(),
|
||||
'state': scope['apl_state'].detach().cpu().numpy(),
|
||||
'inhibition': scope['inhibition'].detach().cpu().numpy()}
|
||||
for name in traces:
|
||||
same = bit_equal(actual[name], traces[name][tick])
|
||||
exact[name] &= same
|
||||
if not same and first_difference is None:
|
||||
first_difference = {'tick': tick, 'field': name,
|
||||
'max_abs_error': float(np.max(np.abs(actual[name].astype(np.float64)-traces[name][tick])))}
|
||||
# Float64 reference is measurement only and never feeds either brain.
|
||||
selected = I64[:, fired]
|
||||
drive64 = selected.sum(axis=1, dtype=np.float64)
|
||||
max_active_terms = max(max_active_terms, int(np.count_nonzero(selected, axis=1).max(initial=0)))
|
||||
Dupper = _up(drive64 / np.nextafter(1.0-gd64, -np.inf))
|
||||
drive_bound = _add(_mul(gd32, Dupper), _mul(gd64, Dupper))
|
||||
state32 = traces['state'][tick].astype(np.float64)
|
||||
normal_arithmetic &= _normal_or_zero(d*previous32, np.finfo(np.float32).tiny)
|
||||
normal_arithmetic &= _normal_or_zero(O64*state32[:, None], np.finfo(np.float32).tiny)
|
||||
e32 = _add(_mul(d, e32), _add(_mul(g232, _add(_mul(d, previous32), Dupper)),
|
||||
_mul(1.0+u32, _mul(gd32, Dupper))))
|
||||
e64 = _add(_mul(d, e64), _add(_mul(g264, _add(_mul(d, state64), Dupper)),
|
||||
_mul(1.0+u64, _mul(gd64, Dupper))))
|
||||
state64 = d*state64 + drive64
|
||||
inhibition64 = np.matmul(state64, O64)*q
|
||||
h32bound = _add(_mul(q, _positive_dot_upper(e32, O64)),
|
||||
_mul(gh32, _mul(q, _positive_dot_upper(state32, O64))))
|
||||
h64bound = _add(_mul(q, _positive_dot_upper(e64, O64)),
|
||||
_mul(gh64, _mul(q, _positive_dot_upper(state64, O64))))
|
||||
normal_arithmetic &= _normal_or_zero(q*np.matmul(state32, O64), np.finfo(np.float32).tiny)
|
||||
_accumulate(statistics['drive'], traces['drive'][tick], drive64, drive_bound)
|
||||
_accumulate(statistics['state'], traces['state'][tick], state64, _add(e32, e64))
|
||||
_accumulate(statistics['inhibition'], traces['inhibition'][tick], inhibition64, _add(h32bound, h64bound))
|
||||
previous32 = state32
|
||||
for metric in statistics.values():
|
||||
metric['mean_abs_error'] = metric.pop('sum_abs_error') / metric['elements']
|
||||
theory_applicable = bool(normal_arithmetic and not environment['cuda_matmul_allow_tf32'] and
|
||||
environment['float32_matmul_precision'] == 'highest')
|
||||
report.update(steps=steps, actual_spike_events=len(event_step),
|
||||
ast_replay_bitwise_exact=exact, first_ast_difference=first_difference,
|
||||
value_units={'drive': 'legacy normalized-contact spike drive',
|
||||
'state': 'legacy normalized-drive accumulator', 'inhibition': 'legacy mV'},
|
||||
decay_f32_bits=f'{int(decay32.view(np.uint32)):08x}', gain_f32_bits=f'{int(gain32.view(np.uint32)):08x}',
|
||||
float64_measurement=statistics,
|
||||
theoretical_bound={'applicable': theory_applicable, 'normal_or_zero_arithmetic': bool(normal_arithmetic),
|
||||
'gamma_definition': 'gamma_n=(n*u)/(1-n*u), upward-rounded; u32=2^-24,u64=2^-53',
|
||||
'drive_n': N-1, 'max_observed_nonzero_drive_terms': max_active_terms,
|
||||
'gamma_drive_f32': float(gd32), 'gamma_drive_f64': float(gd64),
|
||||
'state_recurrence': 'E_p[t] <= d*E_p[t-1] + gamma_2,p*(d*abs(s_p[t-1])+D_upper) + (1+u_p)*gamma_(N-1),p*D_upper',
|
||||
'output_bound': 'q*sum(B*E_p) + gamma_(2*A+1),p*q*sum(B*abs(s_p)); compare with E32+E64',
|
||||
'assumptions': 'nonnegative f32 matrices, spikes exactly 0/1, fixed stored f32 decay/gain, normal finite RN binary32/64 operations; no reduced-precision matmul',
|
||||
'bound_arithmetic': 'Each positive bound operation is rounded toward +infinity via nextafter; denominator toward -infinity',
|
||||
'tolerance_fitted_to_observed_error': False,
|
||||
'f64_values_used_to_drive_brain': False,
|
||||
'no_cpu_gpu_bitwise_expectation_for_f64_measurement': True},
|
||||
input_sha256={name: digest(gpu_case_dir/name) for name in
|
||||
('state.npz', 'summary.json', 'apl-drive.float32.npy', 'apl-state.float32.npy', 'apl-inhibition.float32.npy')})
|
||||
report['pass_all'] = bool(all(matrix_checks.values()) and all(exact.values()) and theory_applicable and
|
||||
all(value['bound_violations'] == 0 for value in statistics.values()))
|
||||
report['limitations'] = [
|
||||
'AST shadow and candidate use the same original Torch/CUDA primitives; this validates preserved component semantics, not an independent CPU whole-brain oracle.',
|
||||
'Float64 is a non-feedback mathematical measurement with explicit gamma error bounds, not replacement APL dynamics.',
|
||||
'Replay uses the recorded actual spike train; it does not establish equivalence for untested closed-loop inputs.',
|
||||
'The N-1 positive-sum bound is deliberately conservative; passing it does not prove physiological validity or learning.']
|
||||
save(out_json, report)
|
||||
return report
|
||||
except Exception as error:
|
||||
report['error'] = {'type': type(error).__name__, 'message': str(error)}
|
||||
if not out_json.exists():
|
||||
save(out_json, report)
|
||||
raise
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument('gpu_case_dir', type=Path)
|
||||
parser.add_argument('bundle_dir', type=Path)
|
||||
parser.add_argument('out_json', type=Path)
|
||||
args = parser.parse_args()
|
||||
result = validate(args.gpu_case_dir, args.bundle_dir, args.out_json)
|
||||
print(json.dumps({'pass_all': result['pass_all'], 'steps': result['steps']}), flush=True)
|
||||
if not result['pass_all']:
|
||||
raise SystemExit(1)
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Run old reference unchanged, writing only new experiment output."""
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
import numpy as np
|
||||
import pandas as pd
|
||||
import yaml
|
||||
from context import ROOT,EXP,CFG,save,check
|
||||
|
||||
|
||||
def main():
|
||||
check();assert not (EXP/'regression/frozen-five.json').exists()
|
||||
rtpath=ROOT/'models/male-cns-lif/deterministic-runtime-v2';pc=yaml.safe_load((rtpath/'phase4p.yaml').read_text())
|
||||
os.environ['CUDA_VISIBLE_DEVICES']=CFG['gpu_uuid'];os.environ['CUDA_HOME']='/usr/local/cuda'
|
||||
os.environ['PATH']=str(ROOT/'models/male-cns-lif/.venv/bin')+':/usr/local/cuda/bin:'+os.environ['PATH']
|
||||
os.environ['MAX_JOBS']='2';os.environ['TORCH_CUDA_ARCH_LIST']='8.9'
|
||||
sys.path.insert(0,str(ROOT/'models/male-cns-lif/terminal-scope-diagnostic-v1'))
|
||||
from diagnostic import queues
|
||||
queues();sys.path.insert(0,str(ROOT/'models/male-cns-lif'));sys.path.insert(0,str(rtpath))
|
||||
import torch,phase4p
|
||||
assert torch.cuda.mem_get_info()[0]>2*1024**3
|
||||
torch.cuda.reset_peak_memory_stats();t0=time.perf_counter();rt=phase4p.CanonicalRuntime(pc,rtpath,EXP);rows=[]
|
||||
for spec in pc['reference_trials']:
|
||||
queues();base=ROOT/'experiments/phase4p-deterministic-runtime/phase4c-v2/reference'/spec['name']
|
||||
ev=pd.read_parquet(base/'spike-events.parquet')
|
||||
ref=dict(step=ev.step.to_numpy(),neuron_idx=ev.neuron_idx.to_numpy(),counts=np.load(base/'per-neuron-spike-counts.int32.npy'),v=np.load(base/'final-voltage.float32.npy'),g=np.load(base/'final-g.float32.npy'))
|
||||
result=rt.run(np.flatnonzero(rt.pops[spec['population']]),spec['rate_hz'],spec['seed'],spec['duration_ms'])
|
||||
row=dict(trial=spec['name'],exact={k:bool(np.array_equal(v,result[k])) for k,v in ref.items()},wall_seconds=result['metrics']['wall_seconds'])
|
||||
rows.append(row);print(row,flush=True)
|
||||
save(EXP/'regression/frozen-five.json',dict(pass_all=all(all(r['exact'].values()) for r in rows),trials=rows,
|
||||
wall_seconds=time.perf_counter()-t0,peak_allocated_vram=int(torch.cuda.max_memory_allocated()),oracle='old frozen deterministic-v2 only'))
|
||||
assert all(all(r['exact'].values()) for r in rows)
|
||||
|
||||
|
||||
if __name__=='__main__':main()
|
||||
Reference in New Issue
Block a user