Add licensed public-resource archive plan and verified LFS tooling

2026-10-01 03:21:48 +08:00
parent be41af186c
commit 7dc85baea9
9 changed files with 1133 additions and 0 deletions
@@ -0,0 +1,55 @@
+# 2026-10-01 公开 LFS 资源发布
+
+这是现有下载资源的归档发布，不是新的科学实验，不表示 Phase 4AR 已完成或学习通过。用户明确选择保留公开仓库，仅上传确认可公开再分发的资源。活动项目工作树和 Phase 1–4AQ 冻结文件不参与写入。
+
+## 范围与许可
+
+| 资源 | 许可/依据 | 本次处理 |
+|---|---|---|
+| MaleCNS v1.0 annotations、neurotransmitters、connectome、SWC、synaptic partners | [官方数据页](https://male-cns.janelia.org/download/)明确指向 [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) | 逐文件原字节归档、署名、来源与 SHA-256 |
+| 原有 Phase 3 / 3B processed 数据 | 基于上述 CC BY 4.0 数据的本项目派生物 | 保留原有过滤/重编号/CSR 转换说明；不重新计算、不更改科学含义 |
+| Cell Type Explorer | 固定 commit 的 LICENSE，CC BY 4.0 | 原字节源码/数据快照，保留 LICENSE |
+| visualpathways | 固定 commit 的 LICENSE 为 GPL v3 文本 | 原字节源码及 companion resources，保留 notices；不替上游判断 only/or-later |
+| DoOR.data 2.0.1 | 固定 commit 的 DESCRIPTION：CC BY-SA 4.0 | 保留作者、原数据来源、ShareAlike 条件 |
+| Shiu Drosophila_brain_model | 固定 commit 的 MIT LICENSE | 已下载作者模型源码与环境描述，不称预训练权重 |
+
+每组精确来源、commit、许可证据 SHA 和修改说明见 `plan.json` 及资源 `catalog.json`；许可证原文随对应上游副本保留。MaleCNS 原数据署名：FlyEM/HHMI Janelia、University of Cambridge Department of Zoology、MRC Laboratory of Molecular Biology、Google Research 及原论文作者。归档封装不意味着这些机构认可本项目模型或结论。
+
+DoOR 作者为 Daniel Münch、C. Giovanni Galizia、Shouwen Ma、Martin Strauch、Anja Nissler；[DoOR 2.0 论文](https://doi.org/10.1038/srep21841)及[原始 DoOR 论文](https://doi.org/10.1093/chemse/bjq042)只引用，不另外打包出版商全文。各第三方资源维持原许可，本项目没有给整个仓库强行统一换证。
+
+## 明确不上传
+
+- FlyVis：冻结 checkout 仅见 MIT classifier，缺独立 LICENSE；数据和预训练权重的再分发权未确认。实际盘点未发现独立下载的预训练 checkpoint/ZIP；不得把源码、几何坐标或功能通道桥接称为下载权重。
+- `datasets/male-cns-v1.0/backups/` 的既有 Phase 1/2 备份包：保留原地；本次已直接归档其 raw 数据，不再重复上传整份备份。
+- Phase 4AD 重复的 Shiu 作者源码：本次仅归档 Phase 4AC 副本；既有目录及历史索引保留。
+- 历史本项目 learned delta、当前 Phase 4AR 正在写入的结果/账本、虚拟环境、缓存、Git 内部目录、凭据、下载日志。
+- ComfyUI 及其他项目、原始 EM 图像、未下载的 syn-points、许可不明的出版商论文全文。
+- visualpathways 的 `pdfversion/` 及 `docs/column_coord.pdf`：已识别为作者图集/配套说明而非一般论文集合，但本次保守等待输出 PDF 的明确许可范围，不把生成程序的 GPL 自动视作所有输出的许可；源码和其余已明确仓库许可的资源仍保留。
+
+未列入 plan 的目录默认不发布。额外路径剔除及环境/缓存规则的实际命中保存在各组 `exclusions.jsonl.gz`；文件内容未被改写或“脱敏后替代原件”。
+
+## 存储及校验
+
+归档位于 `artifacts/public-resources-20261001/<group>/`。每组为独立 tar+zstd，切成最多 64 MiB 的有序 LFS 分块，方便大文件传输中断后重试未完成对象。`members.jsonl.gz` 对每个原始相对路径记录大小和 SHA-256；`catalog.json` 记录每块及拼接归档 SHA、源总数、源字节数、来源和许可。分块改变的是存储容器，不是内部文件内容。
+
+`pack-summary.json` 存在且 complete=true 仅表示本地打包完成，不等于上传/远程恢复成功。上传后使用独立 clone 和新的 LFS object store 实际下载，再逐块、逐归档、逐内部文件流式校验。对应 receipt 只有在真实校验完成后才会提交。
+
+本次不是服务器所有内容的完整灾备。Gitea LFS 与 Git 仓库数据目录均需另有服务端备份；删除服务端 LFS 对象会破坏恢复能力。尚未核验 Gitea 主机磁盘容量和备份策略。
+
+## 安全恢复（Linux 示例）
+
+先安装 Git LFS、Python 3 和 zstd。只在新目录恢复，不要覆盖运行中的 `/data/digital-drosophila`。
+
+```sh
+GIT_LFS_SKIP_SMUDGE=1 git clone --branch codex/lfs-public-resources ssh://git@192.168.10.31:2222/qishizhe/digital-drosophila.git digital-drosophila-restore
+cd digital-drosophila-restore
+git lfs install --local
+git lfs pull --include='artifacts/public-resources-20261001/**'
+python3 -B tools/lfs-publication/publish_archives.py verify --group-dir artifacts/public-resources-20261001/malecns-swc
+```
+
+对每组运行 verify 并全部通过后，按 catalog 顺序拼接 `payload-000000.bin` 等文件，可用 `zstd -d` 和 tar 在**新建的空恢复目录**解包。归档路径从 `datasets/` 或 `experiments/` 起，无绝对路径。不要运行归档中模型/下载脚本来“验证”，本轮只验证内容。正式使用前仍需检查对应科学阶段 manifest；旧 all-body Phase 3 不能替代已批准的 neuron-only SNN 输入。
+
+## 限制
+
+快照通过源文件前后 inode/size/mtime/ctime 和枚举一致性检查，不是文件系统原子快照。秘密扫描是有限的 UTF-8 小文本规则，不是对所有二进制、内嵌归档及第三方资源的安全担保。工具内容校验不独立证明 catalog 的真实性；应固定仓库 commit 和 catalog SHA。源码只读校验不能被表述为重新执行全历史科学 gate。
@@ -0,0 +1,21 @@
+{
+  "schema": "lfs-smoke-roundtrip-v1",
+  "fixture": "artifacts/lfs-smoke/publication-probe.bin",
+  "bytes": 263,
+  "sha256": "93e0e6c4ca4fdc123c21b859ba97d8bd73670eb9c90d04ce9a431d822b5393c5",
+  "uploaded_commit": "be41af186c3e1831dbda0dd8d5f151bf3901552f",
+  "branch": "codex/lfs-public-resources",
+  "upload_exit_code": 0,
+  "fresh_clone_independent_lfs_cache": true,
+  "download_exit_code": 0,
+  "download_sha256_match": true,
+  "lfs_fsck_dry_run_exit_code": 0,
+  "lfs_client_version": "3.8.0",
+  "client_archive_url": "https://github.com/git-lfs/git-lfs/releases/download/v3.8.0/git-lfs-linux-amd64-v3.8.0.tar.gz",
+  "client_archive_sha256": "e455e00f15d9b95661b8d53498ffb0c3367962cf1ec73c31ab7369516cd6ab8d",
+  "client_verification": "SHA-256 matched official GitHub release asset API digest; no PGP verification claimed",
+  "server_version_observed": "1.26.2",
+  "repository_private_observed": false,
+  "secret_headers_saved": false,
+  "scope": "Synthetic 263-byte fixture only; not evidence that scientific payloads have been uploaded"
+}
@@ -0,0 +1,60 @@
+{
+  "schema": "lfs-publication-plan-v1",
+  "source_root": "/data/digital-drosophila",
+  "groups": [
+    {
+      "id": "malecns-raw-core",
+      "paths": ["datasets/male-cns-v1.0/raw/annotations", "datasets/male-cns-v1.0/raw/connectivity", "datasets/male-cns-v1.0/raw/_manifests"],
+      "license": {"data": "CC-BY-4.0", "url": "https://creativecommons.org/licenses/by/4.0/", "local_manifests_scripts": "Project-authored provenance; no new blanket software license"},
+      "provenance": {"version": "male-cns:v1.0", "source_page": "https://male-cns.janelia.org/download/", "source_prefix": "gs://flyem-male-cns/v1.0/connectome-data/flat-connectome/", "attribution": "Male CNS collaboration: FlyEM (HHMI Janelia), University of Cambridge Department of Zoology, MRC Laboratory of Molecular Biology, Google Research; retain original dataset author credits", "changes": "Data file bytes unchanged; archival compression and path manifest only. Download logs excluded."}
+    },
+    {
+      "id": "malecns-swc",
+      "paths": ["datasets/male-cns-v1.0/raw/skeletons"],
+      "license": {"data": "CC-BY-4.0", "url": "https://creativecommons.org/licenses/by/4.0/"},
+      "provenance": {"version": "male-cns:v1.0", "source_page": "https://male-cns.janelia.org/download/", "source_prefix": "gs://flyem-male-cns/v1.0/segmentation/skeletons-malecns/skeletons-swc/", "attribution": "Male CNS collaboration: FlyEM (HHMI Janelia), University of Cambridge Department of Zoology, MRC Laboratory of Molecular Biology, Google Research", "changes": "Unchanged downloaded SWC bytes; not restricted to the later neuron-only canonical set. No EM images."}
+    },
+    {
+      "id": "malecns-processed",
+      "paths": ["datasets/male-cns-v1.0/processed"],
+      "license": {"derived_data": "CC-BY-4.0", "url": "https://creativecommons.org/licenses/by/4.0/", "local_scripts_reports": "Project-authored processing and provenance; no new blanket software license"},
+      "provenance": {"dataset": "male-cns:v1.0", "source_page": "https://male-cns.janelia.org/download/", "attribution": "Original Male CNS collaboration; Digital Drosophila project for local processing", "changes": "Existing Phase 3 all-body and Phase 3B neuron-only tables, index mappings, CSR and audit reports, copied byte-for-byte. These were previously derived by filtering/reindexing official data, not new preprocessing. Phase 3 historical outputs are not the approved SNN input; use neuron-only for current runtime.", "canonical_neurons": 166700, "canonical_edges": 25582938, "canonical_synaptic_weight_sum": 124177617}
+    },
+    {
+      "id": "malecns-synaptic-partners",
+      "paths": ["experiments/phase4aa-apl-pn-scope/external-resources"],
+      "license": {"data": "CC-BY-4.0", "url": "https://creativecommons.org/licenses/by/4.0/", "local_provenance": "Project-authored"},
+      "provenance": {"source_page": "https://male-cns.janelia.org/download/", "source_url": "https://storage.googleapis.com/flyem-male-cns/v1.0/connectome-data/flat-connectome/syn-partners-male-cns-v1.0-minconf-0.5.feather?generation=1780494942562468", "registered_file_sha256": "959d8ef4173b35382a3e6acfaf5167c795b6d10b877572d146af04e1b487bc07", "attribution": "Male CNS collaboration: FlyEM (HHMI Janelia), University of Cambridge Department of Zoology, MRC Laboratory of Molecular Biology, Google Research", "changes": "Unchanged downloaded paired-synapse file and provenance; syn-points and EM were not downloaded."}
+    },
+    {
+      "id": "celltype-explorer",
+      "paths": ["experiments/phase4f-retinotopic-vision/external-resources/celltype-explorer"],
+      "license": {"upstream": "CC-BY-4.0", "url": "https://creativecommons.org/licenses/by/4.0/", "evidence_file": "LICENSE", "evidence_sha256": "9ba9550ad48438d0836ddab3da480b3b69ffa0aac7b7878b5a0039e7ab429411"},
+      "provenance": {"repository": "https://github.com/reiserlab/celltype-explorer-drosophila-male-cns", "commit": "789cc6c105798ce2fd70ba85dab394f90899616b", "attribution": "Reiser Lab and Male CNS Cell Type Explorer contributors; retain upstream notices", "changes": "Unchanged pinned working-tree file bytes; .git, environments, caches and credentials excluded. A source snapshot, not a full Git clone."}
+    },
+    {
+      "id": "visualpathways",
+      "paths": ["experiments/phase4f-retinotopic-vision/external-resources/visualpathways"],
+      "license": {"upstream": "GPL-3.0 as supplied in upstream LICENSE; no assertion of only/or-later option", "evidence_file": "LICENSE", "evidence_sha256": "3972dc9744f6499f0f9b2dbf76696f2ae7ad8af9b23dde66d6af86c9dfb36986", "third_party_notices": "Retained; no relicensing of third-party components"},
+      "provenance": {"repository": "https://github.com/reiserlab/visualpathways", "commit": "23f6ac131529b5f56894c6eeb9b88b17894fc00d", "attribution": "Reiser Lab and visualpathways contributors; original copyright notices and source retained", "changes": "Unchanged source/resource file bytes from pinned checkout; .git, credentials, environments and caches omitted; explicit exclusions recorded. No software execution or model alteration."}
+    },
+    {
+      "id": "door-data",
+      "paths": ["experiments/phase4l-olfactory-grounding/external-resources/DoOR.data"],
+      "license": {"upstream": "CC-BY-SA-4.0", "url": "https://creativecommons.org/licenses/by-sa/4.0/", "evidence_file": "DESCRIPTION", "evidence_sha256": "27db2a92eb651d67969de94959990d51668a5857a33c4750a3b6ef49c5880504"},
+      "provenance": {"repository": "https://github.com/ropensci/DoOR.data", "commit": "eea72d64b42a4e91dafe822cf0b5204de33f8d34", "version": "2.0.1", "attribution": "Daniel Münch, C. Giovanni Galizia, Shouwen Ma, Martin Strauch, Anja Nissler; see upstream DESCRIPTION and data-source references", "papers": ["https://doi.org/10.1038/srep21841", "https://doi.org/10.1093/chemse/bjq042"], "changes": "Unchanged pinned data package; .git omitted. ShareAlike terms remain attached to this package, not asserted over unrelated repository content."}
+    },
+    {
+      "id": "shiu-author-model-source",
+      "paths": ["experiments/phase4ac-lif-semantics/external-resources"],
+      "license": {"upstream": "MIT", "evidence_file": "LICENSE", "evidence_sha256": "3621f6d6476189190e2960fa43f11b275ba4eca848fdac50a1ba2925de6223e8"},
+      "provenance": {"repository": "https://github.com/philshiu/Drosophila_brain_model", "commit": "91bdd1e7dcf193f3e7ca5a8933497fcef63b7960", "attribution": "Philip Shiu and Drosophila_brain_model contributors; original MIT copyright and permission notice retained", "paper": "https://www.nature.com/articles/s41586-024-07763-9", "changes": "The previously downloaded model.py, environment_full.yml, LICENSE and local provenance only. This is model source, not pretrained weights, and not a strict original-model reproduction claim."}
+    }
+  ],
+  "exclude_paths": [
+    "datasets/male-cns-v1.0/processed/_staging",
+    "experiments/phase4f-retinotopic-vision/external-resources/visualpathways/.env_sample",
+    "experiments/phase4f-retinotopic-vision/external-resources/visualpathways/pdfversion",
+    "experiments/phase4f-retinotopic-vision/external-resources/visualpathways/docs/column_coord.pdf"
+  ]
+}
@@ -0,0 +1,123 @@
+{
+  "disk_free_bytes": {
+    "/data/digital-drosophila": 645731733504,
+    "/tmp/malecns-lfs-publication-20261001-bI1u07/repo/registry/publication-20261001": 905806151680
+  },
+  "license_evidence": [
+    {
+      "bytes": 18657,
+      "group": "celltype-explorer",
+      "path": "/data/digital-drosophila/experiments/phase4f-retinotopic-vision/external-resources/celltype-explorer/LICENSE",
+      "sha256": "9ba9550ad48438d0836ddab3da480b3b69ffa0aac7b7878b5a0039e7ab429411"
+    },
+    {
+      "bytes": 35149,
+      "group": "visualpathways",
+      "path": "/data/digital-drosophila/experiments/phase4f-retinotopic-vision/external-resources/visualpathways/LICENSE",
+      "sha256": "3972dc9744f6499f0f9b2dbf76696f2ae7ad8af9b23dde66d6af86c9dfb36986"
+    },
+    {
+      "bytes": 1042,
+      "group": "door-data",
+      "path": "/data/digital-drosophila/experiments/phase4l-olfactory-grounding/external-resources/DoOR.data/DESCRIPTION",
+      "sha256": "27db2a92eb651d67969de94959990d51668a5857a33c4750a3b6ef49c5880504"
+    },
+    {
+      "bytes": 1085,
+      "group": "shiu-author-model-source",
+      "path": "/data/digital-drosophila/experiments/phase4ac-lif-semantics/external-resources/LICENSE",
+      "sha256": "3621f6d6476189190e2960fa43f11b275ba4eca848fdac50a1ba2925de6223e8"
+    }
+  ],
+  "observed_utc": "2026-09-30T19:12:34.846175+00:00",
+  "original_checkout_branch": "codex/lif-author-semantics-lab",
+  "original_checkout_head": "09187e4d4d67badf1b418c5f211258ca5d8fb328",
+  "original_checkout_status": "",
+  "pass_all": true,
+  "phase4ar_registered_sources": [
+    {
+      "bytes": 5374,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/PROTOCOL.md",
+      "sha256": "d61d2cb6a6bff31b5762edb78bb6406ac7d53b1e5eff30aad26eb0eea99fc313"
+    },
+    {
+      "bytes": 29087,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/append_ledger.py",
+      "sha256": "1f981f61cd26950e7d4b5393be70b82753ea7e721c0bb75d05f8fd1e92a453fb"
+    },
+    {
+      "bytes": 7813,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/normal_inputs.py",
+      "sha256": "df8f346b8fba54bb54f0b1a3f81c1511d54987684064094d0dfa46043be03e0d"
+    },
+    {
+      "bytes": 3317,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/normal_io.py",
+      "sha256": "fe6eee89ae98392a28120b69a966d7967916be94376cb7494ef3ce77e63506e0"
+    },
+    {
+      "bytes": 37143,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/normal_metrics.py",
+      "sha256": "82af63d8fadd5a6b67083f5e303ebe88674b76f9a2a8a8fff082b5c828483533"
+    },
+    {
+      "bytes": 11254,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/normal_plan.py",
+      "sha256": "d1a6b840a547cb10b00df40f4d8b287867d09f355c9777e5f7a8b9489f5555ea"
+    },
+    {
+      "bytes": 15006,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/normal_prepare.py",
+      "sha256": "9c8568a3d8c1049ada8737b71192952f72cdd27ccb3b470195e30d6a4b1cea68"
+    },
+    {
+      "bytes": 24699,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/normal_runner.py",
+      "sha256": "41a0c71508d0b39c9488bddeaa4ac2a669312c632f16d5f9b5de24dff2eb14c1"
+    },
+    {
+      "bytes": 1534,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/run_cpu_tests.py",
+      "sha256": "10ed189053731e1661284cf890ca3504a272af05b4531bdb3655b6e88cab6ff9"
+    },
+    {
+      "bytes": 3043,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/run_normal.py",
+      "sha256": "707b6be41caef7af83aa68ea1f102f10c9a4ba81a22c564e54eddf63857e81f3"
+    },
+    {
+      "bytes": 15199,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/test_append_ledger.py",
+      "sha256": "0f638c7b83117410d7fa074ec9f2ef865d9810171b73991c73a78414be581bc6"
+    },
+    {
+      "bytes": 4292,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/test_normal_inputs.py",
+      "sha256": "7aea6a3bcbbf78c3b3d671b33b83cd4d992b2a8ae9c888e29cdf13f9fa420a45"
+    },
+    {
+      "bytes": 17862,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/test_normal_metrics.py",
+      "sha256": "204586b30edbe57bb294c1d8f8ad309863e5bcb3baad32118955ac3d4adcdefb"
+    },
+    {
+      "bytes": 9184,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/test_normal_plan.py",
+      "sha256": "790bca529c58e5214c19a6dcaf07511fca11c07fb9b05ca5d71d3f06160bbef4"
+    },
+    {
+      "bytes": 25798,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/test_normal_runner.py",
+      "sha256": "ca9f69c731e56c50534d864d0ce780f286b5ab2d786a1a85fadb22dc09c0ccf5"
+    },
+    {
+      "bytes": 2076,
+      "path": "/data/digital-drosophila/models/male-cns-lif/normal-conditioning-v1/test_run_normal.py",
+      "sha256": "2ee3ad3c58c0dfd83aaaa4c4c9bd01e84eb245c61048c105ed8f3031db74cb40"
+    }
+  ],
+  "registered_scientific_commit": "80aeb2093db655329774eff7b32fb6e417b8b40b",
+  "registration_sha256": "b63e6281b6505ca3b60acb467dd0b3fae274cf513dcbfd653649ba52e9ba9740",
+  "schema": "lfs-publication-source-guard-v1",
+  "scope": "16 registered Phase 4AR source files, registration, pinned upstream checkout and license evidence; not a scientific rerun or full historical checksum audit"
+}