revert: 移除全部机器学习代码，回退到无 ML 版本
删除 RL/HRL/训练前端及其文档、脚本、测试与依赖：
- src/mcm26b/{rl,hrl,training_ui}/、planner/{rl,hrl}_loop.py、process_control.py
- project/（PPO 策略、训练、环境、评估）
- scripts/{train_rl,train_hrl_*,benchmark_rl,benchmark_hrl,pretrain_*_expert 等}
- tests/{test_rl*,test_hrl*,test_training_ui*}、tests/js/training_ui_harness.cjs
- docs/{algo-rl,hrl-*,training-ui*} 与相关计划条目、论文 PDF 回退
CLI 恢复为 grid/seven/bayes + directional，移除 --strategy rl|hrl、
--q4-strategy、--rl-model、--hrl-* 等参数；pyproject 删除 rl 依赖组与
PyTorch CUDA index，uv.lock 同步重生成。
保留此后的非 ML 改动（referee 回放、Q1/Q3 脚本、代码清理等）。
验证：pytest 232 passed；Q3/Q4 --self-test 全清通过。
This commit is contained in:
Horldsence
@@ -1,87 +0,0 @@
|
|||||||
# 分层强化学习（HRL）实施计划 — 2026 高教社杯 B 题第三、四问
|
|
||||||
|
|
||||||
## 0. 现状与复用决策
|
|
||||||
|
|
||||||
仓库已有生产级基建，**全部复用、不重写**：
|
|
||||||
|
|
||||||
| 现有组件 | 位置 | 用途 |
|
|
||||||
|---|---|---|
|
|
||||||
| `SimulatorClient`（HTTP+JSON、幂等重试、截止时钟、JSONL 日志） | `src/mcm26b/protocol/` | 模拟器对接（prompt 第 1 步已完成） |
|
|
||||||
| 本地评测机（内存回环，一局秒级） | `referee/` + `harness.LocalSimTransport` | 本地简化环境内核 |
|
|
||||||
| 网格幸存集滤波（真源保留定理已证明） | `src/mcm26b/particles.py` | 每频道信念后验 |
|
|
||||||
| 凸候选集 + 五状态机 | `src/mcm26b/belief/` | 频道状态观测 |
|
|
||||||
| 信息增益选点、TSP 2-opt、六边形覆盖、清除证书、后备清除 | `src/mcm26b/planner/` | 候选生成器 + 证书审批兜底 + 启发式基线 |
|
|
||||||
| 单步 MaskablePPO 全套（特征/掩码/BC/训练/审计） | `src/mcm26b/rl/` + `scripts/` | 代码模式模板 + 端到端对照基线 |
|
|
||||||
|
|
||||||
**改进方向**（来自 v2 审计文档已知局限）：单步 RL 只做选点、学不到全场调度；Q4 后备清除偏多、尾部退化（13778s→11797s，距 4000s 目标远）。HRL 高层正是补**跨频道全场调度**决策。
|
|
||||||
|
|
||||||
**已确认的三个决策**：① 代码集成到 `src/mcm26b/hrl/`（README 给出与 prompt 目录树的映射表）；② 低层为字面 8 方向 PPO + 直线控制器对照（诚实预期：部署时只提交终点，低层对虚拟时间增益≈0，价值在架构完整性与消融验证）；③ 本次跑通全流程 + 短程训练（1–2h），完整课程训练交付脚本由用户自行启动。
|
|
||||||
|
|
||||||
## 1. 新建文件
|
|
||||||
|
|
||||||
```
|
|
||||||
src/mcm26b/hrl/
|
|
||||||
├── __init__.py
|
|
||||||
├── constants.py # HRL 接口常量：N_CANDIDATES=96、观测维度，feature_version=3
|
|
||||||
├── candidates.py # 高层候选生成：待清除源/信息增益探索点/覆盖验证点/Q4 扇区探测点 + 动作掩码
|
|
||||||
├── features.py # 高层观测（20 频道×14 统计 + 全局标量 + 96×6 候选特征 ≈864 维）、低层观测
|
|
||||||
├── policy.py # HighLevelPolicy/LowLevelPolicy 加载器（沿用 PointSelectionPolicy 元数据校验模式）
|
|
||||||
├── low_level_env.py # 低层训练环境：随机起点→子目标，Discrete(8)，逐步计费，奖励=−Δd−转向+到达
|
|
||||||
├── hierarchical_env.py # 高层整场环境：10–16 源，动作=子目标槽位，内嵌低层执行器，域随机化+课程参数
|
|
||||||
└── executor.py # 子目标执行器：低层 PPO 步进（本地计费）或直线控制器（可切换），终点提交模拟器
|
|
||||||
|
|
||||||
scripts/
|
|
||||||
├── train_hrl_low.py # 低层：BC 克隆直线控制器热启动 → MaskablePPO 微调；直线对照评估
|
|
||||||
├── train_hrl_high.py # 高层：4 阶段课程（--stage --resume --timesteps），SubprocVecEnv 并行
|
|
||||||
├── pretrain_hrl_expert.py # 可选：从 BayesLoop 调度器克隆 (obs, 子目标) 对热启动高层
|
|
||||||
├── tune_hrl.py # Optuna TPE（prompt §10 搜索空间，预算 15–20 trials，初值复用 bayes 已调参数）
|
|
||||||
├── benchmark_hrl.py # 配对消融：HRL vs bayes/directional/单步 PPO，dev(700-702)/val(91000-91005)/stress 固定场景
|
|
||||||
└── ablate_hrl.py # 四项消融：无信息增益奖励/无动作掩码/无课程/无粒子滤波
|
|
||||||
|
|
||||||
src/mcm26b/planner/hrl_loop.py # HRLLoop 运行时闭环（Q3/Q4），保留证书审批与异常留痕
|
|
||||||
src/mcm26b/cli.py # 新增 --strategy hrl / --q4-strategy hrl / --hrl-model
|
|
||||||
tests/test_hrl.py # 特征布局、掩码、环境形状、奖励恒等式、训练-部署观测一致性
|
|
||||||
docs/hrl.md # 设计文档 + prompt 目录树映射表
|
|
||||||
docs/paper/ 增补 # 算法流程图、伪代码、超参表、训练曲线
|
|
||||||
```
|
|
||||||
|
|
||||||
## 2. 核心设计
|
|
||||||
|
|
||||||
**高层动作空间** Discrete(96) + 掩码：槽位按固定顺序填充（已定位待清除源 ≤16 → PF 驱动探索点 → 覆盖网格未验证点 → Q4 扇区探测点），空槽/已清除频道/接收盘外候选掩码 −∞。
|
|
||||||
|
|
||||||
**高层观测**（≈864 维，参照 v2 的 401 维布局模式）：20 频道 × 14 特征（六状态 one-hot、MEC μx/μy、σx/σy/ρ、log 幸存格数、期望信息增益、距离、频道号）+ 全局 8 维（位置、时间比例、已清除数、最近检测结果编码/示向度/距离）+ 96×6 候选特征。
|
|
||||||
|
|
||||||
**step 语义**：高层选子目标 → executor 步进（低层 8 方向，每步 ≤100m，本地按 Δd/5 计费；部署时仅终点调模拟器）→ 按子目标类型执行 /measure 或 /clear → belief+PF 更新 → 奖励。
|
|
||||||
|
|
||||||
**奖励**：`r = −Δ虚拟耗时/100 + 100·清除 + 5·新检测 + w_info·ΔH − 冗余惩罚`，主口径与目标函数（虚拟时间）一致；w_info 可消融。
|
|
||||||
|
|
||||||
**课程 4 阶段**（prompt §7.1）：1 源/5 频道/无噪 → 3 源/10 频道/±0.5° → 8 源/16 频道/±1° → 10–16 源/20 频道/±1°；每阶段迁移上阶段权重，Q4 从 Q3 权重微调（定向比例递增），域随机化贯穿（源位置/频道/r_eff/朝向/噪声种子）。
|
|
||||||
|
|
||||||
**运行时闭环 HRLLoop**：初始覆盖相位（快速发现信号）→ 主循环高层决策（证书达标自动 /clear，与 BayesLoop 同语义）→ 全清即 /exit。不触碰候选集更新与清除判定的守恒性（真源保留定理依赖），异常留痕兜底保留。
|
|
||||||
|
|
||||||
## 3. 实施顺序（对应 prompt §12）
|
|
||||||
|
|
||||||
1. `constants.py` + `candidates.py` + `features.py`（候选/掩码/观测）+ 单元测试
|
|
||||||
2. `low_level_env.py` + `train_hrl_low.py`：低层短程训练，验证收敛到近直线（报告与直线最优的差距）
|
|
||||||
3. `hierarchical_env.py` + `executor.py` + `hierarchical_env` 单元测试（奖励恒等式：回合回报 = f(虚拟时间、清除数)）
|
|
||||||
4. `train_hrl_high.py` + 课程 + 域随机化 + SubprocVecEnv；`pretrain_hrl_expert.py` BC 热启动
|
|
||||||
5. 短程课程训练（每阶段 2–5 万步，总计 1–2h），产出训练曲线
|
|
||||||
6. `hrl_loop.py` + CLI 接入，`--local-judge` 本地演练验证全清正确性
|
|
||||||
7. `benchmark_hrl.py` 配对对比（沿用 benchmark_rl.py 的 all_valid 有效口径）+ `ablate_hrl.py` 四项消融
|
|
||||||
8. `tune_hrl.py` 受限 Optuna；产出最佳超参表
|
|
||||||
9. 本地 referee 演练 ×3/题，表 1 格式结果（案例编码/清除数/平均定位清除时间/程序运行时间），日志 JSONL 原样归档 `output/hrl/`
|
|
||||||
10. `docs/hrl.md`、README 更新、论文素材（流程图/伪代码/超参表/对比分析，含"为什么分层 PPO、为什么非端到端、为什么非多智能体"论证）
|
|
||||||
|
|
||||||
## 4. 验收标准
|
|
||||||
|
|
||||||
- 全部 pytest 通过（现有 243 项 + 新增 HRL 测试）
|
|
||||||
- `benchmark_hrl.py` 输出 all_valid=true（真值全清、缺席无误判、零异常、正常 exit、计时复算一致）
|
|
||||||
- 短程训练下 Q4 HRL 相对 directional 有可测改进方向（对标单步 RL 的 −14.38%）；Q3 如实报告与 bayes 的差距（v2 审计口径：Q3 RL 无稳定增益）
|
|
||||||
- 所有实验固定种子、命令可复现；消融表含均值±方差（≥5 种子）
|
|
||||||
- 完整训练只需用户执行 `scripts/train_hrl_high.py --full` 级别的命令
|
|
||||||
|
|
||||||
## 5. 风险与对策
|
|
||||||
|
|
||||||
- **高层整场回合过长（100–200 决策步）不收敛** → γ=1 + 密集 Δcost 奖励 + BC 从 bayes 调度器热启动
|
|
||||||
- **低层增益≈0** → 如实消融报告（直线对照差异≈0），论文定位为架构必要性验证而非性能来源
|
|
||||||
- **Q3 可能无增益** → 与 v2 口径一致如实报告，主打 Q4 调度收益；两种策略可共存（HRL 不改默认策略，仅在胜出后由用户决定切换默认）
|
|
||||||
@@ -7,10 +7,6 @@
|
|||||||
- [协议规则表、字段表与状态机（P0 交付）](docs/协议规则表与状态机.md)
|
- [协议规则表、字段表与状态机（P0 交付）](docs/协议规则表与状态机.md)
|
||||||
- [问题一完整解答](docs/问题一解答.md)
|
- [问题一完整解答](docs/问题一解答.md)
|
||||||
- [问题三搜索与总耗时改进方案（含七点覆盖证明和自建消融）](docs/algo-3.md)
|
- [问题三搜索与总耗时改进方案（含七点覆盖证明和自建消融）](docs/algo-3.md)
|
||||||
- [问题3/4 强化学习选点（PPO + CUDA）](docs/algo-rl.md)
|
|
||||||
- [强化学习审核、修复与独立验证（2026-09-11）](docs/强化学习适用性审核与改进-20260911.md)
|
|
||||||
- [HRL 优化方案与 Sol 执行指南（含当前缺口、命令和验收门槛）](docs/hrl-sol-execution.md)
|
|
||||||
- [HRL P0+P1 执行报告（本轮实际命令、smoke 与多轮本地测试）](docs/hrl-p1-execution-report.md)
|
|
||||||
- [论文](docs/paper/main.tex)
|
- [论文](docs/paper/main.tex)
|
||||||
|
|
||||||
## 项目结构
|
## 项目结构
|
||||||
@@ -25,13 +21,10 @@ src/mcm26b/
|
|||||||
├── belief/ # P3 逐频道候选集合 K_c：保守外包多边形 + 半平面裁剪（已完工）
|
├── belief/ # P3 逐频道候选集合 K_c：保守外包多边形 + 半平面裁剪（已完工）
|
||||||
├── coverage/ # P4 49点覆盖账本（网格生成 + 逐频道记录）
|
├── coverage/ # P4 49点覆盖账本（网格生成 + 逐频道记录）
|
||||||
├── planner/ # P3 保证接收盘选点器 / P4 全向闭环 / P5 混合源定向闭环（已完工）
|
├── planner/ # P3 保证接收盘选点器 / P4 全向闭环 / P5 混合源定向闭环（已完工）
|
||||||
│ # P6b RL 接入：planner/rl_loop.py（PPO 选点，惰性加载 torch）
|
|
||||||
├── rl/ # P6b 强化学习扩展：训练环境、观测/候选特征、PPO 模型加载（可选依赖）
|
|
||||||
├── evaluation/ # 日志复算与统计（已完工）
|
├── evaluation/ # 日志复算与统计（已完工）
|
||||||
└── harness/ # 测试侧适配：评测机<->Transport 内存回环、故障注入、回放、场景
|
└── harness/ # 测试侧适配：评测机<->Transport 内存回环、故障注入、回放、场景
|
||||||
tests/ # pytest（含解耦守卫：运行时零导入 referee + 规则表不漂移）
|
tests/ # pytest（含解耦守卫：运行时零导入 referee + 规则表不漂移）
|
||||||
scripts/ # 采集、基准、可视化与校核脚本（含问题四批量仿真 benchmark_q4.py、
|
scripts/ # 采集、基准、可视化与校核脚本（含问题四批量仿真 benchmark_q4.py）
|
||||||
# RL 训练 train_rl.py 与 RL 配对消融 benchmark_rl.py）
|
|
||||||
output/ examples/ # 演练证据、算例与生成产物
|
output/ examples/ # 演练证据、算例与生成产物
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -71,29 +64,11 @@ Python ≥ 3.10，运行时依赖仅 `requests`；测试/自检另需 `pytest`
|
|||||||
|
|
||||||
```bash
|
```bash
|
||||||
uv sync # 安装（可编辑挂载本包与 referee）
|
uv sync # 安装（可编辑挂载本包与 referee）
|
||||||
uv sync --group rl --group dev # 追加 RL 依赖：torch(CUDA)+stable-baselines3+gymnasium
|
|
||||||
uv run pytest # 全部测试（含 referee 90 项 + 解耦守卫）
|
uv run pytest # 全部测试（含 referee 90 项 + 解耦守卫）
|
||||||
uv run python -m referee # 起本地评测机（可选，配合 WebUI 回放）
|
uv run python -m referee # 起本地评测机（可选，配合 WebUI 回放）
|
||||||
uv run mcm26b sim-check # P0 自检：复现附件计时示例 105/111/194/199
|
uv run mcm26b sim-check # P0 自检：复现附件计时示例 105/111/194/199
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test # 问题3自检：随机全向场景闭环（不连官方）
|
uv run mcm26b run --robot-id 2026001 --self-test # 问题3自检：随机全向场景闭环（不连官方）
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test --problem 4 # 问题4自检：混合定向源场景
|
uv run mcm26b run --robot-id 2026001 --self-test --problem 4 # 问题4自检：混合定向源场景
|
||||||
# RL 策略（需先安装 rl 依赖组并训练模型；详见 docs/algo-rl.md）：
|
|
||||||
uv run python scripts/train_rl.py --problem 3 --timesteps 120000 # PPO + CUDA 训练 Q3 选点
|
|
||||||
uv run python scripts/pretrain_rl_expert.py --problem 4 --episodes 3000 # 自建模拟器专家行为克隆
|
|
||||||
uv run python scripts/train_rl.py --problem 4 --timesteps 40000 --init output/rl/q4_ppo_bc_init.zip
|
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test --strategy rl # 问题3 RL 闭环
|
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test --problem 4 --q4-strategy rl # 问题4 RL 闭环
|
|
||||||
uv run python scripts/benchmark_rl.py --seeds 5 # RL vs 手写策略配对消融
|
|
||||||
uv run python scripts/verify_rl_practice.py output/rl/official-practice/q4/20260911-rl-bc-01
|
|
||||||
uv run python scripts/benchmark_hrl.py --variants hrl_heuristic,bayes,directional \
|
|
||||||
--manifest output/hrl/manifests/dev.json --output output/hrl/p1-heuristic
|
|
||||||
uv run python scripts/train_hrl_high.py --problem 3 --stage 1 --timesteps 2048 \
|
|
||||||
--envs 1 --vec-env dummy --device mps --config-profile production \
|
|
||||||
--reward-profile time --model output/hrl/p1-smoke/q3.zip --eval-episodes 2
|
|
||||||
# 训练前端（本地回环 Web UI）：实时进程/进度/参数收敛曲线，详见 docs/training-ui.md
|
|
||||||
uv run python -m mcm26b.training_ui --ui-port 8036 # http://127.0.0.1:8036
|
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test --strategy hrl \
|
|
||||||
--hrl-model output/hrl/p1-smoke/q3.zip --hrl-device mps
|
|
||||||
# 自检默认：源数按题面 10~16 随机；实机预算 30 分钟（--real-budget-s 可改，仅自检生效）；
|
# 自检默认：源数按题面 10~16 随机；实机预算 30 分钟（--real-budget-s 可改，仅自检生效）；
|
||||||
# 找全后持续复扫未清除频道直到限时（--config 关闭 persistent_search 可回到"找全即退"）
|
# 找全后持续复扫未清除频道直到限时（--config 关闭 persistent_search 可回到"找全即退"）
|
||||||
uv run mcm26b run --robot-id <队号> --log output/run/session.jsonl # 对接模拟器正式/演练
|
uv run mcm26b run --robot-id <队号> --log output/run/session.jsonl # 对接模拟器正式/演练
|
||||||
@@ -159,7 +134,6 @@ client.exit()
|
|||||||
| P4 问题3全向闭环（49点覆盖、定位队列、清除、停止） | ✅ 官方问题3演练单案例通过（10/10全清，[验证报告](output/q3/practice/20260911-port2026/report.md)） |
|
| P4 问题3全向闭环（49点覆盖、定位队列、清除、停止） | ✅ 官方问题3演练单案例通过（10/10全清，[验证报告](output/q3/practice/20260911-port2026/report.md)） |
|
||||||
| P5 问题4定向扩展（联合约束、后备清除、预算） | ✅ 自建完成（官方演练待执行） |
|
| P5 问题4定向扩展（联合约束、后备清除、预算） | ✅ 自建完成（官方演练待执行） |
|
||||||
| P6 加速与消融 | ✅ 七点批量（`--strategy seven`，官方演练 12/12）+ 贝叶斯闭环（`--strategy bayes` 默认，自建消融 −30.0%，[算法说明](docs/algo-3-bayes.md)） |
|
| P6 加速与消融 | ✅ 七点批量（`--strategy seven`，官方演练 12/12）+ 贝叶斯闭环（`--strategy bayes` 默认，自建消融 −30.0%，[算法说明](docs/algo-3-bayes.md)） |
|
||||||
| P6b RL 选点 | ✅ PPO+CUDA 训练环境/闭环接入/官方+自建演练验证/专家 BC 初始化（Q3 `--strategy rl`，Q4 `--q4-strategy rl`，[算法说明](docs/algo-rl.md)） |
|
|
||||||
|
|
||||||
## 采集与正式测试约束
|
## 采集与正式测试约束
|
||||||
|
|
||||||
|
|||||||
@@ -194,7 +194,6 @@ AI 的适合用法是把这些模块逐一变成可测试程序、生成边界
|
|||||||
| P4 全向闭环 | 49点覆盖记录、定位队列、清除、停止、日志统计 | 10/16源、边界源和错误响应案例；官方演练全清并正常退出 | 自建闭环已完成：`planner.OmnidirectionalLoop`（10/16/0 源、近边界源、预算耗尽、故障注入测试通过）；官方演练全清待团队执行 |
|
| P4 全向闭环 | 49点覆盖记录、定位队列、清除、停止、日志统计 | 10/16源、边界源和错误响应案例；官方演练全清并正常退出 | 自建闭环已完成：`planner.OmnidirectionalLoop`（10/16/0 源、近边界源、预算耗尽、故障注入测试通过）；官方演练全清待团队执行 |
|
||||||
| P5 定向扩展 | no_signal联合约束、覆盖发现、有限清除后备与预算 | 朝外边界源不漏；真源不误删；格数与退出预算检查 | 已实现：`planner.DirectionalExtension`（49点覆盖对定向源成立、保证接收盘内 no_signal 记为观测到定向、对跖+环探梯子、`fallback_clear_cells` 有限圆盘覆盖后备清除带格数/预算双硬顶）；`tests/test_loop_q4.py` 全过；官方混合源演练待执行 |
|
| P5 定向扩展 | no_signal联合约束、覆盖发现、有限清除后备与预算 | 朝外边界源不漏；真源不误删；格数与退出预算检查 | 已实现：`planner.DirectionalExtension`（49点覆盖对定向源成立、保证接收盘内 no_signal 记为观测到定向、对跖+环探梯子、`fallback_clear_cells` 有限圆盘覆盖后备清除带格数/预算双硬顶）；`tests/test_loop_q4.py` 全过；官方混合源演练待执行 |
|
||||||
| P6 加速消融 | 共享测点、自适应覆盖、目标排序、提前清除 | 配对案例全清不退化，时间/请求数改善可复算 | 可删减；闭环基线数据已可用于消融对照 |
|
| P6 加速消融 | 共享测点、自适应覆盖、目标排序、提前清除 | 配对案例全清不退化，时间/请求数改善可复算 | 可删减；闭环基线数据已可用于消融对照 |
|
||||||
| P6b RL 选点 | 问题3/4 测量点选择的 PPO+CUDA 训练环境与闭环接入 | 自建全清；与手写策略配对消融有可复算报告 | 已完成：`scripts/train_rl.py`/`benchmark_rl.py`，模型 `output/rl/q{3,4}_ppo.zip`，说明 `docs/algo-rl.md` |
|
|
||||||
| P7 正式与论文 | 固定版本、论文表1、日志原名、上传核查 | 问题3/4各3份正式记录与程序版本对应 | 团队后续安排；本任务不执行正式测试 |
|
| P7 正式与论文 | 固定版本、论文表1、日志原名、上传核查 | 问题3/4各3份正式记录与程序版本对应 | 团队后续安排；本任务不执行正式测试 |
|
||||||
|
|
||||||
按2026年9月10日起的剩余时间，建议10日补齐P1并衔接P3，11日完成P4自建边界案例与全向演练，12日完成P5及混合源演练；未达验收就删减P6并优先修正闭环。演练不限3次，正式机会为问题3和4各3次。团队在版本稳定后自行安排正式记录，留出失败恢复和论文整理时间。附件建议9月13日15:30前完成正式测试，17:30后不能启动新的演练或正式测试；已进入上传队列的日志仍会继续上传。此处引用所给附件安排，运行前核对模拟器界面公告；这份排期不改变当前禁止使用正式测试的限制。
|
按2026年9月10日起的剩余时间，建议10日补齐P1并衔接P3，11日完成P4自建边界案例与全向演练，12日完成P5及混合源演练；未达验收就删减P6并优先修正闭环。演练不限3次，正式机会为问题3和4各3次。团队在版本稳定后自行安排正式记录，留出失败恢复和论文整理时间。附件建议9月13日15:30前完成正式测试，17:30后不能启动新的演练或正式测试；已进入上传队列的日志仍会继续上传。此处引用所给附件安排，运行前核对模拟器界面公告；这份排期不改变当前禁止使用正式测试的限制。
|
||||||
|
|||||||
-167
@@ -1,167 +0,0 @@
|
|||||||
# 问题3/4 强化学习选点（PPO + CUDA）
|
|
||||||
|
|
||||||
## 0. 摘要
|
|
||||||
|
|
||||||
**2026-09-11审核更新：**本文件下方的训练成绩与BC准确率是第一版历史记录。
|
|
||||||
复核发现训练/部署特征错位、BC激活函数不一致以及评测只检查completed等问题。
|
|
||||||
当前修复、Maskable PPO新模型和独立配对验证见
|
|
||||||
[强化学习适用性审核与改进](强化学习适用性审核与改进-20260911.md)。
|
|
||||||
新模型未替换原默认检查点；通过`--rl-model`显式加载。
|
|
||||||
|
|
||||||
本项目在保持 P0–P5 保守性与可解释证明不变的前提下，为问题3（全向源）与
|
|
||||||
问题4（定向源）增加了一个可选的**强化学习测量点选择策略**：
|
|
||||||
|
|
||||||
- 把“下一检测点去哪”建模为离散动作的局部决策问题；401维特征是压缩观测，
|
|
||||||
不能认定为完整马尔可夫状态，尤其第四问仍有未知朝向和被省略的历史约束；
|
|
||||||
- 状态由逐频道信念的几何特征 + 60 个候选检测点的几何特征组成（401 维）；
|
|
||||||
- 动作为 60 选 1（保证接收盘环候选、对称交会候选、定向遮挡环探候选）；
|
|
||||||
- 使用 Stable-Baselines3 的 PPO 训练，MLP 策略与价值网络在 **CUDA GPU**
|
|
||||||
（本机 RTX 4060 Laptop, CUDA 12.8 / torch 2.11.0+cu128）上更新；
|
|
||||||
- 训练环境基于自建评测机 `referee`（内存回环，与真实 HTTP 协议同边界）；
|
|
||||||
- 运行闭环仅替换既有选点排序器，覆盖证明、清除证书、定向后备清除全部保留。
|
|
||||||
|
|
||||||
历史结果（第一版自建配对消融，5 个种子）：Q3 用 12 万步随机初始化 PPO（与 `bayes`
|
|
||||||
基本持平，+1.18%）；Q4 用**专家行为克隆初始化 + PPO 微调**，混合源平均
|
|
||||||
虚拟时间相对 `directional` 基线降低 **7.16%**（14250.1s → 13230.2s）。
|
|
||||||
Q3 的 BC 初始化在消融中反而退化，因此保留随机初始化模型；Q4 的 BC 初始化
|
|
||||||
是当时五场比较中均值最好的版本。官方演练验证见 §5：两次 Q4 演练（3 定向混合源、10 定向全定向
|
|
||||||
场景）与一次 Q3 演练均全清，且通过离线审计。
|
|
||||||
|
|
||||||
## 1. 环境搭建（CUDA）
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
# 安装可选依赖组 rl（PyTorch CUDA 12.8 + stable-baselines3 + gymnasium）
|
|
||||||
uv sync --group rl --group dev
|
|
||||||
|
|
||||||
# 验证 CUDA
|
|
||||||
uv run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_name(0))"
|
|
||||||
```
|
|
||||||
|
|
||||||
`pyproject.toml` 中 `[tool.uv.sources] torch = { index = "pytorch-cu128" }`
|
|
||||||
与 `[[tool.uv.index]]` 指向 PyTorch 官方 CUDA 12.8 轮子；下载约 2.6 GB，
|
|
||||||
首次同步建议 `$env:UV_HTTP_TIMEOUT='300'`。默认 PyPI 的 Windows torch 轮子
|
|
||||||
为 `+cpu`，可以使用CPU训练；本轮修订模型即采用CPU，CUDA并非训练前提。
|
|
||||||
|
|
||||||
## 2. 第一版局部决策环境
|
|
||||||
|
|
||||||
训练环境：`src/mcm26b/rl/env.py::LocalizationEnv`。
|
|
||||||
|
|
||||||
| 项 | 定义 |
|
|
||||||
| --- | --- |
|
|
||||||
| 初始状态 | 自建模拟器随机布设一个源（Q3 全向；Q4 定向比例可调，训练默认 1.0），机器狗完成一次保证得到 `direction` 的首测 |
|
|
||||||
| 观测 | 401 维：21 个全局特征 + 20 维频道 one-hot + 60×6 维候选点特征（见 `features.py`） |
|
|
||||||
| 动作 | `Discrete(60)`：0=候选集 MEC 圆心，1..36=保证接收盘 3 环×12 方向，37..51=对称交会 15 点，52..59=定向遮挡环探 8 点 |
|
|
||||||
| 转移 | 按动作位置调用 `/measure`，用 `BeliefState` 保守更新候选集；Q4 保证接收盘内 `no_signal` 记为朝向遮挡并进入 `blocked` 序列；证书达标后自动 `/clear`；遮挡梯子用尽后模拟 20m 网格后备清除 |
|
|
||||||
| 奖励 | `-Δvirtual_time/100`（移动+检测+换频+清除真实耗时）、候选半径收缩加成、首次遮挡证据 +2、重复测点 -0.75、清除成功 +10、失败/截断负奖励 |
|
|
||||||
|
|
||||||
约束说明：候选点全部来自既有几何生成器（`planner.candidate_points`、
|
|
||||||
`planner.safe_disk_radius`、定向环探梯子），RL 不生成新的空间候选；它只做
|
|
||||||
排序。清除仍须通过几何证书；但“保证收到方向”只适用于全向源且实际点位于
|
|
||||||
距离保证盘内。第一版的52—59号环探点可能在盘外，第四问的定向遮挡也不受
|
|
||||||
距离盘保证。v2对第三问加入盘外动作屏蔽，第四问保留探索分支。
|
|
||||||
|
|
||||||
## 3. 训练与自建模拟器逻辑利用
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
# 1) 用自建模拟器上的手写专家生成 (观测, 动作) 并行为克隆
|
|
||||||
uv run python scripts/pretrain_rl_expert.py --problem 3 --episodes 3000
|
|
||||||
uv run python scripts/pretrain_rl_expert.py --problem 4 --episodes 3000
|
|
||||||
|
|
||||||
# 2) 从当前v2 BC初始化继续普通PPO（算法类型需显式匹配）
|
|
||||||
uv run python scripts/train_rl.py --problem 4 --algorithm ppo --timesteps 40000 \
|
|
||||||
--init output/rl/q4_ppo_bc_init.zip
|
|
||||||
|
|
||||||
# Q3 保留随机初始化长训练版本：
|
|
||||||
uv run python scripts/train_rl.py --problem 3 --timesteps 120000
|
|
||||||
```
|
|
||||||
|
|
||||||
专家逻辑与自建模拟器同源：Q3 用 worst-posterior 选点器，Q4 用对跖/环探
|
|
||||||
遮挡梯子；`pretrain_rl_expert.py` 在 `LocalizationEnv` 上记录每个状态的
|
|
||||||
专家动作，训练与 PPO policy_net/action_net 同构的 MLP，再把权重拷入 PPO
|
|
||||||
作为初始化。产物位于 `output/rl/bc/` 与 `output/rl/q{3,4}_ppo_bc_init.zip`。
|
|
||||||
|
|
||||||
实测：Q3 专家数据 5141 条、BC 验证准确率 38.9%，微调后自建消融退化
|
|
||||||
（保留 12 万步随机初始化模型）；Q4 专家数据 14181 条、BC 验证准确率
|
|
||||||
65.3%，微调后自建消融从 −3.84% 提升到 **−7.16%**。但审核确认BC与PPO的
|
|
||||||
激活函数不一致，这些数字不能证明成功迁移了专家策略，也不足以判断全向分支已最优。
|
|
||||||
|
|
||||||
输出：`output/rl/q3_ppo.zip`、`output/rl/q4_ppo.zip` 及同名 JSON 报告；
|
|
||||||
训练脚本先打印 `train_start`（含 device/gpu 字段），再执行 PPO 并在独立
|
|
||||||
种子场景上确定性评估成功率、平均步数与平均回报。Q4 4 万步微调约 236s；
|
|
||||||
Q3 12 万步约 625s（RTX 4060 Laptop，2 个 CPU 环境）。
|
|
||||||
|
|
||||||
## 4. 运行闭环接入
|
|
||||||
|
|
||||||
- Q3：`uv run mcm26b run --robot-id 2026001 --self-test --strategy rl`
|
|
||||||
`RLBayesLoop` 继承 `BayesLoop`（七点覆盖 + 网格粒子滤波 + 滚动 2-opt），
|
|
||||||
仅 `_select_point` 改由 PPO 输出候选编号。
|
|
||||||
- Q4：`uv run mcm26b run --robot-id 2026001 --self-test --problem 4 --q4-strategy rl`
|
|
||||||
`RLDirectionalLoop` 继承 `DirectionalExtension`（49 点覆盖 + 遮挡梯子 +
|
|
||||||
20m 有限圆盘后备清除），仅 `_next_probe` 改由 PPO 输出。
|
|
||||||
- 模型加载时校验观测 401 维 / 动作 60 维，失配直接拒绝运行。
|
|
||||||
- 默认运行路径不导入 torch/referee；只有显式选择 `rl` 才惰性导入，项目
|
|
||||||
运行时依赖仍保持 `requests` 一项。
|
|
||||||
|
|
||||||
## 5. 消融基准
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
uv run python scripts/benchmark_rl.py --seeds 5
|
|
||||||
```
|
|
||||||
|
|
||||||
同种子同场景配对比较，结果写入 `output/rl/benchmark.json`。以下为第一版历史结果：
|
|
||||||
|
|
||||||
| 场景 | 基线 | 平均虚拟时间（基线 / RL） | 相对差 | 全清 |
|
|
||||||
| --- | --- | --- | --- | --- |
|
|
||||||
| Q3（10 源全向 ×5 种子，scratch PPO） | `bayes` | 3088.3s / 3124.8s | +1.18% | 5/5 |
|
|
||||||
| Q4（10 源、3 定向 ×5 种子，BC+PPO） | `directional` | 14250.1s / 13230.2s | −7.16% | 5/5 |
|
|
||||||
|
|
||||||
训练局评估：Q3 单源 40 局成功率 100%、平均 3.55 步；Q4 定向单源 40 局
|
|
||||||
成功率 90%、平均 4.45 步；其余单源训练局未完成，不能由此宣称已兜底成功。随机策略对照：
|
|
||||||
Q3 平均回报 13.28 vs PPO 17.71；Q4 平均回报 8.72 vs BC+PPO 12.62。
|
|
||||||
|
|
||||||
### 5.1 官方模拟器演练验证（仅演练，未触碰正式测试）
|
|
||||||
|
|
||||||
使用目录内官方 `jammers-simulator-full` 的**演练测试**（不限次数），通过
|
|
||||||
官方 HTTP 接口运行同一闭环；官方产物按原名归档并离线审计：
|
|
||||||
|
|
||||||
| 问题 | 案例 | 场景真值（演练结束后可见） | 结果 | 虚拟时间 | 请求数 | 审计 |
|
|
||||||
| --- | --- | --- | --- | --- | --- | --- |
|
|
||||||
| Q3 | `F38Y-9Q9J-B82V-CB9Z` | 13 全向 | 13/13 全清 | 3538.8s | 124 | ✅ |
|
|
||||||
| Q4 | `X43X-VMRF-8GRZ-3SYA` | 7 全向 + 3 定向 | 10/10 全清 | 14003.1s | 619 | ✅ |
|
|
||||||
| Q4(BC) | `BXSA-TFQ3-2DJ6-DHH7` | **10 定向** | 10/10 全清 | 14562.8s | 691 | ✅ |
|
|
||||||
|
|
||||||
前两次为随机初始化模型，第三次为 BC 初始化微调模型。两个 Q4 官方案例的
|
|
||||||
源构成不同（3 定向 vs 10 定向），**虚拟时间不可直接相减比较**；可确认的
|
|
||||||
是 RL 在官方混合源与全定向源演练中都能完成全部清除，且
|
|
||||||
`all_timestamps_match_microsecond_recomputation`、官方 `.jlog` 的
|
|
||||||
SHA-256 校验全部通过。归档目录：`output/rl/official-practice/`。
|
|
||||||
|
|
||||||
### 5.2 自建 referee 演练验证
|
|
||||||
|
|
||||||
`uv run mcm26b run --local-judge` 走自建 referee 的 `q3_practice` /
|
|
||||||
`q4_practice` 模块（与官方相同的演练生命周期，真值在结束后可见）：
|
|
||||||
|
|
||||||
| 问题 | 案例 | 场景 | 结果 | 虚拟时间 | 归档 |
|
|
||||||
| --- | --- | --- | --- | --- | --- |
|
|
||||||
| Q3 | `FKUY-K8KJ-RXRL-FMBM` | 10 全向 | 10/10 全清 | 3317.6s | `output/rl/local-practice/q3/20260911-rl-01` |
|
|
||||||
| Q4 | `3J8T-L9GY-HRGL-LYVZ` | 7 全向 + 3 定向 | 10/10 全清 | 11942.8s | `output/rl/local-practice/q4/20260911-rl-01` |
|
|
||||||
| Q4(BC) | `VM2G-UVC6-4N7R-TVGW` | 7 全向 + 3 定向 | 10/10 全清 | 12520.8s | `output/rl/local-practice/q4/20260911-rl-bc-01` |
|
|
||||||
|
|
||||||
不同案例仍不可直接比较；成对可比的提升证据是 §5 的 5 种子同场景消融。
|
|
||||||
|
|
||||||
## 6. 结论与边界
|
|
||||||
|
|
||||||
以下是第一版结论；BC权重拷贝缺陷使“同构初始化”的解释不成立，性能判断
|
|
||||||
应以新审核报告的同场景验证及不确定性为准，而不能由BC分类准确率推出。
|
|
||||||
|
|
||||||
- RL 已接入问题3/4 完整闭环，并在自建 referee 演练与官方模拟器演练中
|
|
||||||
全清验证；这是方法验证，不是对正式成绩的承诺。
|
|
||||||
- 第一版Q4的BC初始化后均值变化为−3.84% → −7.16%，但初始化存在激活函数
|
|
||||||
缺陷，样本数也不足以支持“明确收益”的归因；当前判断见新审核报告。
|
|
||||||
- 训练环境使用自建场景假设（源均匀分布、逐点固定误差场、定向源朝向随机）；
|
|
||||||
官方隐藏分布未必相同。正式测试仍应默认使用经过演练验证的手写策略；
|
|
||||||
`--strategy rl` / `--q4-strategy rl` 适合消融、方法对比与后续调参。
|
|
||||||
- 官方演练仅使用“演练测试”按钮与开放接口，从未点击“正式测试”；官方
|
|
||||||
`.jlog/.result.json` 保持原样，未解密。
|
|
||||||
- 保守性不变式：RL 只改变选点排序，不改变覆盖账本、候选集更新、清除证书
|
|
||||||
与后备清除的数学证明；所有策略均以 `/exit` 与完整日志收尾。
|
|
||||||
@@ -1,117 +0,0 @@
|
|||||||
# HRL P0+P1 执行报告（本地自建评测机）
|
|
||||||
|
|
||||||
日期：2026-09-11（本机工作区）
|
|
||||||
范围：按用户确认，仅完成 P0 正确性回归 + P1 最小可训练/可评测闭环；
|
|
||||||
P2 的 BC、完整课程长训、Optuna、正式消融与正式测试表留待下一阶段。
|
|
||||||
|
|
||||||
## 1. 本轮代码交付
|
|
||||||
|
|
||||||
| 文件 | 内容 |
|
|
||||||
| --- | --- |
|
|
||||||
| `scripts/benchmark_hrl.py` | 同一 manifest 真值场景复用；`hrl`/`hrl_heuristic`/`bayes`/`directional` 配对评测；写 `results.json`、`aggregate.json/csv`、每局 `summary.json` 与 `session.jsonl` |
|
|
||||||
| `scripts/train_hrl_high.py` | 高层 MaskablePPO 训练；`--problem/--stage/--envs/--vec-env/--device/--config-profile/--reward-profile/--model/--resume/--init/--eval-manifest`；模型元数据固化 feature/action/reward 版本；墙钟硬上限 |
|
|
||||||
| `src/mcm26b/hrl/constants.py` | 新增 `ACTION_LAYOUT_VERSION=1`、`REWARD_VERSION=1` |
|
|
||||||
| `src/mcm26b/hrl/policy.py` | 高层模型严格校验元数据；低层模型保持独立基础校验 |
|
|
||||||
| `src/mcm26b/cli.py` | 新增 `--strategy hrl`、`--q4-strategy hrl`、`--hrl-model`、`--hrl-low-model`、`--hrl-device`、`--trace-low` |
|
|
||||||
| `src/mcm26b/hrl/hierarchical_env.py` | 新增 `HRLReward.time_only()`：P0.4 建议的纯时间目标，去掉逐源清除/发现/熵减塑形 |
|
|
||||||
| `project/` | 严格目录树薄兼容层，转发真实实现，不复制算法 |
|
|
||||||
| `tests/test_hrl_p1.py` | manifest 往返、真值隔离、纯时间奖励、project re-export 回归 |
|
|
||||||
|
|
||||||
## 2. 全量回归
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/python -m pytest -q --tb=short
|
|
||||||
# 275 passed in 124.57s
|
|
||||||
```
|
|
||||||
|
|
||||||
## 3. P1 smoke 训练（本机 Apple MPS）
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/python scripts/train_hrl_high.py \
|
|
||||||
--problem 3 --stage 1 --timesteps 2048 --seed 320001 \
|
|
||||||
--envs 1 --vec-env dummy --device mps \
|
|
||||||
--config-profile production --reward-profile time \
|
|
||||||
--model output/hrl/p1-smoke-time/q3.zip \
|
|
||||||
--eval-episodes 2 --rollout-steps 128 --batch-size 128 \
|
|
||||||
--n-epochs 3 --max-wall-s 800 --verbose 0
|
|
||||||
```
|
|
||||||
|
|
||||||
结果：
|
|
||||||
|
|
||||||
- `actual_timesteps=2048`，墙钟 `260.069 s`；
|
|
||||||
- 保存后确定性评估 2/2 成功，`accounted=20`，无异常；
|
|
||||||
- 模型元数据含 `feature_version=4`、`action_layout_version=1`、
|
|
||||||
`reward_version=1`、`reward_profile=time`、`model_sha256`。
|
|
||||||
|
|
||||||
模型重载已由以下两条路径验证；`--resume` 的 `--timesteps` 是增量步数
|
|
||||||
（SB3 `reset_num_timesteps=False` 语义），元数据写 `timesteps_semantics="additional"`：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/python -m mcm26b.cli run \
|
|
||||||
--robot-id 2026001 --self-test --strategy hrl \
|
|
||||||
--hrl-model output/hrl/p1-smoke-time/q3.zip --hrl-device mps \
|
|
||||||
--sources 3 --seed 1 --summary-steps 5
|
|
||||||
|
|
||||||
python scripts/benchmark_hrl.py --variants hrl \
|
|
||||||
--hrl-model output/hrl/p1-smoke-time/q3.zip --hrl-device mps \
|
|
||||||
--problem 3 --scenes 1 --sources 3 --output output/hrl/p1-bench-hrl
|
|
||||||
```
|
|
||||||
|
|
||||||
## 4. 多轮本地配对测试
|
|
||||||
|
|
||||||
### Q3：3 个独立场景 × 3 个变体
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python scripts/benchmark_hrl.py --variants hrl,hrl_heuristic,bayes \
|
|
||||||
--hrl-model output/hrl/p1-smoke-time/q3.zip --hrl-device mps \
|
|
||||||
--problem 3 --scenes 3 --sources 3 --start-seed 91000 \
|
|
||||||
--output output/hrl/p1-multiround-q3
|
|
||||||
```
|
|
||||||
|
|
||||||
| variant | episodes | valid_rate | cleared_mean | virtual_time_mean_s | virtual_time_std_s | program_run_time_mean_s |
|
|
||||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
||||||
| bayes | 3 | 1.0 | 3.0 | 2518.371 | 74.183 | 0.748 |
|
|
||||||
| hrl_heuristic | 3 | 1.0 | 3.0 | 2518.371 | 74.183 | 0.883 |
|
|
||||||
| hrl | 3 | 1.0 | 3.0 | 3731.380 | 120.536 | 1.853 |
|
|
||||||
|
|
||||||
结论：P1 模型只在 stage 1 单源课程上训练 2048 步，可直接重载并在 3 源场景保持 3/3 有效全清；
|
|
||||||
但用时明显未优于 bayes/heuristic，符合本轮不做正式性能声明的边界。
|
|
||||||
|
|
||||||
### Q4：3 个独立混合源场景 × 2 个变体
|
|
||||||
|
|
||||||
```bash
|
|
||||||
python scripts/benchmark_hrl.py --variants hrl_heuristic,directional \
|
|
||||||
--problem 4 --scenes 3 --sources 3 --directional 1 --start-seed 92000 \
|
|
||||||
--output output/hrl/p1-multiround-q4 --force
|
|
||||||
```
|
|
||||||
|
|
||||||
| variant | episodes | valid_rate | cleared_mean | virtual_time_mean_s | virtual_time_std_s | program_run_time_mean_s |
|
|
||||||
| --- | ---: | ---: | ---: | ---: | ---: | ---: |
|
|
||||||
| directional | 3 | 1.0 | 3.0 | 13442.997 | 263.902 | 1.063 |
|
|
||||||
| hrl_heuristic | 3 | 1.0 | 3.0 | 12292.198 | 344.827 | 6.284 |
|
|
||||||
|
|
||||||
说明：Q4 高层 PPO 尚未训练；本轮只验证 Q4 的 `hrl_heuristic` 与 `directional` 闭环、真值隔离与计时审计。
|
|
||||||
|
|
||||||
## 5. P0 状态
|
|
||||||
|
|
||||||
- `--resume` 重载：16 步起步模型续训 32 增量步后 `actual_timesteps=48`，
|
|
||||||
校验 `feature_version/action_layout_version/reward_version` 后保存；
|
|
||||||
|
|
||||||
- 槽位唯一性：`test_hrl.py` 覆盖 channel 17–20 与 1–4 同时 clear/locate 不丢；
|
|
||||||
- 清除事件一次且仅一次：fallback 成功写 `_step_clears` 已有回归；
|
|
||||||
- 候选快照：`test_opportunity_measure_precedes_candidate_snapshot` 已覆盖；
|
|
||||||
- 奖励对齐：新增 `HRLReward.time_only()`，训练脚本默认 `--reward-profile time`；
|
|
||||||
- 终止/安全网：现有 `HierarchicalEnv` 回合测试继续通过；
|
|
||||||
- 训练/部署配置：`--config-profile production` 使用生产配置，`coarse-pretrain`
|
|
||||||
显式映射到 fast 评分配置并写入模型元数据；
|
|
||||||
- 依赖守卫：全量测试 275 项在 dev+rl 环境通过。
|
|
||||||
|
|
||||||
## 6. 未完成 / P2 待办
|
|
||||||
|
|
||||||
- `scripts/pretrain_hrl_expert.py` 已建可 `--help` 的接口占位，专家轨迹 + BC 功能待实现；
|
|
||||||
- `scripts/ablate_hrl.py` 已建接口占位，无信息增益/无掩码/无课程/无粒子统计消融待实现；
|
|
||||||
- `scripts/tune_hrl.py` 已建接口占位，Optuna TPE 搜索待实现；
|
|
||||||
- Q3→Q4 迁移训练与配对实验；
|
|
||||||
- 完整 10–16 源课程训练、多个训练种子、冻结测试清单；
|
|
||||||
- 官方三次正式测试表：必须有真实官方案例码与导出日志后才能填写；
|
|
||||||
本地 `referee` 结果只标 `local_self_test`。
|
|
||||||
@@ -1,267 +0,0 @@
|
|||||||
# HRL 续作执行说明：交给 Sol 的工程任务书
|
|
||||||
|
|
||||||
适用日期：2026-09-11。工作目录：本仓库根目录。先检查当前代码与本轮测试结果，再按 P0 → P1 → P2 → P3 顺序执行。
|
|
||||||
|
|
||||||
目标是交付可运行、可审计的整场调度学习系统；训练收益由完整场景验证决定，不把训练脚本存在或局部奖励上升视为完成。
|
|
||||||
本说明更新了原粘贴 Prompt 与 `.zcode/plans/plan-sess_cf390556-4ef3-4ce6-b91e-782aee1718f9.md` 中不符合现有协议或证据的部分。
|
|
||||||
|
|
||||||
## 1. 现在有什么，缺什么
|
|
||||||
|
|
||||||
| 组件 | 当前状态 | Sol 的操作 |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| `protocol/`、`referee/`、`harness.py` | 已有客户端、独立评测机、内存回环与计时审计 | 复用，不另建模拟器客户端 |
|
|
||||||
| `belief/`、`particles.py`、`planner/` | 已有保守候选、网格幸存集、覆盖证明、证书清除与后备清除 | 保持证明前提；改调度时复用 |
|
|
||||||
| `rl/`、`train_rl.py`、现有 RL 检查点 | 已有单源选点 PPO/MaskablePPO 与完整对照 | 作为局部策略和对照，勿命名为端到端 PPO |
|
|
||||||
| `hrl/constants.py`、`scheduler.py`、`planner/hrl_loop.py` | 已有高层候选、特征、执行与闭环骨架；存在本轮需修复的 P0 项 | 先完成第 4 节门槛 |
|
|
||||||
| `hrl/hierarchical_env.py`、`curriculum.py` | 已有训练环境和课程；奖励、终止、训练配置仍需审核 | 在长训前修正，覆盖独立环境/部署一致性 |
|
|
||||||
| `hrl/low_level_env.py`、`executor.py`、`train_hrl_low.py` | 有八方向到达实验；训练入口尚有掩码等缺陷 | 作为可选离线实验，不能卡住主线 |
|
|
||||||
| `tests/test_hrl.py` | 有接口和启发式 episode 测试 | 增补行为验证，不能只检验公式与实现一致 |
|
|
||||||
| `train_hrl_high.py`、`pretrain_hrl_expert.py` | 尚未实现 | P1 编写最小可训练闭环，P2 再加入 BC |
|
|
||||||
| `benchmark_hrl.py`、`ablate_hrl.py`、`tune_hrl.py` | 尚未实现 | 先评测，再训练，再做消融，最后受限调参 |
|
|
||||||
| CLI 的 `hrl` 策略和 `--hrl-model` | 当前入口尚未接入 | P1 接入并检验 `--help` 与离线闭环 |
|
|
||||||
| `output/hrl/` 的高层模型、曲线、评测表 | 本次审阅未发现这些产物 | 不声称已有 HRL 模型或训练成绩 |
|
|
||||||
|
|
||||||
现有证据：修订单步 RL 的 Q3 独立验证均值比 `bayes` 低 0.69%，区间跨 0；Q4 比 `directional` 低 14.38%，比旧 PPO 低 2.26%，后一比较区间跨 0。
|
|
||||||
这些来自固定模型、本地 18 场/题的验证；新 Q4 的后备清除次数更多，边界压力性能有退化。详情见 `docs/强化学习适用性审核与改进-20260911.md`。
|
|
||||||
没有证据证明 HRL 已优于上述基线，没有证据支持“Q4 迁移节约 80% 训练成本”或“全部合法场景小于 4000 秒”。
|
|
||||||
|
|
||||||
## 2. 必须纠正的原 Prompt 假设
|
|
||||||
|
|
||||||
1. `/measure` 与 `/clear` 提交终点，移动按两次实际动作位置间的直线距离除以 5 计费；没有朝向、转弯或八方向移动协议。
|
|
||||||
2. 八方向策略仅在本地推演再强制补上终点，无法影响模拟器实际路径；不能把这段推演称为比赛收益来源。
|
|
||||||
3. `/clear` 不切换接收机频道，目标频道与当前接收机频道应分别记录；`near` 是测量结果，清除仍需独立请求。
|
|
||||||
4. 接收半径逐源未知，动作掩码只能使用观测与保守界，不能读真实半径、源位置、类型或朝向。
|
|
||||||
5. Q3 的 `no_signal` 只能按已证明的保守半径规则删格；Q4 的 `no_signal` 可能由方向遮挡造成，不能直接删空间候选。
|
|
||||||
6. Q4 联合位置/半径/朝向粒子是后续研究选项；普通二维粒子加“角度门控”并不会自动得到正确联合后验。
|
|
||||||
7. 固定地点误差固定；同地重测不是新独立样本。保留读数舍入裕量，不收紧真实的 ±1° 误差界来追求表面收敛。
|
|
||||||
8. 源数最多 16 不等于只有 16 个频道；所有按频道编号索引的动作必须覆盖 1–20。
|
|
||||||
9. 连续无信号次数不能作为全场结束条件；覆盖参数不能随意调到破坏已证明的发现覆盖。
|
|
||||||
10. “无信息增益”“无动作掩码”“无粒子滤波”必须精确定义；不能通过取消清除证书或真源保留约束制造消融。
|
|
||||||
11. 原 Prompt 的 6 个状态与实现里的状态枚举不一致；以 `ChannelStatus` 为准，额外 TRACKING/LOCATED 只能定义为派生特征。
|
|
||||||
12. 协议文档 §6 仍写持续搜索默认开启；当前 `config.py` 为 `persistent_search=False`，CLI 用 `--persistent` 显式开启。先同步文档，不恢复过时默认值。
|
|
||||||
|
|
||||||
## 3. 推荐架构与工程边界
|
|
||||||
|
|
||||||
采用“高层学习调度 + 低层局部选点 + 协议执行”的分层架构。高层学习先处理哪个频道、继续覆盖还是定位、何时使用后备清除。
|
|
||||||
低层默认调用成熟几何选点；可切换现有 `PointSelectionPolicy` 学习局部检测点，形成第二个学习层。直线移动与设备动作继续由协议客户端执行。
|
|
||||||
Q3 首版低层用 `bayes` 几何选点；Q4 先保留 `directional` 的遮挡处理，待接线一致性通过后对比旧/新局部 PPO。
|
|
||||||
|
|
||||||
```text
|
|
||||||
SimulatorClient.state + belief + particles + coverage ledger
|
|
||||||
↓ 同一份候选快照和动作掩码
|
|
||||||
高层 MaskablePPO：选择频道/覆盖站/清除任务
|
|
||||||
↓
|
|
||||||
局部选点：geometry | 已有 v2 PointSelectionPolicy
|
|
||||||
↓
|
|
||||||
证书/预算/覆盖约束 + 实际 measure/clear 宏动作
|
|
||||||
↓
|
|
||||||
accepted 请求与响应 → 时间、信念、清除与覆盖账本
|
|
||||||
```
|
|
||||||
|
|
||||||
八方向 PPO 保留为 `low_level_env.py` 的离线附录实验；生产默认无需加载。若论文要求字面双 PPO，用高层调度 PPO + 局部选点 PPO 对应真正可影响成绩的两个层次。
|
|
||||||
现有高层一个动作可以包含整站逐频道扫描或多次后备清除，因此是变长宏动作。记录每个宏动作的实际持续时间、请求数和中断原因。
|
|
||||||
先采用 `gamma=1` 与实际时间成本，避免把“每宏动作折扣一次”误当成“每秒折扣一次”；只有明确改变目标才引入按持续时间折扣。
|
|
||||||
第一版保留 120 槽位 MLP，不同时重写注意力网络或连续动作；需要多候选/频道时另升动作版本并独立比较。
|
|
||||||
|
|
||||||
建议新增的共享接口如下；这些是待实现契约，避免训练和 CLI 分别拼接策略。
|
|
||||||
|
|
||||||
```python
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class HRLPolicyConfig:
|
|
||||||
local_selector: str = "geometry" # geometry | rl
|
|
||||||
local_model_path: str | None = None
|
|
||||||
|
|
||||||
def make_hrl_loop(client, config, *, problem: int,
|
|
||||||
high_policy=None, policy_config=HRLPolicyConfig()) -> HRLLoop: ...
|
|
||||||
|
|
||||||
def validate_episode(summary: dict, events: list[dict],
|
|
||||||
truth_channels: set[int]) -> dict: ... # 仅评测模块
|
|
||||||
|
|
||||||
def run_episode(scene: dict, variant: dict, *, output_dir: Path) -> dict: ...
|
|
||||||
```
|
|
||||||
|
|
||||||
## 4. P0：修复正确性，未过门槛不得长训
|
|
||||||
|
|
||||||
以下表中的状态以本轮提交和测试报告为准；Sol 接手时先查看 diff，已完成的项只做回归，不重复实现。
|
|
||||||
|
|
||||||
| 项目 | 精确入口与问题 | 目标与验收 |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| P0.1 槽位冲突 | `constants.py`；`scheduler.build_candidates/heuristic_action`：clear 频道 17–20 与 locate 频道 1–4 重叠 | 本轮优先修复；每类按 20 频道分配，构造同时就绪冲突场景验证任务均保留 |
|
|
||||||
| P0.2 清除事件遗漏 | `scheduler._hrl_execute_fallback` 成功未写 `_step_clears` | 本轮优先修复；证书/后备/宏动作内部清除均由相同成功事件口径计数，一次且仅一次 |
|
|
||||||
| P0.3 候选过期 | `HRLLoop._coverage_phase` 在构建候选后 `_measure_at_stop` 可改变信念 | 本轮优先修复；选动作前候选/特征/掩码属于同一个状态快照 |
|
|
||||||
| P0.4 奖励对齐 | `HierarchicalEnv.step/_reward_from/HRLReward` | 改默认奖励，成功/异常/安全网/截断均计入已经发生的真实时间 |
|
|
||||||
| P0.5 回合终止 | `HierarchicalEnv.step/reset/close` | 实际完成、失败终止与采样截断分开；收尾不丢 /exit，不把零观测用于有效截断 bootstrap |
|
|
||||||
| P0.6 训练部署配置 | `curriculum.training_config` 默认网格 29.3m，部署 14.65m | 首个可比较模型使用生产几何配置；粗网格只能显式快速预训练，后接完整保真微调与评测 |
|
|
||||||
| P0.7 无进展循环 | `scheduler` 的 locate/probe/fallback 掩码及预算检查 | 去重、已清除/无效证书、后备预算不足在动作前处理；合法动作不能无限零成本自循环 |
|
|
||||||
| P0.8 测试与依赖 | `tests/test_hrl.py` 导入守卫、`policy.py/scheduler.py` 顶层 numpy | 不安装 RL 组时基础 CLI 可导入；完整 RL 环境运行测试时不得意外全部 skip |
|
|
||||||
|
|
||||||
建议槽位布局为 clear 0–19、locate 20–39、station 40–88、fallback 89–108、保留 109–119；所有槽位互斥且小于 120。
|
|
||||||
槽位含义改变必须升 `FEATURE_VERSION`，即使仍是 1248 维/120 动作，也必须拒绝旧语义检查点。
|
|
||||||
模型元数据应要求 `level/problem/algorithm/feature_version/action_layout_version/reward_version`，并记录模型 SHA、配置、依赖版本、代码版本。
|
|
||||||
缺字段不能默认为兼容；Q3 权重初始化 Q4 必须走显式训练迁移参数，部署加载仍严格检查题号。
|
|
||||||
|
|
||||||
默认奖励先用：`r = -实际本步虚拟时间增量 / 100`；证明完成时加固定完成奖，未完成的真实终止加失败罚。
|
|
||||||
失败惩罚按冻结的 episode 最大成本界设置并记录，不随意用不足以覆盖数千秒的 `-100`。
|
|
||||||
相同全清场景里逐源常数奖可能只是常数偏移，但不同终止路径、事件漏记和失败场景会改变激励；故首版去掉新发现/熵减/每源清除附加奖。
|
|
||||||
如需塑形，另做势函数差 `gamma * Phi(next_state) - Phi(state)`，真正终止态统一势函数值，并检验总和望远镜消去。
|
|
||||||
如果优化题目平均时间，用整场时间作为训练目标时明确这是所选代理；不同源数的跨场景均值与 `T/N` 均值不等价，不能暗称完全一致。
|
|
||||||
人工步数上限若仅为采样截断，返回可用的终末观测并允许 bootstrap；若定义为任务失败期限，应作为真实终止计罚。按该语义设计测试。
|
|
||||||
|
|
||||||
P0 验收至少覆盖以下行为，不只验证 reward 按代码相加：
|
|
||||||
|
|
||||||
- 20 个频道的各类槽位唯一；频道 17–20 与 1–4 同时有 clear/locate 不丢任务。
|
|
||||||
- 同一观测轨迹分别构建独立训练环境与部署闭环，比较候选、mask、特征、下一实际请求及状态。
|
|
||||||
- 固定动作序列跨成功、失败、后备与安全网路径：reward 的时间分项总和等于 accepted 日志复算时间。
|
|
||||||
- Q4 保证盘内/盘外无信号不误删真源；正读数重置遮挡序列；无效证书不能执行清除。
|
|
||||||
- 10/16 源、频道含 20、边界源与预算中断均检查真实源集合、缺席集合、正常退出、异常与计时。
|
|
||||||
- 评测端真值仅进 `validate_episode`，策略构建、候选、mask、观测与模型选择均不得读取真值。
|
|
||||||
|
|
||||||
## 5. P1：最小完整闭环，先评测再短训
|
|
||||||
|
|
||||||
先实现 `scripts/benchmark_hrl.py` 与共享审计函数，使 `HRLLoop(high_policy=None)` 可以与原 `bayes/directional` 同场景对比。
|
|
||||||
这个启发式 HRL 骨架是重要对照：若差距来自任务候选、机会测量、Q4 保底或预算语义变化，先修或如实量化，不归因给神经网络。
|
|
||||||
每次运行将 scene、误差种子、配置、summary、完整 JSONL/accepted 事件、时间复算和模型 SHA 写入新目录；目录已存在则拒绝覆盖。
|
|
||||||
单场异常也写结果；聚合存在任一无效局时 `all_valid=false` 且脚本非零退出，不从平均值表中静默删除失败场景。
|
|
||||||
|
|
||||||
随后实现 `scripts/train_hrl_high.py`，第一版提供以下参数并通过 `--help` 验证：
|
|
||||||
|
|
||||||
```text
|
|
||||||
--problem {3,4} --stage {1,2,3,4} --timesteps INT --seed INT
|
|
||||||
--envs INT --vec-env {dummy,subproc} --device {cpu,cuda,mps,auto}
|
|
||||||
--config-profile {production,coarse-pretrain}
|
|
||||||
--model PATH --resume PATH --eval-manifest PATH --verbose INT
|
|
||||||
--local-selector {geometry,rl} --local-model PATH
|
|
||||||
```
|
|
||||||
|
|
||||||
阶段 1 即保持 20 频道与固定观测维度，只改变布景复杂度；早期课程可减少源数，但完整训练必须覆盖 10/13/16 源与实际 ±1° 固定地点误差。
|
|
||||||
Q4 完整课程同时覆盖少量/半数/大量定向源及近边界向外朝向；Q3→Q4 迁移与 Q4 随机初始化作为配对实验。
|
|
||||||
禁止让策略知道课程场景真实源数或隐藏的占用频道列表；初始位置保持官方 `(0,0)`，除非任务被明确标为局部交接课程。
|
|
||||||
先测 1/2/4 个环境的采样吞吐、内存和每次决策耗时，再选择 Dummy/Subproc；小 MLP 不假定 GPU 更快。
|
|
||||||
Subproc 环境类内部提供 `action_masks()`；Windows/macOS 使用入口守卫及可序列化工厂。
|
|
||||||
随机种子覆盖 Python/NumPy/Torch、PPO、每个环境、场景误差；每个 worker 使用独立 Monitor 文件。
|
|
||||||
保存实际 `model.num_timesteps` 与请求步数，恢复时说明增量还是累计；续训加载正确算法/特征版本且不覆盖原模型。
|
|
||||||
使用掩码评估接口，如 `MaskableEvalCallback`；同时在确定性推理中传入同源 mask。
|
|
||||||
|
|
||||||
CLI 加入 `--strategy hrl`、`--q4-strategy hrl`、`--hrl-model` 与局部策略选项，经 `make_hrl_loop` 统一构建；默认策略保持现状。
|
|
||||||
显式指定缺失模型应报可操作错误，不能把启发式降级结果记成 HRL 模型成绩；启发式骨架单独命名 `hrl_heuristic`。
|
|
||||||
|
|
||||||
P1 门槛：启发式骨架全场审计通过；512–2048 步训练冒烟可保存、重载、确定性推理；CLI 内存回环至少 Q3/Q4 各一场通过。
|
|
||||||
短训先用单个种子验证流程，不用这次模型的成功率作论文结论；完成后记录吞吐再定时间预算，不预设 1–2 小时足够。
|
|
||||||
|
|
||||||
## 6. P2：完整保真训练、独立评测与消融
|
|
||||||
|
|
||||||
实现 `pretrain_hrl_expert.py`：运行 `hrl_heuristic`，保存 `(episode_id, state, candidates, mask, action)`；按 episode 划分训练/验证。
|
|
||||||
BC 与 PPO 的网络层、激活、归一化、mask 处理保持一致；复制后在真实样本上逐位比较 logits，critic 单独初始化。
|
|
||||||
记录专家动作无法映射到候选的比率与原因；若大量回退到最近候选，先改专家映射，不用假专家标签继续训练。
|
|
||||||
至少比较 BC+PPO 与随机初始化 PPO；BC 准确率单独报告，不能代替独立闭环用时与有效全清率。
|
|
||||||
|
|
||||||
冻结数据集分工：
|
|
||||||
|
|
||||||
| 集合 | 用途与约束 |
|
|
||||||
| --- | --- |
|
|
||||||
| 已有 700–702 开发场景 | 查实现错误、探索配置与性能分解 |
|
|
||||||
| 已有 91000–91005 场景 | 历史回归；结果已经被审阅，不再宣称是本次全新盲测 |
|
|
||||||
| 新开发/选择清单 | 固定 JSON manifest，记录 seed、源数、定向组成、error_seed；用于训练选择和调参 |
|
|
||||||
| 新冻结最终测试清单 | 与训练、BC、调参及选择清单不重叠；模型冻结后运行，失败后修改则新开一轮测试 |
|
|
||||||
| 压力场景 | 圆周边界、1000m 半径、向外/切向定向、高频道、固定误差与预算边界；单独汇总 |
|
|
||||||
|
|
||||||
同场景每个变体用相同位置、频道、半径、朝向和地点误差函数。每个模型至少跑 30 个完整场景，再按预算扩大；“30”是工程起点，不是统计充分性证明。
|
|
||||||
最终用于稳定收益结论的主方法至少 5 个独立训练种子；场景种子与训练种子分开报告，不能用 5 个场景替代 5 次训练。
|
|
||||||
Q3、Q4 分开统计，并按源数/定向占比分层；所有基线与候选使用相同停止、预算和计时口径。
|
|
||||||
|
|
||||||
| 指标 | 固定定义 |
|
|
||||||
| --- | --- |
|
|
||||||
| 有效全清 | 成功清除频道集合等于评测真值；缺席集合与真值不交；completed、正常 exit、无异常、计时一致 |
|
|
||||||
| `virtual_time_s` | 实际退出前的完整虚拟时间；主竞赛口径显式 `persistent_search=False` |
|
|
||||||
| `avg_locate_clear_time_s` | 每场 `T / 实际清除数`；有效全清主表再做逐场平均，另报源数分组 |
|
|
||||||
| `accounted_virtual_time_s` | 首次完成证明的时间；与最终时间并列，不悄悄用它替换正式计分 |
|
|
||||||
| 运行时间 | 每场实际墙钟、模型推理耗时分位数；本地内存回环时间不冒充官方程序运行时间 |
|
|
||||||
| 成本分解 | 移动秒、测量秒、换频秒、成功/失败 clear 秒、后备尝试、重复测量、总请求数 |
|
|
||||||
| 尾部与失败 | 中位数/P90/P95/最大值、有效全清率、失败明细；小样本尾分位注明估计局限 |
|
|
||||||
|
|
||||||
平均 `T/N` 不等于 `平均 T / 平均 N`；若另用总体总时间/总源数，明确标为按源数加权口径。
|
|
||||||
配对效应以同场景时间差为基础，报告均值相对变化、胜出场数、95% 配对 bootstrap 区间；相同场景家族共享种子时按家族重采样。
|
|
||||||
多训练种子结果报告种子间分散程度；必要时对训练种子和场景做分层重采样，不能用单一场景区间代表训练稳定性。
|
|
||||||
|
|
||||||
最小比较矩阵：`bayes/directional`、旧局部 PPO、修订局部 MaskablePPO、`hrl_heuristic`、高层 PPO+几何、后续高层 PPO+局部 PPO。
|
|
||||||
此矩阵先回答“改调度是否有用”和“增加局部学习是否有用”；既有单步 RL 不能命名为无几何的端到端基线。
|
|
||||||
若仍要端到端 PPO，必须单独定义输入、实际动作、安全执行边界并训练，标记预算；它不是 P1 完成条件。
|
|
||||||
|
|
||||||
每项消融使用相同训练步数/场景预算/种子和评测集，单次只改变一个因素：
|
|
||||||
|
|
||||||
- 调度：学习高层 vs `hrl_heuristic`，保持候选生成和局部选点一致。
|
|
||||||
- 局部策略：同一高层与执行接口下，geometry vs 现有局部 PPO；报告是冻结替换还是联合重训。
|
|
||||||
- 塑形：纯时间目标 vs 经过恒等式核验的势函数塑形；无信息增益奖励就是系数为 0。
|
|
||||||
- 课程：相同总交互步数的课程训练 vs 从完整场景开始训练；Q4 迁移另列，不预设节约百分比。
|
|
||||||
- 掩码：只取消重复/低效候选等策略性屏蔽；协议合法性、证书和预算硬守卫始终保留，标明实际取消范围。
|
|
||||||
- 粒子特征：保留几何守恒内核，先将粒子统计输入和评分消融；若真的移除粒子模块，另实现保守几何替代并重新验证。
|
|
||||||
- 八方向：只报告离线到达率/路径效率/推演耗时，以及实际协议终点相同时成绩相同的事实。
|
|
||||||
|
|
||||||
P2 推广门槛：所有最终与压力场景有效，Q4 相对最强已冻结基线的时间差区间上界小于 0；尾部容忍度先写入 manifest。
|
|
||||||
建议初始尾部门槛为 P95 不退化超过 5%，这是可调整的工程标准，不是题设要求；任何调整都在重跑最终测试前完成。
|
|
||||||
Q3 若未显示稳定收益，保留 `bayes` 默认并交付负结果。仅比手写基线快、却未优于现有 PPO，不能宣布替换默认模型。
|
|
||||||
|
|
||||||
## 7. P3：受限调参、论文与官方验证
|
|
||||||
|
|
||||||
只有 P2 主方法具备竞争力才实现并运行 `tune_hrl.py`。先搜索学习率、熵系数、网络宽度、GAE、合法候选排序参数。
|
|
||||||
不搜索真实误差上限、光学清除半径、无信号停止次数或破坏覆盖证明的网格间距；确定性网格没有粒子重采样阈值可调。
|
|
||||||
Optuna 只用开发/选择清单，失败配置先按有效性淘汰；预算与失败/剪枝逻辑写入结果。最终模型重新用固定配置多种子训练。
|
|
||||||
文档、流程图与论文表明确：学习负责哪些决策、证明由哪些模块提供、本地生成分布未知是否对应官方分布。
|
|
||||||
训练曲线、实际步数、配置、全部对照与失败明细均归档；不要只挑最好种子或去掉不利的边界场景。
|
|
||||||
官方三次测试需要对应演练已经启动、真实案例码与官方导出日志；本地 referee 的三场不能填写成“三次正式测试”。
|
|
||||||
不覆盖旧官方归档；官方结果与本地结果分表，不对不同官方案例直接计算改进百分比。
|
|
||||||
|
|
||||||
## 8. 命令：已经存在与待实现的严格区分
|
|
||||||
|
|
||||||
以下入口当前已经存在；依赖需先安装，命令失败要修复或如实报告，不能跳过错误声称通过：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv sync --group dev --group rl
|
|
||||||
uv run python -m pytest -q --tb=short
|
|
||||||
uv run python -m pytest -q tests/test_hrl.py --tb=short
|
|
||||||
uv run mcm26b run --help
|
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test --strategy bayes
|
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test --problem 4 --q4-strategy directional
|
|
||||||
uv run python scripts/train_rl.py --help
|
|
||||||
uv run python scripts/train_hrl_low.py --help
|
|
||||||
```
|
|
||||||
|
|
||||||
`train_hrl_low.py` 目前仅 `--help` 可用于检查入口，不把完整训练列为已可用：
|
|
||||||
其 `MaskablePPO.learn()` 需要环境掩码，当前 `LowLevelEnv` 没有 `action_masks()`；改普通 PPO 或提供全 True 掩码后再试。
|
|
||||||
还需修 BC 按 episode 划分、统一随机种子、每 worker 独立 Monitor 文件、评估 seed 生效、实际路径长度与成功率分开统计。
|
|
||||||
低层到达失败不能靠 executor 最后强制补终点伪装成策略到达；环境裁剪后的实际移动长度也不能简单记为 `steps*100m`。
|
|
||||||
|
|
||||||
下面是 P1/P2 实现后必须支持并实测的命令契约，当前尚不能直接运行：
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# P1：先验证不加载模型的共享调度闭环
|
|
||||||
uv run python scripts/benchmark_hrl.py --variants hrl_heuristic,bayes,directional --manifest output/hrl/manifests/dev.json --output output/hrl/p1-heuristic
|
|
||||||
# P1：完整保真冒烟；模型目录必须是新目录
|
|
||||||
uv run python scripts/train_hrl_high.py --problem 3 --stage 1 --timesteps 2048 --seed 320001 --envs 1 --vec-env dummy --device cpu --config-profile production --model output/hrl/p1-smoke/q3.zip --verbose 0
|
|
||||||
# P1：CLI 明确加载高层模型
|
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test --strategy hrl --hrl-model output/hrl/p1-smoke/q3.zip
|
|
||||||
# P2：正式阶段训练；步数先依据采样吞吐与学习曲线再调整
|
|
||||||
uv run python scripts/train_hrl_high.py --problem 4 --stage 4 --timesteps 60000 --seed 420001 --envs 2 --vec-env dummy --device cpu --config-profile production --model output/hrl/p2/q4-seed420001.zip --eval-manifest output/hrl/manifests/select.json --verbose 0
|
|
||||||
# P2：冻结模型完整对照；variants 的模型位置来自同目录的配置清单
|
|
||||||
uv run python scripts/benchmark_hrl.py --variants hrl,hrl_heuristic,legacy,revised,baseline --manifest output/hrl/manifests/final.json --output output/hrl/p2-final
|
|
||||||
```
|
|
||||||
|
|
||||||
命令使用的 manifests 也要由脚本生成、校验并归档；尚不存在的模型不能当作初始化输入。
|
|
||||||
添加 `--full` 一键课程入口时，它应展开并记录每阶段命令/配置/检查点链，失败即停，可继续，不是忽略错误的 shell 包装。
|
|
||||||
参考现有依赖配置：基础运行依赖仅 requests；RL 为可选组。固定环境版本并记录硬件，勿假定当前执行机就是旧报告中的 RTX 4060。
|
|
||||||
|
|
||||||
## 9. 可以直接粘贴给 Sol 的续作 Prompt
|
|
||||||
|
|
||||||
```text
|
|
||||||
继续当前仓库的 HRL 工作。先读 docs/hrl-sol-execution.md、最新测试结果与 git diff，保留现有修改。
|
|
||||||
已有客户端/referee/几何/PPO 都要复用；不要重写骨架，不把缺失脚本或 CLI 说成已完成。
|
|
||||||
先完成 P0 尚未关闭的奖励、终止、完整保真配置、掩码和依赖问题；本轮已修的槽位/清除事件/候选过期只回归。
|
|
||||||
采用高层 MaskablePPO 调度 + 几何/已有局部 PPO 选点 + 协议直线执行；八方向只作为可选离线实验。
|
|
||||||
按 P1 先实现 benchmark_hrl 与真值隔离审计，再实现 train_hrl_high、模型元数据和 CLI，跑可重载的短训冒烟。
|
|
||||||
P0/P1 没通过就修实现，不用加训练步数掩盖问题。通过后再开展 BC、完整保真课程、多训练种子、冻结场景配对评测。
|
|
||||||
保持 Q3 bayes/Q4 现有默认策略；完整实验胜出前不覆盖默认模型。主指标包含有效全清、T、逐场 T/N、请求分解和尾部。
|
|
||||||
每阶段交付具体文件、实际执行命令、测试结果、模型/日志路径与未完成项；不预设 Q4 80% 节约或 4000 秒保证。
|
|
||||||
官方测试只有真实演练开始并获得案例码/导出日志才记正式结果，本地实验单独标注。
|
|
||||||
```
|
|
||||||
|
|
||||||
接口参考：[MaskablePPO 官方文档](https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html)，[Gymnasium 终止与时间限制](https://gymnasium.farama.org/tutorials/gymnasium_basics/handling_time_limits/)。
|
|
||||||
Binary file not shown.
@@ -1,261 +0,0 @@
|
|||||||
# 训练前端 Windows 兼容性与「暂停」审计（结论 + 修复方案）
|
|
||||||
|
|
||||||
> 审计对象：`src/mcm26b/training_ui/`、`src/mcm26b/process_control.py`、
|
|
||||||
> `scripts/train_hrl_high.py`。
|
|
||||||
> 触发问题：「无法暂停」「频繁出现 os 无 killpg 等」。
|
|
||||||
> 结论一句话：**这两个现象同源——训练前端的停止/暂停路径是纯 POSIX 设计；
|
|
||||||
> Windows 上 `os.killpg` 根本不存在，异常又没被兜住，所以点了停止等于没点。**
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 0. 结论摘要
|
|
||||||
|
|
||||||
| # | 现象 | 根因 | 严重度 |
|
|
||||||
| --- | --- | --- | --- |
|
|
||||||
| 1 | 点「停止训练」返回 500 / 无效果，训练照跑，于是「无法暂停」 | `TrainingJob.stop()` 直接调用 `os.killpg(os.getpgid(...))`；这两个符号**只存在于 POSIX**，Windows 上抛 `AttributeError`，而旧 `except` 只接 `ProcessLookupError/PermissionError` | 阻断 |
|
|
||||||
| 2 | 控制台/日志反复出现 `module 'os' has no attribute 'killpg'` | 同上：错误最终落到 `_UIHandler.do_POST` 的兜底 `except Exception`，返回 500 并带出 `AttributeError` 文本；每点一次复现一次 | 阻断 |
|
|
||||||
| 3 | 即使把 killpg 换成 `terminate()`，停止后模型/元数据仍然丢失（Windows 上） | Windows 的 `os.kill(pid, SIGTERM)` 语义是 `TerminateProcess` **硬杀**，训练脚本里"收到信号后在 step 边界保存"的处理器不会执行 | 高 |
|
|
||||||
| 4 | 界面根本没有「暂停」能力（POSIX 也没有） | 原设计只有 start/stop；且 `SIGSTOP/SIGCONT` 在 Windows 上不存在，**原生暂停在 Windows 上无法实现**，必须改成协作式 | 高 |
|
|
||||||
| 5 | 子进程进程组隔离在 Windows 上失效 | `Popen(start_new_session=True)` 在 Windows 被**静默忽略**（CPython 的 Windows `_execute_child` 形参名就是 `unused_start_new_session`） | 中 |
|
|
||||||
| 6 | subproc worker 在 Windows 下可能抢在主进程收尾前退出 | worker 只忽略 `SIGTERM/SIGINT`，没处理 Windows 的 `SIGBREAK`（`CTRL_BREAK_EVENT` 的落点） | 中 |
|
|
||||||
|
|
||||||
**修复原则**：把"暂停/继续/停止"从**信号语义**改成**控制文件语义**。
|
|
||||||
控制文件在 POSIX 与 Windows 上都只是普通文件 IO，行为完全一致；信号与强杀退居
|
|
||||||
兜底。这样 Windows 上暂停、继续、优雅停止、落盘保证全部成立。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 1. 失败链复现（旧代码）
|
|
||||||
|
|
||||||
```
|
|
||||||
用户点「停止训练」
|
|
||||||
└─ POST /api/train/stop
|
|
||||||
└─ TrainingUI.stop() → TrainingJob.stop()
|
|
||||||
└─ os.killpg(os.getpgid(process.pid), signal.SIGTERM)
|
|
||||||
├─ POSIX ：成功，进程组收到 SIGTERM，脚本优雅收尾
|
|
||||||
└─ Windows：AttributeError: module 'os' has no attribute 'killpg'
|
|
||||||
↑ 只 except (ProcessLookupError, PermissionError)，接不住
|
|
||||||
└─ _UIHandler.do_POST 兜底 except Exception → HTTP 500
|
|
||||||
└─ 训练子进程完全没收到任何指令，继续跑 → 「无法暂停 / 停止没反应」
|
|
||||||
```
|
|
||||||
|
|
||||||
关键代码（修复前，`src/mcm26b/training_ui/server.py`）：
|
|
||||||
|
|
||||||
```python
|
|
||||||
def stop(self) -> bool:
|
|
||||||
with self._lock:
|
|
||||||
process = self.process
|
|
||||||
if process is None or process.poll() is not None:
|
|
||||||
return False
|
|
||||||
try:
|
|
||||||
os.killpg(os.getpgid(process.pid), signal.SIGTERM)
|
|
||||||
except (ProcessLookupError, PermissionError):
|
|
||||||
process.terminate()
|
|
||||||
self.status = "stopping"
|
|
||||||
return True
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 2. 根因逐条（含证据）
|
|
||||||
|
|
||||||
### 2.1 `os.killpg` / `os.getpgid` 是 POSIX-only
|
|
||||||
|
|
||||||
CPython 只在 `posix` 构建里注册这两个函数；Windows 的 `os` 模块没有它们。
|
|
||||||
访问会得到 `AttributeError`（不是 `OSError` 子类），所以旧 `except` 接不住。
|
|
||||||
|
|
||||||
### 2.2 `start_new_session=True` 在 Windows 被静默忽略
|
|
||||||
|
|
||||||
`subprocess.Popen` 接受该参数，但 Windows 的 `_execute_child` 形参名是
|
|
||||||
`unused_start_new_session`，也就是**根本不读**。要隔离进程组，Windows 必须传
|
|
||||||
`creationflags=CREATE_NEW_PROCESS_GROUP`。
|
|
||||||
|
|
||||||
### 2.3 Windows 的 `SIGTERM` 是硬杀
|
|
||||||
|
|
||||||
`os.kill(pid, sig)` 在 Windows 上只有 `CTRL_C_EVENT` / `CTRL_BREAK_EVENT` 是
|
|
||||||
"软"信号；**任何其它值**（含 `SIGTERM`）都会走 `TerminateProcess`。因此
|
|
||||||
`scripts/train_hrl_high.py` 里的：
|
|
||||||
|
|
||||||
```python
|
|
||||||
for _signum in (signal.SIGTERM, signal.SIGINT): # 旧代码
|
|
||||||
signal.signal(_signum, _on_stop_signal)
|
|
||||||
```
|
|
||||||
|
|
||||||
在 Windows 上形同虚设——处理器注册了，但没人会用它。这也解释了为什么"停止后
|
|
||||||
模型没保存"。
|
|
||||||
|
|
||||||
### 2.4 Windows 没有 `SIGSTOP`，暂停无法用信号实现
|
|
||||||
|
|
||||||
暂停（挂起/继续）在 POSIX 上可以用 `SIGSTOP/SIGCONT`，但 Windows 既无这两个
|
|
||||||
信号，也无等价的"无侵入挂起"用户态 API（真正的挂起要 `SuspendThread`/调试 API）。
|
|
||||||
所以**任何跨平台暂停都必须是协作式的**：由训练进程自己在 step 边界检查标志并
|
|
||||||
让出执行。本项目原先没有这条通道，界面也没有暂停按钮。
|
|
||||||
|
|
||||||
### 2.5 `SIGBREAK` 未处理
|
|
||||||
|
|
||||||
Windows 的 `CTRL_BREAK_EVENT`（发给进程组的软信号）在 Python 里落到
|
|
||||||
`signal.SIGBREAK`。旧的 handler 安装与 worker 屏蔽列表都只含
|
|
||||||
`SIGTERM/SIGINT`，Windows 下等于裸奔。
|
|
||||||
|
|
||||||
### 2.6 异常类型没兜住
|
|
||||||
|
|
||||||
`except (ProcessLookupError, PermissionError)` 漏掉 `AttributeError`、以及
|
|
||||||
更一般的 `OSError`；`_hard_kill` 里 `signal.SIGKILL` 在 Windows 上也不存在
|
|
||||||
（`getattr(signal, "SIGKILL", None) is None`），同样会 `AttributeError`。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 3. 修复方案（已实施）
|
|
||||||
|
|
||||||
### 3.1 新增 `src/mcm26b/process_control.py`（零第三方依赖）
|
|
||||||
|
|
||||||
| 能力 | 实现 | 作用 |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| 子进程隔离 | `spawn_kwargs()`：POSIX `start_new_session=True`；Windows `creationflags=CREATE_NEW_PROCESS_GROUP` | 前端 Ctrl+C 不误伤训练；Windows 也能组杀进程树 |
|
|
||||||
| 控制协议读写 | `write_control()` / `read_control()`：JSON + 同目录临时文件 + `os.replace` 原子替换；损坏/缺失一律视为"无指令" | 两端统一的暂停/停止通道 |
|
|
||||||
| 强杀兜底 | `terminate_process_tree(process, grace_s)`：先等宽限期；超时后 POSIX 走 `killpg(SIGKILL)`、Windows 走 `taskkill /F /T /PID`；所有 POSIX 符号用 `getattr` 探测 | 彻底移除 `os.killpg` 无条件调用；返回"是否动用了强杀" |
|
|
||||||
|
|
||||||
`IS_WINDOWS` 是运行期常量而非导入期分支，便于测试直接 monkeypatch 模拟 Windows。
|
|
||||||
|
|
||||||
### 3.2 控制文件协议
|
|
||||||
|
|
||||||
路径由前端生成并随命令下发：`--control-file <abs>`（默认落在
|
|
||||||
`output/hrl/ui/control/<job_id>.control.json`，`output/` 已 gitignore）。
|
|
||||||
|
|
||||||
```json
|
|
||||||
{"paused": false, "stop": false, "seq": 1712345678901234567, "ts": 1712345678.9}
|
|
||||||
```
|
|
||||||
|
|
||||||
- 前端**只写**：`pause()` → `paused=true`；`resume()` → `paused=false`；
|
|
||||||
`stop()` → `stop=true`（同时清零 paused，保证阻塞中的暂停循环能被唤醒）。
|
|
||||||
- 训练脚本**只读**：`_ControlPoll` 每 32 个 step 抽样读一次盘（避免每 step 一次
|
|
||||||
文件 IO）；暂停时在 `_on_step` 内阻塞轮询（`poll_s=0.2`），进程存活、显存仍占用，
|
|
||||||
但不消耗算力。
|
|
||||||
- 墙钟上限 `--max-wall-s` **扣除累计暂停时长**，避免"挂起一小时回来直接判超时"。
|
|
||||||
|
|
||||||
### 3.3 训练脚本 `scripts/train_hrl_high.py`
|
|
||||||
|
|
||||||
- 新增 `--control-file`；`_ControlPoll`（纯标准库、可单测）+ 薄薄的
|
|
||||||
`_ControlChannel(BaseCallback)` 接到 SB3 回调链**最前面**。
|
|
||||||
- 收尾信号集合扩为 `_stop_signals()`：POSIX `SIGTERM/SIGINT`；Windows 追加
|
|
||||||
`SIGBREAK`。worker 屏蔽同一集合，保证主进程 `vec_env.close()` 之前 worker 不先退。
|
|
||||||
- 落盘保证不变：停止后仍写最终模型 + 元数据，`stopped_by_signal=true`，
|
|
||||||
`stop_signal` 记 `control_file` 或信号名；异常仍落 `*_emergency.zip`。
|
|
||||||
|
|
||||||
### 3.4 训练前端 `server.py` / `static/`
|
|
||||||
|
|
||||||
- 新增 `POST /api/train/pause`、`POST /api/train/resume`；`stop` 语义改为
|
|
||||||
"写控制文件 → 宽限期内自行退出；超时强杀"，强杀在**独立看门狗线程**里做，
|
|
||||||
HTTP 请求不会被 15s 等待卡住。
|
|
||||||
- 状态机新增 `paused`；`_sync_control_events()` 用训练脚本回显的
|
|
||||||
`paused/resumed/stop_requested` 事件校正状态。
|
|
||||||
- 快照新增 `forced_kill`，前端横幅区分"优雅停止"与"超时强杀"。
|
|
||||||
- 前端新增「暂停训练 / 继续训练」按钮、暂停横幅；停止确认文案更新。
|
|
||||||
|
|
||||||
### 3.5 停止/暂停时序
|
|
||||||
|
|
||||||
```
|
|
||||||
pause: UI 写 {paused:true} ──► 脚本 _on_step 读到 ──► 回显 paused，阻塞轮询
|
|
||||||
resume: UI 写 {paused:false} ─► 暂停循环退出 ────────► 回显 resumed，继续训练
|
|
||||||
stop: UI 写 {stop:true} ────► 脚本 step 边界收尾落盘 ──► 退出码 0
|
|
||||||
│
|
|
||||||
└─(宽限 15s 未退)─► terminate_process_tree 强杀；forced_kill=true
|
|
||||||
```
|
|
||||||
|
|
||||||
### 3.6 急停（强杀）也必须可恢复
|
|
||||||
|
|
||||||
宽限期一到就强杀，训练脚本**没有任何 Python 收尾机会**——不写最终模型、不写
|
|
||||||
元数据、不写 `training_failure.json`。如果磁盘上只有 `--checkpoint-every` 的稀疏
|
|
||||||
历史 checkpoint，急停就可能丢掉上万步，或者在"保存途中被杀"时留下一个打不开的
|
|
||||||
半截 zip。为此加了四条保证：
|
|
||||||
|
|
||||||
| 保证 | 实现 | 效果 |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| 起步基线 | `recovery_every > 0` 时，`learn()` 之前先写一份 `<stem>_latest.zip` | 训练刚起就急停，也有可加载、可 `--resume` 的产物 |
|
|
||||||
| 滚动恢复点 | 每 `--recovery-every`（默认 2000）环境步覆盖写同一份 `<stem>_latest.zip` | 急停最多丢 `recovery_every` 步，而不是 `checkpoint-every` 步 |
|
|
||||||
| 原子落盘 | 模型 zip / 旁挂 json / `resume.json` 一律「同目录 `.tmp-` + `os.replace`」 | 保存途中被强杀，目标路径仍是上一份**完整**快照；不会出现半截文件 |
|
|
||||||
| 权威恢复指针 | 每次落盘刷新 `<run_dir>/resume.json`（model / timesteps / 增量语义 / 生效 PPO 超参）；前端 `/api/train/state` 暴露为 `recovery` | 无论怎么停，都有一份明确的"从哪继续"；界面一键「用最近检查点续训」 |
|
|
||||||
|
|
||||||
前端在强杀后还会补写 `<run_dir>/forced_stop.json`（`job_id`、`pid`、`reason`、
|
|
||||||
`recovery.model`），因为此时训练脚本已经无法记录任何事情。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 4. 平台兼容性矩阵（修复后）
|
|
||||||
|
|
||||||
| 能力 | POSIX | Windows |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| 启动子进程并隔离 | `start_new_session=True`（setsid） | `CREATE_NEW_PROCESS_GROUP` |
|
|
||||||
| 暂停 / 继续 | 控制文件（协作式） | 控制文件（协作式） |
|
|
||||||
| 优雅停止 + 落盘 | 控制文件；`SIGTERM` 亦可 | 控制文件；`CTRL_BREAK_EVENT`→`SIGBREAK` 亦可 |
|
|
||||||
| 超时强杀进程树 | `killpg(SIGKILL)` | `taskkill /F /T /PID` |
|
|
||||||
| `os.killpg` 依赖 | 有 | **无（不再调用）** |
|
|
||||||
| 暂停期间是否算墙钟 | 否（扣除） | 否（扣除） |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 5. 验证
|
|
||||||
|
|
||||||
- 新增/更新回归测试：
|
|
||||||
- `tests/test_training_ui.py`：控制文件协议往返与损坏容错、`spawn_kwargs`
|
|
||||||
平台分支、Windows 强杀走 `taskkill`、**无 `killpg` 环境下强杀不抛异常**
|
|
||||||
（直接复现旧 bug）、暂停/继续/停止全链路、控制文件被忽略时强杀并写
|
|
||||||
`forced_stop.json` 恢复指引、`discover_recovery` 的
|
|
||||||
`resume.json`/扫描/悬空指针三分支、HTTP `pause/resume` 端到端。
|
|
||||||
- `tests/test_hrl_checkpoint.py`：控制文件状态机单测；原子保存失败不破坏上一份
|
|
||||||
快照；JSON 原子替换；历史清理不误删滚动恢复点；起步基线可加载；
|
|
||||||
**强杀急停后 `resume.json` + 滚动恢复点仍可 `--resume`**；控制文件停止后
|
|
||||||
模型/元数据落盘（`stop_signal=control_file`）与暂停→继续；POSIX 信号路径
|
|
||||||
单列并加平台跳过。以上集成用例都用**真实训练脚本**跑 subproc。
|
|
||||||
- 手工验证：`--recovery-every 32` 短程实跑后强杀，`resume.json` 指向
|
|
||||||
`q3_ckpt_latest.zip`；以它为 `--resume` 目标再跑 32 步，元数据
|
|
||||||
`actual_timesteps` 从 64 变 96、`timesteps_semantics=additional`。
|
|
||||||
- 全量测试：`pytest tests referee/tests` 全绿。
|
|
||||||
- 全仓库 `pylint src scripts project tests referee main.py` 达到 10.00/10
|
|
||||||
（口径见 §7）。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 6. 残余风险与后续建议
|
|
||||||
|
|
||||||
1. **暂停粒度** = `poll_every` 个 env step（默认 32，约 0.1–3s 量级），不是即时；
|
|
||||||
若要更快可调小 `_ControlChannel(poll_every=...)`，代价是更频繁的文件 stat。
|
|
||||||
2. **宽限期固定 15s**（`server.STOP_GRACE_S`）。大模型保存可能更久；但因为落盘是
|
|
||||||
原子的，即使宽限期到点被强杀，也只是"新的最终模型没写成"，**上一份滚动恢复点
|
|
||||||
与历史 checkpoint 完好**，不会出现半截文件。需要更长可调大该常量。
|
|
||||||
3. **急停的损失上界 = `--recovery-every`**（默认 2000 环境步）。要更小就调小它；
|
|
||||||
调到 0 表示放弃滚动恢复点（此时只剩 `--checkpoint-every` 的稀疏历史
|
|
||||||
checkpoint 与起步基线）。
|
|
||||||
4. **Windows 强杀依赖 `taskkill`**（系统自带）。若被安全策略禁用，会退回
|
|
||||||
`Popen.kill()`（只杀主进程，spawn 的 worker 可能残留）。
|
|
||||||
5. 更严格的 Windows 方案是 **Job Object**（子进程随 job 句柄一起销毁），但需要
|
|
||||||
`ctypes`/`pywin32`，本仓库基础依赖刻意保持轻量，未引入。
|
|
||||||
6. **前端进程重启后**，内存中的任务对象没了，「用最近检查点续训」按钮不会自动
|
|
||||||
出现；但恢复信息都落在磁盘上（`<run_dir>/resume.json`、`forced_stop.json`、
|
|
||||||
`checkpoints/<stem>_latest.zip`），在表单里填同一个模型输出路径并勾选
|
|
||||||
`--resume` 即可继续。
|
|
||||||
7. 界面 `--device mps` 选项在 Windows 上无意义；`auto` 会回退 cuda/cpu，属既有
|
|
||||||
行为，未改。
|
|
||||||
8. `_ControlPoll` 的暂停循环会占用一个 Python 线程的调度（每 0.2s 一次读盘），
|
|
||||||
长暂停期间 CPU 占用可忽略。
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 7. pylint 口径说明
|
|
||||||
|
|
||||||
配置写在 `pyproject.toml` 的 `[tool.pylint.*]`，目标是
|
|
||||||
`pylint src scripts project tests referee main.py` = 10.00/10。
|
|
||||||
|
|
||||||
**改代码修掉的真实问题**（不是 disable）：未使用的 import/变量/参数、重复字典键
|
|
||||||
（`reward_version`）、`value != value` 判 NaN、`__dataclass_fields__` 私有访问、
|
|
||||||
`raise` 非异常类型、`pytest.importorskip` 之外的赋值不确定、过长行、无谓括号、
|
|
||||||
`len(x)` 判空、循环闭包变量、`subprocess.run` 未显式 `check`、`os.path` 循环导入等。
|
|
||||||
|
|
||||||
**有意 disable 的类别**（每条在 pyproject 里写了理由）：项目刻意的懒加载
|
|
||||||
（`import-outside-toplevel`）、防御式宽异常（`broad-exception-caught`）、测试访问
|
|
||||||
内部状态（`protected-access`）、pytest fixture 同名（`redefined-outer-name`）、
|
|
||||||
无头环境不实现 `gym.Env.render`（`abstract-method`）、以及静态分析的已知误报
|
|
||||||
（`possibly-used-before-assignment`）。
|
|
||||||
|
|
||||||
**注意**：`good-names` 不覆盖 `redefined-builtin`，stdlib 回调签名里的
|
|
||||||
`log_message(self, format, ...)` 只能在对应 `def` 上行内 disable。
|
|
||||||
@@ -1,160 +0,0 @@
|
|||||||
# 模型训练前端（training-ui）
|
|
||||||
|
|
||||||
本地回环 Web UI，用于**实时观察一次 HRL 高层 PPO 训练**：子进程状态、步进进度、
|
|
||||||
参数收敛曲线、逐回合记录与完整 stdout 日志。设计与评测前端（`referee/` WebUI）
|
|
||||||
同构——同一套暗色面板语言、同一个「只监听 127.0.0.1 + 静态白名单 + JSON 轮询
|
|
||||||
API」模式——差别在于本页托管的是一个**训练子进程**而不是评测会话。
|
|
||||||
|
|
||||||
## 启动
|
|
||||||
|
|
||||||
```bash
|
|
||||||
uv run python -m mcm26b.training_ui --ui-port 8036 # 打开 http://127.0.0.1:8036
|
|
||||||
uv run python scripts/train_hrl_ui.py --ui-port 8036 # 等价薄入口（支撑材料命令）
|
|
||||||
uv run python -m mcm26b.training_ui --print-schema # 只看配置契约，不起服务
|
|
||||||
uv run python -m mcm26b.training_ui --print-preview # 只看离线示例快照
|
|
||||||
```
|
|
||||||
|
|
||||||
`--no-browser` 阻止自动打开浏览器；Ctrl+C 退出时会先停止训练子进程。
|
|
||||||
|
|
||||||
无训练进程时，界面顶栏「离线预览」可渲染三组合成快照（上升 / 平台期 / KL 过猛），
|
|
||||||
用于检查曲线、指标卡、时间线与日志着色——预览不启动任何进程、不写任何文件。
|
|
||||||
|
|
||||||
## 界面结构
|
|
||||||
|
|
||||||
| 区块 | 内容 |
|
|
||||||
| --- | --- |
|
|
||||||
| 顶栏 | 任务摘要、进程/进度/墙钟/设备/连接五个状态 chip、刷新·预览·暂停/继续·停止 |
|
|
||||||
| 执行进度与进程 | 步进进度条、剩余时间与预计结束、PID/吞吐/退出码、PPO 阶段时间线、完整命令行 |
|
|
||||||
| 训练配置 | 由 `/api/train/schema` 生成的五组表单 + 四个预设，运行中锁定 |
|
|
||||||
| 参数收敛状态 | 判读 chip + 判据说明，每个指标一张卡片（最新值、增量、迷你趋势线、异常高亮） |
|
|
||||||
| 曲线 | 主指标（平均回合奖励）+ 可切换诊断量，悬停十字线读数、KL=0.05 等参考线 |
|
|
||||||
| 实时运行日志 | 关键字过滤、错误/进度 JSON 开关、自动跟随、最近 4000 行 |
|
|
||||||
| 逐回合记录 | Monitor CSV 的每回合奖励/长度/时刻 |
|
|
||||||
|
|
||||||
## 数据来源与契约
|
|
||||||
|
|
||||||
三个端点都在 `src/mcm26b/training_ui/server.py`：
|
|
||||||
|
|
||||||
| 端点 | 说明 |
|
|
||||||
| --- | --- |
|
|
||||||
| `GET /api/train/schema` | 字段/分组/预设/指标字典，**与 `TrainConfig` 同源**（`schema.py`） |
|
|
||||||
| `GET /api/train/state` | 快照；支持 `log_tail`、`metric_limit`、`after_metric` 游标与 `preview=1` |
|
|
||||||
| `POST /api/train/start` | 允许只提交部分字段（其余取默认值），校验失败返回 409 |
|
|
||||||
| `POST /api/train/pause` | 写控制文件请求在 step 边界挂起（跨平台） |
|
|
||||||
| `POST /api/train/resume` | 解除挂起，从暂停处继续 |
|
|
||||||
| `POST /api/train/stop` | 请求优雅停止；宽限期（`STOP_GRACE_S`，默认 15s）后强杀进程树 |
|
|
||||||
|
|
||||||
暂停/停止走**控制文件**而不是信号，原因与平台差异见
|
|
||||||
[`training-ui-windows-audit.md`](training-ui-windows-audit.md)：Windows 没有
|
|
||||||
`SIGSTOP`，且 `os.kill(pid, SIGTERM)` 会退化成 `TerminateProcess` 硬杀，训练脚本
|
|
||||||
"保存模型再退出"的收尾分支不会执行。控制文件在 POSIX 与 Windows 上都只是普通
|
|
||||||
文件 IO，语义完全一致；信号（POSIX `SIGTERM/SIGINT`、Windows `SIGBREAK`）只作兜底。
|
|
||||||
|
|
||||||
进度数据的两个来源，服务端都已解析：
|
|
||||||
|
|
||||||
1. **progress JSON**（主来源）：`train_hrl_high.py --progress-json` 每若干环境步打印
|
|
||||||
一行 `{"event": "progress", "timesteps": ..., "ep_rew_mean": ...}`，字段最全；
|
|
||||||
热身阶段间隔会自动缩小，短程 smoke 也能看到完整曲线。
|
|
||||||
2. **SB3 verbose 表格**（补充来源）：解析 `| key | value |` 表格，用于补
|
|
||||||
`clip_fraction`/`learning_rate` 等诊断量。表格按段分组缓存、等本表的
|
|
||||||
`total_timesteps` 到位后统一打横轴刻度；与 JSON 同一步数的表格版本会被跳过，
|
|
||||||
避免重复点。
|
|
||||||
|
|
||||||
`timesteps` 是整个前端的横轴：服务端不伪造刻度（缺刻度的表宁可丢弃，截断的表
|
|
||||||
按末尾已知刻度兜底并在收尾提交），前端不再自行编号。
|
|
||||||
|
|
||||||
## 收敛判读口径
|
|
||||||
|
|
||||||
`summarize_convergence()` 只做**可解释的启发式判读**，不替代正式学习曲线分析：
|
|
||||||
|
|
||||||
| 状态 | 触发条件 |
|
|
||||||
| --- | --- |
|
|
||||||
| 等待首个 rollout | 尚无任何指标 |
|
|
||||||
| 预热阶段 | 进度点 < 3，无法判断趋势 |
|
|
||||||
| KL 偏大，策略更新可能过猛 | 最近 `approx_kl` > 0.05（判定优先级最高） |
|
|
||||||
| 价值函数尚未拟合 | 最近 `explained_variance` < 0 |
|
|
||||||
| 奖励未形成上升趋势 | 最近窗口奖励斜率为负 |
|
|
||||||
| 已趋于平稳（疑似收敛） | 窗口对半位移 ≤ 逐点噪声、变异系数 ≤ 0.15、极差 ≤ 15% 量级 |
|
|
||||||
| 收敛趋势良好 | 其余（斜率正向） |
|
|
||||||
|
|
||||||
平台判据特意用「对半窗口位移 / 逐点噪声」而不是单看斜率绝对值：慢速但持续的
|
|
||||||
提升（斜率很小）不会被误判成收敛。所有判据数字都会写进 `notes`，便于人工复核。
|
|
||||||
|
|
||||||
## 安全边界
|
|
||||||
|
|
||||||
- 只监听 `127.0.0.1`；静态资源是固定白名单，不提供目录遍历。
|
|
||||||
- 命令由服务端以**参数列表**构造，不经过 shell；字符串参数只拒绝控制字符。
|
|
||||||
- 同一时刻只允许一个训练任务；暂停中的任务仍算「在跑」，必须先停止才能开新一局。
|
|
||||||
- 暂停/停止走控制文件（`--control-file`，默认 `output/hrl/ui/control/<job_id>.control.json`），
|
|
||||||
进程组隔离按平台选择（POSIX `setsid` / Windows `CREATE_NEW_PROCESS_GROUP`），
|
|
||||||
超时强杀按平台选择（POSIX `killpg(SIGKILL)` / Windows `taskkill /F /T`）。
|
|
||||||
- 产物路径限定在仓库根下解析，模型输出沿用训练脚本自身的 `--force`/`--resume`
|
|
||||||
语义（两者互斥，由后端校验）。
|
|
||||||
|
|
||||||
## 暂停、停止与落盘保证
|
|
||||||
|
|
||||||
「暂停」是**协作式**的：前端原子写控制文件，`train_hrl_high.py` 的
|
|
||||||
`_ControlPoll` 每 32 个环境步抽样读一次；暂停时训练阻塞在 `_on_step` 的轮询里，
|
|
||||||
进程与显存保留、不消耗算力，点「继续」从同一个 step 边界恢复。暂停时长不计入
|
|
||||||
`--max-wall-s`（否则"挂起一小时回来直接判超时"）。
|
|
||||||
|
|
||||||
「停止」同样是控制文件请求（**跨平台**）。`train_hrl_high.py` 有多层兜底，
|
|
||||||
点一下「停止」不再等于这一局白跑；而**急停**（强杀，例如 `SIGKILL` /
|
|
||||||
`taskkill /F`，进程没有任何收尾机会）也保证可恢复：
|
|
||||||
|
|
||||||
| 机制 | 行为 |
|
|
||||||
| --- | --- |
|
|
||||||
| 滚动恢复点 | `--recovery-every`（默认 2000 环境步）把 `<checkpoints>/<stem>_latest.zip` **原子覆盖写**一次；训练开始前先写一份**起步基线**。因此急停后最多只丢 `--recovery-every` 个环境步，且磁盘上始终有一份完整、可加载、可 `--resume` 的产物 |
|
|
||||||
| 原子落盘 | 模型 zip、旁挂 json、`resume.json` 都是「同目录临时文件 + `os.replace`」：保存途中被强杀不会留下半截文件，也不会破坏上一份完整快照 |
|
|
||||||
| 恢复指针 | `<run_dir>/resume.json` 记录当前最佳恢复目标 + 步数（增量语义）+ 生效过的 PPO 超参；前端顶部的「用最近检查点续训」按钮直接读它 |
|
|
||||||
| 定期 checkpoint | `--checkpoint-every`（默认 20000 环境步）另留 `<stem>_<步数>.zip`，`--checkpoint-keep` 只保留最近 K 个（默认 3），供返祖对比与多档恢复；滚动恢复点与起步基线不受清理影响 |
|
|
||||||
| 优雅停止 | 控制文件 `stop=true`（或 POSIX 信号 / Windows `SIGBREAK`）后，主进程在 step 边界收尾，保存最终模型与元数据（`stopped_by_signal=true`、`stop_signal`、`checkpoint_files`、`recovery_model`）并正常关闭环境 |
|
|
||||||
| 超时强杀 | 宽限期（`STOP_GRACE_S`，默认 15s）内没退出则强杀整棵进程树，快照 `forced_kill=true`，并在 `<run_dir>/forced_stop.json` 写明 `recovery.model`（可直接续训的目标） |
|
|
||||||
| 异常兜底 | `learn()` 抛出任何异常时先落 `*_emergency.zip` + 元数据 + `resume.json`，再写 `training_failure.json` 并抛出 |
|
|
||||||
|
|
||||||
急停后恢复的操作方式：界面点「用最近检查点续训」；命令行则
|
|
||||||
`--resume --model <resume.json 里的 model>`，`--timesteps` 按**追加**步数理解。
|
|
||||||
|
|
||||||
checkpoint 与最终模型共用同一份 json 契约，因此 `--resume <checkpoint.zip>` 直接
|
|
||||||
续训（`--timesteps` 为增量步数）。注意 `--resume`/`--init` 会沿用 checkpoint 内的
|
|
||||||
PPO 超参（`n_steps`/`batch_size`/`n_epochs`）：命令行这三项被忽略，脚本会打一条
|
|
||||||
`hyperparams_from_checkpoint` 事件说明差异，元数据 `ppo_hyperparams` 记录**实际生效**值。
|
|
||||||
回归测试见 `tests/test_hrl_checkpoint.py`，其中端到端用例真起 subproc 子进程，覆盖
|
|
||||||
控制文件停止、POSIX 信号停止、**强杀后仍可续训**与暂停→继续（信号路径在 Windows
|
|
||||||
上自动跳过）。
|
|
||||||
|
|
||||||
## 测试
|
|
||||||
|
|
||||||
```bash
|
|
||||||
.venv/bin/python -m pytest tests/test_training_ui.py tests/test_training_ui_js.py tests/test_hrl_checkpoint.py -q
|
|
||||||
```
|
|
||||||
|
|
||||||
- `tests/test_training_ui.py`：配置契约同源、预设可直接提交、参数校验、SB3 表格
|
|
||||||
刻度打点、游标分页、`describe_process` 进度/ETA、示例快照与真实快照同键；
|
|
||||||
用**桩训练脚本真起子进程**覆盖启动/轮询/暂停/继续/停止/退出码全链路；并覆盖
|
|
||||||
跨平台进程控制——控制文件往返与损坏容错、`spawn_kwargs` 平台分支、Windows 走
|
|
||||||
`taskkill`、**无 `killpg` 环境下强杀不抛异常**（旧 bug 的直接回归）、控制文件被
|
|
||||||
忽略时强杀并写 `forced_stop.json` 恢复指引、`discover_recovery` 的
|
|
||||||
`resume.json`/扫描/悬空指针三分支、HTTP `pause/resume`。
|
|
||||||
- `tests/js/training_ui_harness.cjs`（经 `tests/test_training_ui_js.py` 调起）：
|
|
||||||
DOM 桩加载发布的 `charts.js`/`app.js` 原文，用服务端生成的真实快照驱动渲染，
|
|
||||||
断言表单/指标卡/曲线/日志/逐回合表都拿到数据，覆盖增量游标三条分支与
|
|
||||||
`classList`、HTML 转义、极端数值等边界。
|
|
||||||
- `tests/test_hrl_checkpoint.py`：落盘与恢复的端到端回归——原子保存失败不破坏
|
|
||||||
上一份快照、JSON 原子替换、历史清理不误删滚动恢复点、起步基线可加载、
|
|
||||||
**强杀急停后 `resume.json` 与滚动恢复点仍可继续训练**、控制文件停止落盘、
|
|
||||||
暂停→继续、POSIX 信号停止。
|
|
||||||
|
|
||||||
## 已知取舍
|
|
||||||
|
|
||||||
- 进度点保留上限 5000（服务端 deque 与前端本地缓存同值），超长训练只保留最近一段。
|
|
||||||
- 表格解析依赖 SB3 的段名（`rollout/`/`time/`/`train/`/`system/`）；SB3 若改表格
|
|
||||||
格式，退化路径是「只显示 progress JSON」，不会导致界面崩溃。
|
|
||||||
- 收敛判读是启发式；正式结论仍需用 `scripts/benchmark_hrl.py` 的配对评测与
|
|
||||||
学习曲线人工复核。
|
|
||||||
- 暂停粒度是 `poll_every` 个环境步（默认 32），不是即时挂起；停止宽限期默认 15s。
|
|
||||||
- Windows 强杀依赖系统自带的 `taskkill`；若被策略禁用会退回 `Popen.kill()`
|
|
||||||
（只杀主进程）。更严格的 Job Object 方案需要额外依赖，未引入。
|
|
||||||
|
|
||||||
Windows 兼容性问题的完整根因与修复记录见
|
|
||||||
[`training-ui-windows-audit.md`](training-ui-windows-audit.md)。
|
|
||||||
@@ -1,165 +0,0 @@
|
|||||||
# 第三、四问强化学习适用性审核与改进
|
|
||||||
|
|
||||||
本轮审核第一版强化学习代码、专家数据、模型及官方演练归档，修复训练与部署不一致的问题，训练两个新的 Maskable PPO 模型，并用独立实现的本地 `referee` 完成同场景对照。所有新性能数字均为本地结果，没有启动新的官方演练。
|
|
||||||
|
|
||||||
结论：强化学习适合在现有几何算法内尝试“下一检测点”的排序，第四问比第三问更有收益空间。当前证据支持第四问的混合方案优于手写 `directional` 基线；尚不能证明新模型稳定优于旧 PPO。第三问没有稳定增益，应保留 `bayes` 默认策略。新模型作为实验版本显式加载，没有覆盖旧默认模型。
|
|
||||||
|
|
||||||
## 1. 题设和方法适用范围
|
|
||||||
|
|
||||||
附录2(3)的交会必须使用带误差的角域，不能把两条示向线的交点当作真值；同地重测不会提供新的独立误差样本。附录2(4)的单接收机逐频道检测，结合动作规则，须计入移动、5秒检测及实际1秒换频。第三问的全向假设允许用无信号排除接收圆盘；第四问的无信号还可能来自方向遮挡，不能沿用这一排除结论。题设及几何算法的完整审核见[算法审核与优化](算法审核与优化-20260911.md)。
|
|
||||||
|
|
||||||
当前 RL 实际只替换 `RLBayesLoop._select_point` 和 `RLDirectionalLoop._next_probe`。七站/49站覆盖、频道发现、全局任务调度、保守候选集、清除证书和第四问后备清除仍由原算法负责。因此，完整场景全清验证的是混合系统，不是神经网络独立获得了完备性证明。
|
|
||||||
|
|
||||||
401维输入概括几何、历史读数和60个候选点，但省略完整历史、候选多边形细节及位置—朝向联合约束，不能直接称为完整马尔可夫状态。局部单源训练也没有学习全场覆盖路线与多源调度。PPO可以作为这种压缩观测下的近似策略尝试，但其目标并不自动等于全场最短时间；评估必须使用独立完整场景。[SB3官方训练与评估建议](https://stable-baselines3.readthedocs.io/en/master/guide/rl_tips.html)也强调独立评估、多随机种子和状态信息完整性。
|
|
||||||
|
|
||||||
## 2. 确认的实现缺陷及修复
|
|
||||||
|
|
||||||
| 缺陷 | 证据和影响 | 已实施修复 |
|
|
||||||
| --- | --- | --- |
|
|
||||||
| 训练传入整个 `BeliefState`，部署传入 `ChannelBelief` | 同一状态的特征索引7、14、15、17、18不同；训练丢失测量次数、末次结果和方位信息 | `env._observe` 使用目标频道信念；测试逐位比较部署输入，并覆盖跨频道交接 |
|
|
||||||
| BC使用ReLU，接收权重的PPO使用Tanh | 仅复制线性权重无法保留函数。原始专家输入探针中，Q3的90.96%、Q4的67.42%出现不同argmax | BC改为Tanh；复制时检查完整前向输出，重新初始化critic；旧ReLU/旧特征检查点拒绝直接迁移 |
|
|
||||||
| BC按观测行随机拆分 | 同一episode的相关状态可能同时出现在训练和验证中，分类准确率不足以说明独立场景泛化 | 保存episode编号，按episode划分训练/验证 |
|
|
||||||
| 第四问训练遮挡状态与运行时不一致 | 训练只记录距离保证盘内的无信号，且获得新方向后未正确重置连续遮挡；后备清除触发轨迹偏离部署 | 所有连续无信号记录为探测序列，仅盘内无信号用于确认定向；正观测清空序列，统一4次遮挡/8次定位阈值 |
|
|
||||||
| 奖励鼓励首次遮挡等中间事件 | “获得遮挡证据”额外+2、半径收缩等奖励会改变实际时间目标；较高回报不一定更快 | 主要奖励改为实际虚拟耗时的负值，成功清除统一+20，失败/截断重罚；取消这些中间奖励，采用gamma=1 |
|
|
||||||
| 未屏蔽同地重测、重合候选和第三问盘外梯子点 | 浪费固定误差条件下的检测时间；旧文档“所有候选保证接收”不成立 | 训练/推理共用动作屏蔽，排除已测0.5m格位置、重复候选、越界点；第三问存在距离保证盘时排除盘外点 |
|
|
||||||
| 保证盘半径恰为0时动作数量可能改变 | 依赖去重后的候选生成器会破坏固定60动作接口 | 即使位置重合也保留编号，交由动作屏蔽处理 |
|
|
||||||
| 只校验401/60维度不能识别语义变化 | 接收机频道被目标频道替代；旧模型与新模型同形但输入含义不同 | 加入特征版本与算法元数据，v2使用实际接收机频道，旧版保留原输入语义；拒绝问题编号或屏蔽模型元数据不匹配 |
|
|
||||||
| 基准将 `completed` 当作全清 | 内部状态自洽不代表全部真实源已清除 | 用评测端真实频道集合对照成功清除集合，并检查真实频道没有被误判缺席、正常退出、异常及逐步计时；旧基准新增 `all_valid` 和失败退出码 |
|
|
||||||
| 官方审计没有严格对应问题与策略名称 | 第三问/第四问摘要可能交叉通过名称检查 | `verify_rl_practice.py` 分别要求 `rl` / `rl_q4` |
|
|
||||||
|
|
||||||
BC探针是对旧网络和“拷贝后换激活”的网络进行比较，并不是新模型错误率，也不是重新测得的BC验证准确率。旧报告38.9%/65.3%的分类准确率不能直接解释为已迁移PPO的模仿准确率。本轮训练的新模型均从随机初始化开始，未使用BC；BC修复通过前向一致性测试，未据此声称重新训练BC已经带来收益。
|
|
||||||
|
|
||||||
动作屏蔽采用官方 `MaskablePPO` 的 `action_masks` 接口，训练和确定性推理都传入相同规则。[SB3-Contrib官方文档](https://sb3-contrib.readthedocs.io/en/master/modules/ppo_mask.html)说明了这两处的用法。第四问仍允许探索盘外候选，距离保证不等于方向可见。退化到所有动作均被屏蔽时保留中心动作，由步数/预算边界处理，因此不能把本次零重复观测扩展为所有可能状态的严格保证。
|
|
||||||
|
|
||||||
正常成功episode的累计奖励为 `20 - 局部实际虚拟时间/100`；环境仍保留异常与失败处理，包括证书清除失败的额外惩罚。局部时间从首测与可选交接完成后开始，不能用训练报告中的局部均值替代整场成绩。第三问训练自动清除仍使用证书圆心，而完整 `BayesLoop` 可在证书范围内平移清除点；全局调度、预算与局部环境也有差异，完整闭环验证因此不可省略。
|
|
||||||
|
|
||||||
## 3. 训练和数据
|
|
||||||
|
|
||||||
第一版Q3为120000步随机初始化PPO；第一版Q4为BC初始化后40000步PPO。原专家数据分别为5141、14181条，原5场完整测试均值分别是Q3：3088.33/3124.76秒、Q4：14250.15/13230.20秒（手写/RL）。这些历史数字受旧环境和特征影响，不与本轮新课程的单源成功率直接比较。
|
|
||||||
|
|
||||||
修订课程保留首次有效测向，并加入35%的跨频道、跨位置交接，20%的区域边缘且最小接收半径场景。定向首测覆盖朝向±89.9°，更接近半平面边界。源位置、朝向、半径和逐点误差的抽样仍是自建分布，不是已知的官方生成分布；真值仅用于布设和评测，不进入策略观测或动作屏蔽。
|
|
||||||
|
|
||||||
两模型使用 `[128,128]` Tanh MLP、2个内存环境、`n_steps=512`、`batch_size=128`、6轮更新、学习率0.0003。实际均完成60416步，来自完整rollout批次向上取整；元数据中的60000是请求步数。安装版本为SB3 2.9.0、SB3-Contrib 2.9.0、Torch 2.11.0+cu128。本次选择CPU、单Torch线程，机器CUDA可用。
|
|
||||||
|
|
||||||
| 项目 | Q3新模型 | Q4新模型 |
|
|
||||||
| --- | ---: | ---: |
|
|
||||||
| 训练种子 | 310001 | 410001 |
|
|
||||||
| 定向概率 | 0 | 0.75 |
|
|
||||||
| 训练墙钟时间 | 276.6s | 250.6s |
|
|
||||||
| 独立单源评估成功 | 59/60 | 60/60 |
|
|
||||||
| 单源平均局部时间 | 270.04s | 349.51s |
|
|
||||||
| 平均动作步数 | 2.42 | 3.13 |
|
|
||||||
|
|
||||||
Q3有一次局部训练任务未完成，不能声称单源成功率100%。完整闭环仍有自身调度与后备路径。两问题各只有一个新模型训练种子，未覆盖训练随机性。
|
|
||||||
|
|
||||||
## 4. 完整场景同条件对照
|
|
||||||
|
|
||||||
比较三种固定策略：当前手写基线（Q3 `bayes`、Q4 `directional`）、旧默认PPO、新Maskable PPO。相同场景使用相同源位置、频道、半径、朝向及逐点固定误差种子。`referee` 为单独实现，策略不读取模拟器真值。每场保存场景、完整请求/响应、summary、模型哈希和计时核验。
|
|
||||||
|
|
||||||
开发集为种子700—702、10/16源，各问题6场。随后固定新模型，不再据验证成绩调参；验证集为种子91000—91005、10/13/16源，各问题18场。第四问定向数覆盖1、约半数、仅一个全向三种组成。另加两种圆周边界压力场景。开发、验证、压力共52个不同场景，三策略共156次完整运行，全部满足真实源全清、缺席无误判、正常exit、零异常、逐步耗时复算一致。
|
|
||||||
|
|
||||||
### 开发集平均整场时间
|
|
||||||
|
|
||||||
| 问题 | 手写基线 | 旧PPO | 新模型 |
|
|
||||||
| --- | ---: | ---: | ---: |
|
|
||||||
| Q3，6场 | 3294.62s | 3206.20s | 3264.62s |
|
|
||||||
| Q4，6场 | 14289.76s | 12443.43s | 11758.72s |
|
|
||||||
|
|
||||||
### 独立验证集
|
|
||||||
|
|
||||||
| 问题/指标 | 手写基线 | 旧PPO | 新模型 |
|
|
||||||
| --- | ---: | ---: | ---: |
|
|
||||||
| Q3有效全清 | 18/18 | 18/18 | 18/18 |
|
|
||||||
| Q3平均时间 | 3330.72s | 3254.61s | 3307.65s |
|
|
||||||
| Q3中位时间 | 3309.16s | 3262.64s | 3222.50s |
|
|
||||||
| Q3最大时间 | 3690.11s | 3786.98s | 3776.55s |
|
|
||||||
| Q4有效全清 | 18/18 | 18/18 | 18/18 |
|
|
||||||
| Q4平均时间 | 13778.82s | 12070.53s | 11797.57s |
|
|
||||||
| Q4中位时间 | 14587.80s | 12891.83s | 12488.23s |
|
|
||||||
| Q4最大时间 | 17273.98s | 15365.76s | 15894.44s |
|
|
||||||
| Q4累计未成功clear尝试 | 616 | 719 | 1380 |
|
|
||||||
| Q4累计同频道同点重测 | 12 | 28 | 0 |
|
|
||||||
|
|
||||||
Q3三组都没有失败clear或同点重测。Q4新模型减少了重测，但更多使用后备光学清除尝试，平均耗时降低的同时尾部表现有所退化。失败clear本身可来自允许的后备搜索，不能仅凭该计数认定证书错误；其移动和3秒动作耗时全部纳入结果。
|
|
||||||
|
|
||||||
对独立验证场景做10000次配对bootstrap，负值表示新模型更快：
|
|
||||||
|
|
||||||
| 对比 | 均值相对变化 | 95%场景重采样区间 | 新模型更快场数 |
|
|
||||||
| --- | ---: | ---: | ---: |
|
|
||||||
| Q3新模型对手写 | -0.69% | [-4.50%, +2.95%] | 9/18 |
|
|
||||||
| Q3新模型对旧PPO | +1.63% | [-1.64%, +5.19%] | 8/18 |
|
|
||||||
| Q4新模型对手写 | -14.38% | [-17.93%, -11.48%] | 18/18 |
|
|
||||||
| Q4新模型对旧PPO | -2.26% | [-6.08%, +1.71%] | 13/18 |
|
|
||||||
|
|
||||||
新旧PPO的区间均跨0，不能据小幅平均变化宣称稳定改进。区间只反映固定模型在这些场景上的差异，不覆盖训练种子、未知官方分布或所有合法边界。此外本轮同时修改多项训练因素，这不是能归因到单个修复的消融实验。
|
|
||||||
|
|
||||||
### 边界压力与4000秒目标
|
|
||||||
|
|
||||||
源全部放在半径1800m圆周、接收半径1000m；第四问隔一个源采用向外发射方向。10/16源各一场，各策略都完成全清。
|
|
||||||
|
|
||||||
| 问题 | 手写平均 | 旧PPO平均 | 新模型平均 |
|
|
||||||
| --- | ---: | ---: | ---: |
|
|
||||||
| Q3 | 4472.58s | 4429.04s | 4648.26s |
|
|
||||||
| Q4 | 18248.87s | 14404.01s | 15210.08s |
|
|
||||||
|
|
||||||
新Q3的两场分别为4659.06s、4637.46s，仍超过4000s；新Q4为14573.62s、15846.54s，且比旧PPO退化。此前几何优化在另一独立Q3验证组达到平均3315.89s、最大3981.74s，本轮新Q3验证组最大3776.55s；二者只能说明各自验证组达标，不能承诺全部题设场景小于4000s。第四问尚未达到4000s。
|
|
||||||
|
|
||||||
## 5. 官方证据与测试
|
|
||||||
|
|
||||||
重新审计已有官方归档，三次均通过题号/策略、官方包SHA、源数、完整清除、正常退出、请求时窗及逐步计时检查，未改写原归档：
|
|
||||||
|
|
||||||
| 问题 | 既有官方案例 | 全清 | 整场时间 |
|
|
||||||
| --- | --- | ---: | ---: |
|
|
||||||
| Q3 | F38Y-9Q9J-B82V-CB9Z | 13/13 | 3538.81s |
|
|
||||||
| Q4，3个定向源 | X43X-VMRF-8GRZ-3SYA | 10/10 | 14003.09s |
|
|
||||||
| Q4，10个定向源 | BXSA-TFQ3-2DJ6-DHH7 | 10/10 | 14562.82s |
|
|
||||||
|
|
||||||
这些是旧模型的官方可运行证据，不是新模型成绩；不同案例也不能直接计算优化百分比。新模型目前通过本地验证，未在官方新演练中运行。
|
|
||||||
|
|
||||||
全量回归：`python -m pytest -q --tb=short`，243项通过。初次沙箱运行有24个临时目录权限导致的fixture错误，获得权限后重跑全部通过。新增5项测试覆盖训练/部署观测、动作屏蔽、方向读数重置遮挡、实际成本奖励和BC完整前向一致性。之后对基准/官方审计入口的小修补分别用4次完整场景冒烟运行及三份官方归档复核通过。
|
|
||||||
|
|
||||||
## 6. 产物与复现
|
|
||||||
|
|
||||||
主目录为 `output/rl/review-20260911/`：
|
|
||||||
|
|
||||||
- `q3_masked.zip` / `.json`、`q4_masked.zip` / `.json`：新模型与训练元数据，必须成对保存。
|
|
||||||
- `evaluation/development/{controls,revised}/results.json`：开发对照。
|
|
||||||
- `evaluation/validation/final/results.json`：独立验证与每场完整日志。
|
|
||||||
- `evaluation/stress/final/results.json`：边界压力结果与日志。
|
|
||||||
- `review_evidence.json`：配对区间、BC/特征错位探针、旧官方重审结果和代码哈希。
|
|
||||||
- `benchmark-smoke.json`：旧基准有效性口径修复后的冒烟结果。
|
|
||||||
|
|
||||||
新Q3模型SHA-256：`b8b3a2bb103588348d8d7791bde9a2b0635c04778f0ac64f91c6f5c7cf87ca6e`。
|
|
||||||
新Q4模型SHA-256：`6865d541ff9bdee81886d327345f2660d5f7d72f4b39810a58ae4b867ab37e42`。
|
|
||||||
|
|
||||||
从项目根目录复现训练时，使用新目录保留本轮检查点：
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
.venv/Scripts/python.exe scripts/train_rl.py --problem 3 --algorithm maskable_ppo --timesteps 60000 --seed 310001 --device cpu --model output/rl/reproduction/q3_masked.zip --eval-episodes 60 --verbose 0
|
|
||||||
.venv/Scripts/python.exe scripts/train_rl.py --problem 4 --algorithm maskable_ppo --timesteps 60000 --seed 410001 --directional-probability 0.75 --device cpu --model output/rl/reproduction/q4_masked.zip --eval-episodes 60 --verbose 0
|
|
||||||
```
|
|
||||||
|
|
||||||
复核本轮冻结模型的完整对照（label须使用尚不存在的名称）：
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
.venv/Scripts/python.exe scripts/evaluate_rl_revision.py --split validation --label rerun-validation
|
|
||||||
.venv/Scripts/python.exe scripts/evaluate_rl_revision.py --split stress --label rerun-stress
|
|
||||||
.venv/Scripts/python.exe scripts/summarize_rl_revision.py
|
|
||||||
```
|
|
||||||
|
|
||||||
本地内存自检可显式加载对应检查点；`--self-test`不会连接官方模拟器：
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
.venv/Scripts/mcm26b.exe run --robot-id 2026001 --self-test --strategy rl --rl-model output/rl/review-20260911/q3_masked.zip
|
|
||||||
.venv/Scripts/mcm26b.exe run --robot-id 2026001 --self-test --problem 4 --q4-strategy rl --rl-model output/rl/review-20260911/q4_masked.zip
|
|
||||||
```
|
|
||||||
|
|
||||||
在用户另行启动相应问题的官方演练后，使用实际登录队号和`--base-url`连接。例如第三问（路径使用本轮尚未创建的官方演练目录）：
|
|
||||||
|
|
||||||
```powershell
|
|
||||||
.venv/Scripts/mcm26b.exe run --robot-id 2026001 --base-url http://127.0.0.1:2026 --problem 3 --strategy rl --rl-model output/rl/review-20260911/q3_masked.zip --log output/rl/review-20260911/practice-q3/session.jsonl
|
|
||||||
```
|
|
||||||
|
|
||||||
该命令保存请求/响应日志，汇总输出到终端；官方结果文件与汇总还需按原归档流程保存。第四问改用`--problem 4 --q4-strategy rl`及对应模型。本轮未执行上述官方连接命令。
|
|
||||||
|
|
||||||
进一步研究应先缩小单源训练与完整调度的差异，并处理Q4后备清除和边界退化，再做多个训练种子及新的冻结验证集。当前结果不足以将新模型自动替换为默认策略，也不足以把训练步数增加当作达到4000秒的依据。
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
# logs/
|
|
||||||
|
|
||||||
正式测试/演练导出的 JSONL 与汇总日志放置目录。本地 P1 训练/评测日志实际写入
|
|
||||||
`output/hrl/...`；本目录用于最终支撑材料归档，不修改文件名。
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
# models/
|
|
||||||
|
|
||||||
最终冻结模型归档目录。训练脚本默认把模型写到 `--model` 指定路径；
|
|
||||||
P1 smoke 产物见 `output/hrl/p1-smoke-time/`。正式提交前把冻结模型复制到本目录并保留同名 `.json` 元数据。
|
|
||||||
@@ -1,29 +0,0 @@
|
|||||||
# project/ 薄兼容层
|
|
||||||
|
|
||||||
题目 Prompt 要求 `project/` 目录树；本仓库的真实算法保持唯一一份，
|
|
||||||
位于 `src/mcm26b`、`scripts/`。本目录只做 re-export/转发，不复制实现。
|
|
||||||
|
|
||||||
| Prompt 路径 | 真实实现 |
|
|
||||||
|---|---|
|
|
||||||
| `env/simulator_client.py` | `src/mcm26b/protocol/client.py` |
|
|
||||||
| `env/local_env.py` | `src/mcm26b/hrl/hierarchical_env.py`, `low_level_env.py` |
|
|
||||||
| `env/particle_filter.py` | `src/mcm26b/particles.py` |
|
|
||||||
| `policies/high_level_ppo.py` | `src/mcm26b/hrl/policy.py` |
|
|
||||||
| `policies/low_level_ppo.py` | `src/mcm26b/hrl/policy.py`, `low_level_env.py` |
|
|
||||||
| `train/curriculum.py` | `src/mcm26b/hrl/curriculum.py` |
|
|
||||||
| `train/domain_randomization.py` | `src/mcm26b/hrl/curriculum.py` + 配置包装 |
|
|
||||||
| `eval/evaluate.py` | `scripts/benchmark_hrl.py` |
|
|
||||||
| `eval/ablation.py` | `scripts/ablate_hrl.py`（P2） |
|
|
||||||
| `baselines/heuristic.py` | `src/mcm26b/planner/hrl_loop.py` 启发式降级 |
|
|
||||||
| `baselines/end_to_end_ppo.py` | 显式未实现；单源局部 PPO 不冒充 E2E |
|
|
||||||
|
|
||||||
使用示例：
|
|
||||||
|
|
||||||
```python
|
|
||||||
from project.env.local_env import make_local_env
|
|
||||||
env = make_local_env(problem=3, stage=1, seed=7)
|
|
||||||
obs, info = env.reset(seed=7)
|
|
||||||
```
|
|
||||||
|
|
||||||
P1 当前交付：`benchmark_hrl.py`、`train_hrl_high.py`、CLI `--strategy hrl`、
|
|
||||||
`--q4-strategy hrl`。P2 的 BC/Optuna/消融在 smoke 通过后另开。
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
"""题目 Prompt 所需目录树的薄兼容层。
|
|
||||||
|
|
||||||
真实实现仍位于 ``src/mcm26b``；本包只做 re-export / CLI 转发，避免两份算法逻辑。
|
|
||||||
"""
|
|
||||||
@@ -1,3 +0,0 @@
|
|||||||
from .heuristic import HeuristicHRL, run_heuristic
|
|
||||||
|
|
||||||
__all__ = ["HeuristicHRL", "run_heuristic"]
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
"""端到端 PPO 基线占位。
|
|
||||||
|
|
||||||
按本轮确认：当前只交付局部 PPO 基线；raw-observation/raw-action 的真正
|
|
||||||
端到端 PPO 不在 P0+P1 范围内，因此显式标记未实现，避免把单源局部选点
|
|
||||||
PPO 误称为端到端 baseline。
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
|
|
||||||
def train_end_to_end_ppo(*args, **kwargs):
|
|
||||||
raise NotImplementedError(
|
|
||||||
"本轮只交付局部 PPO 基线；真正的端到端 PPO 基线标记为未实现，"
|
|
||||||
"需单独立项定义原始观测、动作与安全执行边界。")
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["train_end_to_end_ppo"]
|
|
||||||
@@ -1,16 +0,0 @@
|
|||||||
"""兼容入口：信息增益/TSP/覆盖规则 + 高层启发式调度。"""
|
|
||||||
from mcm26b.planner.bayes_loop import BayesLoop
|
|
||||||
from mcm26b.planner.hrl_loop import HRLLoop
|
|
||||||
|
|
||||||
|
|
||||||
def HeuristicHRL(*args, **kwargs): # pylint: disable=invalid-name # 兼容入口保留题面公开名
|
|
||||||
"""返回无模型的高层 HRL 闭环（heuristic_action 降级路径）。"""
|
|
||||||
kwargs.setdefault("high_policy", None)
|
|
||||||
return HRLLoop(*args, **kwargs)
|
|
||||||
|
|
||||||
|
|
||||||
def run_heuristic(*args, **kwargs):
|
|
||||||
return HeuristicHRL(*args, **kwargs).run()
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["HeuristicHRL", "run_heuristic", "BayesLoop", "HRLLoop"]
|
|
||||||
Vendored
-6
@@ -1,6 +0,0 @@
|
|||||||
from .simulator_client import SimulatorClient
|
|
||||||
from .local_env import HierarchicalEnv, LowLevelEnv, HRLReward, make_local_env
|
|
||||||
from .particle_filter import ChannelParticles
|
|
||||||
|
|
||||||
__all__ = ["SimulatorClient", "HierarchicalEnv", "LowLevelEnv", "HRLReward",
|
|
||||||
"make_local_env", "ChannelParticles"]
|
|
||||||
Vendored
-22
@@ -1,22 +0,0 @@
|
|||||||
"""兼容入口：本地简化训练环境。
|
|
||||||
|
|
||||||
HierarchicalEnv 是高层宏动作环境；LowLevelEnv 是独立八方向导航环境。
|
|
||||||
本文件不复制实现，只转发 mcm26b.hrl 下的真实类。
|
|
||||||
"""
|
|
||||||
from mcm26b.hrl.hierarchical_env import HRLReward, HierarchicalEnv
|
|
||||||
from mcm26b.hrl.low_level_env import LowLevelEnv
|
|
||||||
|
|
||||||
__all__ = ["HierarchicalEnv", "LowLevelEnv", "HRLReward", "make_local_env"]
|
|
||||||
|
|
||||||
|
|
||||||
def make_local_env(*, problem: int = 3, stage: int = 4, seed: int | None = None,
|
|
||||||
fidelity: str = "production"):
|
|
||||||
"""按题目 Prompt 的四阶段编号创建训练环境。"""
|
|
||||||
from mcm26b.hrl.curriculum import STAGES
|
|
||||||
if not 1 <= int(stage) <= len(STAGES):
|
|
||||||
raise ValueError(f"stage 必须在 1..{len(STAGES)}")
|
|
||||||
env = HierarchicalEnv(problem=problem, stage=STAGES[int(stage) - 1],
|
|
||||||
fidelity=fidelity)
|
|
||||||
if seed is not None:
|
|
||||||
env.reset(seed=seed)
|
|
||||||
return env
|
|
||||||
Vendored
-4
@@ -1,4 +0,0 @@
|
|||||||
"""兼容入口：逐频道网格粒子/幸存集。"""
|
|
||||||
from mcm26b.particles import ChannelParticles, angular_gap
|
|
||||||
|
|
||||||
__all__ = ["ChannelParticles", "angular_gap"]
|
|
||||||
Vendored
-8
@@ -1,8 +0,0 @@
|
|||||||
"""兼容入口：真实类见 mcm26b.protocol.client。"""
|
|
||||||
from mcm26b.protocol.client import (ActionRejected, ClearOutcome, DeadlineExceeded,
|
|
||||||
ExitInfo, MeasureOutcome, ProtocolError,
|
|
||||||
ProtocolViolation, SimulatorClient)
|
|
||||||
|
|
||||||
__all__ = ["SimulatorClient", "MeasureOutcome", "ClearOutcome", "ExitInfo",
|
|
||||||
"ProtocolError", "ActionRejected", "ProtocolViolation",
|
|
||||||
"DeadlineExceeded"]
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
from .evaluate import run_benchmark
|
|
||||||
from .ablation import run_ablation
|
|
||||||
|
|
||||||
__all__ = ["run_benchmark", "run_ablation"]
|
|
||||||
@@ -1,20 +0,0 @@
|
|||||||
"""兼容入口：调用 scripts/ablate_hrl.py（P2 后启用）。"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import importlib.util
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
def run_ablation(argv=None) -> int:
|
|
||||||
path = Path(__file__).resolve().parents[2] / "scripts" / "ablate_hrl.py"
|
|
||||||
if not path.exists():
|
|
||||||
raise NotImplementedError(
|
|
||||||
"scripts/ablate_hrl.py 尚未实现；P1 阶段只保证 benchmark/training/CLI。")
|
|
||||||
spec = importlib.util.spec_from_file_location("ablate_hrl_compat", path)
|
|
||||||
module = importlib.util.module_from_spec(spec)
|
|
||||||
assert spec.loader is not None
|
|
||||||
spec.loader.exec_module(module)
|
|
||||||
return int(module.main(argv))
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["run_ablation"]
|
|
||||||
@@ -1,17 +0,0 @@
|
|||||||
"""兼容入口：调用 scripts/benchmark_hrl.py。"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import importlib.util
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
def run_benchmark(argv=None) -> int:
|
|
||||||
path = Path(__file__).resolve().parents[2] / "scripts" / "benchmark_hrl.py"
|
|
||||||
spec = importlib.util.spec_from_file_location("benchmark_hrl_compat", path)
|
|
||||||
module = importlib.util.module_from_spec(spec)
|
|
||||||
assert spec.loader is not None
|
|
||||||
spec.loader.exec_module(module)
|
|
||||||
return int(module.main(argv))
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["run_benchmark"]
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
from .high_level_ppo import HighLevelPolicy, load_high_policy
|
|
||||||
from .low_level_ppo import LowLevelPolicy, load_low_policy
|
|
||||||
|
|
||||||
__all__ = ["HighLevelPolicy", "load_high_policy",
|
|
||||||
"LowLevelPolicy", "load_low_policy"]
|
|
||||||
@@ -1,10 +0,0 @@
|
|||||||
"""兼容入口：高层 MaskablePPO 推理策略。"""
|
|
||||||
from mcm26b.hrl.policy import HighLevelPolicy
|
|
||||||
|
|
||||||
|
|
||||||
def load_high_policy(model_path: str, *, device: str = "auto",
|
|
||||||
problem: int | None = None) -> HighLevelPolicy:
|
|
||||||
return HighLevelPolicy(model_path, device=device, problem=problem)
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["HighLevelPolicy", "load_high_policy"]
|
|
||||||
@@ -1,9 +0,0 @@
|
|||||||
"""兼容入口：低层八方向到达策略（离线导航实验）。"""
|
|
||||||
from mcm26b.hrl.policy import LowLevelPolicy
|
|
||||||
|
|
||||||
|
|
||||||
def load_low_policy(model_path: str, *, device: str = "auto") -> LowLevelPolicy:
|
|
||||||
return LowLevelPolicy(model_path, device=device)
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["LowLevelPolicy", "load_low_policy"]
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
from .curriculum import STAGES, Stage, make_scene, training_config
|
|
||||||
from .domain_randomization import randomize_problem_config
|
|
||||||
|
|
||||||
__all__ = ["STAGES", "Stage", "make_scene", "training_config",
|
|
||||||
"randomize_problem_config"]
|
|
||||||
@@ -1,6 +0,0 @@
|
|||||||
"""兼容入口：课程学习与域随机化。"""
|
|
||||||
from mcm26b.hrl.curriculum import (STAGES, Stage, make_scene, stage_with_noise,
|
|
||||||
training_config, training_spec)
|
|
||||||
|
|
||||||
__all__ = ["STAGES", "Stage", "make_scene", "stage_with_noise",
|
|
||||||
"training_config", "training_spec"]
|
|
||||||
@@ -1,31 +0,0 @@
|
|||||||
"""兼容入口：本地域随机化参数包装。
|
|
||||||
|
|
||||||
当前真实随机化在 mcm26b.hrl.curriculum.make_scene 内完成；这里只暴露
|
|
||||||
ProblemConfig 的显式替换助手，便于 Optuna/消融在实验配置中记录随机化范围。
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import dataclasses
|
|
||||||
from random import Random
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
|
|
||||||
|
|
||||||
def randomize_problem_config(base: ProblemConfig, *, seed: int,
|
|
||||||
bearing_error_half_deg: float | None = None,
|
|
||||||
r_eff_range: tuple[float, float] | None = None
|
|
||||||
) -> ProblemConfig:
|
|
||||||
"""返回带域随机化标量的新配置；不修改全局默认配置。"""
|
|
||||||
_rng = Random(seed) # 固定种子留作兼容：真实随机化在 make_scene 内完成
|
|
||||||
field = base.field
|
|
||||||
if bearing_error_half_deg is None:
|
|
||||||
bearing_error_half_deg = field.bearing_error_half_deg
|
|
||||||
if r_eff_range is not None:
|
|
||||||
lo, hi = r_eff_range
|
|
||||||
if not 0 <= lo <= hi:
|
|
||||||
raise ValueError("r_eff_range 必须为递增非负区间")
|
|
||||||
return dataclasses.replace(base, field=dataclasses.replace(
|
|
||||||
field, bearing_error_half_deg=float(bearing_error_half_deg)))
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["randomize_problem_config"]
|
|
||||||
@@ -1,20 +0,0 @@
|
|||||||
"""兼容入口：调用 scripts/tune_hrl.py（P2 后启用）。"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import importlib.util
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
def run_tuning(argv=None) -> int:
|
|
||||||
path = Path(__file__).resolve().parents[2] / "scripts" / "tune_hrl.py"
|
|
||||||
if not path.exists():
|
|
||||||
raise NotImplementedError(
|
|
||||||
"scripts/tune_hrl.py 尚未实现；P1 阶段先冻结超参并留待 P2。")
|
|
||||||
spec = importlib.util.spec_from_file_location("tune_hrl_compat", path)
|
|
||||||
module = importlib.util.module_from_spec(spec)
|
|
||||||
assert spec.loader is not None
|
|
||||||
spec.loader.exec_module(module)
|
|
||||||
return int(module.main(argv))
|
|
||||||
|
|
||||||
|
|
||||||
__all__ = ["run_tuning"]
|
|
||||||
+1
-18
@@ -15,30 +15,13 @@ dev = [
|
|||||||
# 独立评测机（自建模拟器）：仅测试侧依赖，运行时代码零导入（懒加载 + 守卫测试）
|
# 独立评测机（自建模拟器）：仅测试侧依赖，运行时代码零导入（懒加载 + 守卫测试）
|
||||||
"mcm26b-referee",
|
"mcm26b-referee",
|
||||||
]
|
]
|
||||||
rl = [
|
|
||||||
"gymnasium>=1.1",
|
|
||||||
"sb3-contrib==2.9.0",
|
|
||||||
"stable-baselines3>=2.6",
|
|
||||||
"torch==2.11.0",
|
|
||||||
]
|
|
||||||
|
|
||||||
# uv 工作区：referee/ 是独立包（零第三方依赖、不依赖 mcm26b），双方仅通过 HTTP 协议耦合
|
# uv 工作区：referee/ 是独立包（零第三方依赖、不依赖 mcm26b），双方仅通过 HTTP 协议耦合
|
||||||
[tool.uv.workspace]
|
[tool.uv.workspace]
|
||||||
members = ["referee"]
|
members = ["referee"]
|
||||||
|
|
||||||
# RL 训练使用 CUDA 12.8 版 PyTorch（本机 RTX 4060；运行时依赖保持轻量）。
|
|
||||||
[[tool.uv.index]]
|
|
||||||
name = "pytorch-cu128"
|
|
||||||
url = "https://download.pytorch.org/whl/cu128"
|
|
||||||
explicit = true
|
|
||||||
|
|
||||||
[tool.uv.sources]
|
[tool.uv.sources]
|
||||||
mcm26b-referee = { workspace = true }
|
mcm26b-referee = { workspace = true }
|
||||||
# CUDA 12.8 版 PyTorch 仅限 Windows/Linux（RTX 4060 训练机）；
|
|
||||||
# macOS (Apple Silicon, MPS) 回落到 PyPI 官方轮子。
|
|
||||||
torch = [
|
|
||||||
{ index = "pytorch-cu128", marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[build-system]
|
[build-system]
|
||||||
requires = ["hatchling"]
|
requires = ["hatchling"]
|
||||||
@@ -102,7 +85,7 @@ disable = [
|
|||||||
"missing-module-docstring", # 内部小模块与 __main__ 不强制
|
"missing-module-docstring", # 内部小模块与 __main__ 不强制
|
||||||
"missing-class-docstring",
|
"missing-class-docstring",
|
||||||
"missing-function-docstring",
|
"missing-function-docstring",
|
||||||
"import-outside-toplevel", # torch/SB3/referee 等重依赖与可选依赖懒加载
|
"import-outside-toplevel", # referee 等重依赖与可选依赖懒加载
|
||||||
"wrong-import-position", # 同上：importorskip 守卫后的延迟导入
|
"wrong-import-position", # 同上：importorskip 守卫后的延迟导入
|
||||||
"ungrouped-imports", # 同上：延迟导入无法与顶部导入分组
|
"ungrouped-imports", # 同上：延迟导入无法与顶部导入分组
|
||||||
"broad-exception-caught", # 子进程 / IO / HTTP 边界防御式兜底，逐处注释
|
"broad-exception-caught", # 子进程 / IO / HTTP 边界防御式兜底，逐处注释
|
||||||
|
|||||||
@@ -1,25 +0,0 @@
|
|||||||
"""HRL 消融入口（P2 待实现）。
|
|
||||||
|
|
||||||
预定义消融维度：时间目标 vs prompt 塑形、信息增益奖励系数、动作掩码取消范围、
|
|
||||||
课程学习、粒子统计特征。硬性协议守卫/证书/预算永远保留，不在消融中取消。
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("--variants", default="time,prompt,no_info,no_mask,no_curriculum")
|
|
||||||
parser.add_argument("--manifest", default="output/hrl/manifests/final.json")
|
|
||||||
parser.add_argument("--output", default="output/hrl/p2-ablation")
|
|
||||||
parser.add_argument("--timesteps", type=int, default=2048)
|
|
||||||
parser.add_argument("--seeds", type=int, default=5)
|
|
||||||
parser.parse_args(argv)
|
|
||||||
raise NotImplementedError(
|
|
||||||
"P2 尚未实现：必须先完成专家 BC 与冻结场景对照，"
|
|
||||||
"再对相同训练步数/场景/种子逐项消融；P1 只保证 benchmark/training/CLI。")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,428 +0,0 @@
|
|||||||
"""HRL/基线统一评测入口（P1）。
|
|
||||||
|
|
||||||
特点：
|
|
||||||
- 同一 manifest 中的真值场景被所有 variant 复用，保证配对比较；
|
|
||||||
- 真值仅在测试端用于构造 ``LocalSimulator`` 与最终核对，策略端只收到协议响应；
|
|
||||||
- 支持 ``hrl_heuristic``（高层启发式降级）、``bayes``、``directional``；
|
|
||||||
- 支持已训练高层模型 ``hrl``（需 ``--hrl-model`` 或 manifest 中的 variant 配置）。
|
|
||||||
|
|
||||||
输出目录（默认拒绝覆盖已有结果）：
|
|
||||||
<output>/manifest.json
|
|
||||||
<output>/runs/<scene_id>/<variant>/summary.json
|
|
||||||
<output>/runs/<scene_id>/<variant>/session.jsonl
|
|
||||||
<output>/results.json
|
|
||||||
<output>/aggregate.json
|
|
||||||
<output>/aggregate.csv
|
|
||||||
|
|
||||||
示例：
|
|
||||||
python scripts/benchmark_hrl.py \
|
|
||||||
--variants hrl_heuristic,bayes,directional \
|
|
||||||
--manifest output/hrl/manifests/dev.json \
|
|
||||||
--output output/hrl/p1-heuristic
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import csv
|
|
||||||
import json
|
|
||||||
import statistics
|
|
||||||
import sys
|
|
||||||
import time
|
|
||||||
from collections import defaultdict
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.harness import LocalSimTransport, mixed_scene, random_omni_scene
|
|
||||||
from mcm26b.harness.scenes import referee_spec
|
|
||||||
from mcm26b.protocol import SimulatorClient
|
|
||||||
from mcm26b.protocol.log import SessionLog
|
|
||||||
|
|
||||||
try: # 测试端依赖；运行时代码不导入
|
|
||||||
from referee import LocalSimulator, Source
|
|
||||||
except ImportError as exc: # pragma: no cover
|
|
||||||
raise SystemExit("benchmark_hrl 需要 dev 依赖 referee。请先 `uv sync`。") from exc
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# manifest / scene
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class SceneSpec:
|
|
||||||
scene_id: str
|
|
||||||
seed: int
|
|
||||||
error_seed: int
|
|
||||||
problem: int
|
|
||||||
sources: list[dict[str, Any]]
|
|
||||||
|
|
||||||
@property
|
|
||||||
def truth_channels(self) -> tuple[int, ...]:
|
|
||||||
return tuple(sorted(int(src["channel"]) for src in self.sources))
|
|
||||||
|
|
||||||
|
|
||||||
def _source_dict(source: Source) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"channel": int(source.channel),
|
|
||||||
"x": float(source.x),
|
|
||||||
"y": float(source.y),
|
|
||||||
"r_eff_m": float(source.r_eff_m),
|
|
||||||
"directional": bool(source.directional),
|
|
||||||
"heading_deg": float(source.heading_deg),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _sources_to_dicts(sources: list[Source]) -> list[dict[str, Any]]:
|
|
||||||
return [_source_dict(source) for source in sources]
|
|
||||||
|
|
||||||
|
|
||||||
def _source_from_dict(data: dict[str, Any]) -> Source:
|
|
||||||
return Source(
|
|
||||||
channel=int(data["channel"]),
|
|
||||||
x=float(data["x"]),
|
|
||||||
y=float(data["y"]),
|
|
||||||
r_eff_m=float(data["r_eff_m"]),
|
|
||||||
directional=bool(data.get("directional", False)),
|
|
||||||
heading_deg=float(data.get("heading_deg", 0.0)),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def make_default_manifest(*, problem: int, scenes: int, sources: int | None,
|
|
||||||
directional: int, start_seed: int,
|
|
||||||
config: ProblemConfig) -> dict[str, Any]:
|
|
||||||
"""按 CLI 参数生成可复现的固定场景清单。"""
|
|
||||||
entries: list[dict[str, Any]] = []
|
|
||||||
for index in range(scenes):
|
|
||||||
seed = start_seed + index
|
|
||||||
if problem == 4:
|
|
||||||
scene_sources = mixed_scene(seed=seed, count=sources,
|
|
||||||
directional=directional, config=config)
|
|
||||||
else:
|
|
||||||
scene_sources = random_omni_scene(seed=seed, count=sources,
|
|
||||||
config=config)
|
|
||||||
entries.append({
|
|
||||||
"scene_id": f"scene-{seed}",
|
|
||||||
"seed": seed,
|
|
||||||
"error_seed": seed + 10_000,
|
|
||||||
"problem": problem,
|
|
||||||
"sources": _sources_to_dicts(scene_sources),
|
|
||||||
})
|
|
||||||
return {"version": 1, "problem": problem, "scenes": entries}
|
|
||||||
|
|
||||||
|
|
||||||
def load_or_generate_manifest(path: Path, *, problem: int, scenes: int,
|
|
||||||
sources: int | None, directional: int,
|
|
||||||
start_seed: int,
|
|
||||||
config: ProblemConfig) -> dict[str, Any]:
|
|
||||||
if path.exists():
|
|
||||||
manifest = json.loads(path.read_text(encoding="utf-8"))
|
|
||||||
else:
|
|
||||||
manifest = make_default_manifest(
|
|
||||||
problem=problem, scenes=scenes, sources=sources,
|
|
||||||
directional=directional, start_seed=start_seed, config=config)
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
path.write_text(json.dumps(manifest, ensure_ascii=False, indent=2),
|
|
||||||
encoding="utf-8")
|
|
||||||
if "scenes" not in manifest or not manifest["scenes"]:
|
|
||||||
raise ValueError(f"manifest 无 scenes: {path}")
|
|
||||||
for raw in manifest["scenes"]:
|
|
||||||
if not {"scene_id", "error_seed", "problem", "sources"} <= set(raw):
|
|
||||||
raise ValueError(f"manifest scene 字段缺失: {raw!r}")
|
|
||||||
if not raw["sources"]:
|
|
||||||
raise ValueError(f"scene {raw.get('scene_id')} 没有来源真值")
|
|
||||||
return manifest
|
|
||||||
|
|
||||||
|
|
||||||
def scenes_from_manifest(manifest: dict[str, Any]) -> list[SceneSpec]:
|
|
||||||
result: list[SceneSpec] = []
|
|
||||||
for raw in manifest["scenes"]:
|
|
||||||
result.append(SceneSpec(
|
|
||||||
scene_id=str(raw["scene_id"]),
|
|
||||||
seed=int(raw.get("seed", 0)),
|
|
||||||
error_seed=int(raw["error_seed"]),
|
|
||||||
problem=int(raw["problem"]),
|
|
||||||
sources=[dict(item) for item in raw["sources"]],
|
|
||||||
))
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# variants
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
def _build_loop(variant: str, client: SimulatorClient, config: ProblemConfig,
|
|
||||||
problem: int, *, hrl_model: str | None,
|
|
||||||
hrl_device: str = "auto"):
|
|
||||||
"""延迟导入变体实现，避免 benchmark 顶层拉起重依赖。"""
|
|
||||||
if variant == "bayes":
|
|
||||||
from mcm26b.planner.bayes_loop import BayesLoop
|
|
||||||
return BayesLoop(client, config)
|
|
||||||
if variant == "directional":
|
|
||||||
from mcm26b.planner import DirectionalExtension
|
|
||||||
return DirectionalExtension(client, config)
|
|
||||||
if variant == "hrl_heuristic":
|
|
||||||
from mcm26b.planner.hrl_loop import HRLLoop
|
|
||||||
return HRLLoop(client, config, problem=problem, high_policy=None)
|
|
||||||
if variant == "hrl":
|
|
||||||
if not hrl_model:
|
|
||||||
raise RuntimeError("variant=hrl 需要 --hrl-model")
|
|
||||||
from mcm26b.hrl.policy import HighLevelPolicy
|
|
||||||
from mcm26b.planner.hrl_loop import HRLLoop
|
|
||||||
policy = HighLevelPolicy(hrl_model, device=hrl_device, problem=problem)
|
|
||||||
return HRLLoop(client, config, problem=problem, high_policy=policy)
|
|
||||||
raise ValueError(f"未知 variant: {variant!r}")
|
|
||||||
|
|
||||||
|
|
||||||
def _validity(summary: dict[str, Any], truth_channels: tuple[int, ...]) -> dict[str, Any]:
|
|
||||||
"""测试端有效性审计。真值只在此处进入。"""
|
|
||||||
cleared = set(int(c) for c in summary.get("cleared_channels", []))
|
|
||||||
absent = set(int(c) for c in summary.get("absent_channels", []))
|
|
||||||
truth = set(truth_channels)
|
|
||||||
declared_absent = truth & absent
|
|
||||||
cleared_matches_truth = cleared == truth
|
|
||||||
no_source_declared_absent = not declared_absent
|
|
||||||
completed = bool(summary.get("completed"))
|
|
||||||
anomalies = list(summary.get("anomalies", []))
|
|
||||||
# 本地 benchmark 要求：清洗真值且无守恒性异常；缺席集合不得含真值。
|
|
||||||
valid = completed and cleared_matches_truth and no_source_declared_absent \
|
|
||||||
and not anomalies
|
|
||||||
return {
|
|
||||||
"valid": valid,
|
|
||||||
"completed": completed,
|
|
||||||
"cleared_matches_truth": cleared_matches_truth,
|
|
||||||
"no_source_declared_absent": no_source_declared_absent,
|
|
||||||
"truth_channels": sorted(truth),
|
|
||||||
"declared_absent_truth": sorted(declared_absent),
|
|
||||||
"anomaly_count": len(anomalies),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _row_from_summary(*, scene: SceneSpec, variant: str, summary: dict[str, Any],
|
|
||||||
wall_s: float) -> dict[str, Any]:
|
|
||||||
validity = _validity(summary, scene.truth_channels)
|
|
||||||
cleared_count = len(summary.get("cleared_channels", []))
|
|
||||||
virtual_s = float(summary.get("virtual_time_s") or 0.0)
|
|
||||||
avg_s = summary.get("avg_locate_clear_time_s")
|
|
||||||
totals = summary.get("step_totals", {}) or {}
|
|
||||||
return {
|
|
||||||
"scene_id": scene.scene_id,
|
|
||||||
"variant": variant,
|
|
||||||
"problem": scene.problem,
|
|
||||||
"truth_count": len(scene.truth_channels),
|
|
||||||
"cleared_count": cleared_count,
|
|
||||||
"virtual_time_s": virtual_s,
|
|
||||||
"avg_locate_clear_time_s": (float(avg_s) if avg_s is not None else None),
|
|
||||||
"accounted_virtual_time_s": summary.get("accounted_virtual_time_s"),
|
|
||||||
"program_run_time_s": round(wall_s, 6),
|
|
||||||
"real_elapsed_total_s": totals.get("real_elapsed_total_s"),
|
|
||||||
"total_requests": summary.get("total_requests"),
|
|
||||||
"measures": summary.get("measures"),
|
|
||||||
"clears": summary.get("clears"),
|
|
||||||
"clear_success": summary.get("clear_success"),
|
|
||||||
"completed": bool(summary.get("completed")),
|
|
||||||
"stop_reason": summary.get("stop_reason"),
|
|
||||||
"exit_reason": summary.get("exit_reason"),
|
|
||||||
**validity,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _error_row(scene: SceneSpec, variant: str, exc: BaseException) -> dict[str, Any]:
|
|
||||||
return {
|
|
||||||
"scene_id": scene.scene_id,
|
|
||||||
"variant": variant,
|
|
||||||
"problem": scene.problem,
|
|
||||||
"truth_count": len(scene.truth_channels),
|
|
||||||
"cleared_count": 0,
|
|
||||||
"virtual_time_s": None,
|
|
||||||
"avg_locate_clear_time_s": None,
|
|
||||||
"accounted_virtual_time_s": None,
|
|
||||||
"program_run_time_s": None,
|
|
||||||
"real_elapsed_total_s": None,
|
|
||||||
"total_requests": None,
|
|
||||||
"measures": None,
|
|
||||||
"clears": None,
|
|
||||||
"clear_success": None,
|
|
||||||
"completed": False,
|
|
||||||
"stop_reason": None,
|
|
||||||
"exit_reason": None,
|
|
||||||
"valid": False,
|
|
||||||
"cleared_matches_truth": False,
|
|
||||||
"no_source_declared_absent": False,
|
|
||||||
"truth_channels": sorted(scene.truth_channels),
|
|
||||||
"declared_absent_truth": [],
|
|
||||||
"anomaly_count": 0,
|
|
||||||
"error": f"{type(exc).__name__}: {exc}",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def run_scene_variant(scene: SceneSpec, variant: str, *, config: ProblemConfig,
|
|
||||||
hrl_model: str | None, run_dir: Path,
|
|
||||||
hrl_device: str = "auto") -> dict[str, Any]:
|
|
||||||
"""运行一个 scene × variant。返回可序列化结果行与 summary。"""
|
|
||||||
run_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
sources = [_source_from_dict(raw) for raw in scene.sources]
|
|
||||||
simulator = LocalSimulator(
|
|
||||||
sources,
|
|
||||||
robot_id="hrl-benchmark",
|
|
||||||
spec=referee_spec(config),
|
|
||||||
error_seed=scene.error_seed,
|
|
||||||
stress_mode=False,
|
|
||||||
)
|
|
||||||
# 真值只用于 LocalSimulator；策略端仅持有 client/loop，不读取 simulator.sources。
|
|
||||||
log_path = run_dir / "session.jsonl"
|
|
||||||
log = SessionLog(path=log_path)
|
|
||||||
client = SimulatorClient("hrl-benchmark", config=config,
|
|
||||||
transport=LocalSimTransport(simulator), log=log)
|
|
||||||
loop = _build_loop(variant, client, config, scene.problem,
|
|
||||||
hrl_model=hrl_model, hrl_device=hrl_device)
|
|
||||||
started = time.monotonic()
|
|
||||||
try:
|
|
||||||
summary = loop.run()
|
|
||||||
finally:
|
|
||||||
wall_s = time.monotonic() - started
|
|
||||||
row = _row_from_summary(scene=scene, variant=variant, summary=summary,
|
|
||||||
wall_s=wall_s)
|
|
||||||
# 完整 summary 可能很大；仍写入 runs 目录，供复算与异常审计。
|
|
||||||
(run_dir / "summary.json").write_text(
|
|
||||||
json.dumps(summary, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
||||||
return {"row": row, "summary": summary}
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
# aggregate
|
|
||||||
# ---------------------------------------------------------------------------
|
|
||||||
def _aggregate(rows: list[dict[str, Any]]) -> dict[str, Any]:
|
|
||||||
grouped: dict[str, list[dict[str, Any]]] = defaultdict(list)
|
|
||||||
for row in rows:
|
|
||||||
grouped[row["variant"]].append(row)
|
|
||||||
result: dict[str, Any] = {"variants": {}, "all_valid": True}
|
|
||||||
for variant, items in sorted(grouped.items()):
|
|
||||||
virtual = [float(item["virtual_time_s"]) for item in items
|
|
||||||
if item.get("virtual_time_s") is not None]
|
|
||||||
per_source = [float(item["avg_locate_clear_time_s"])
|
|
||||||
for item in items
|
|
||||||
if item.get("avg_locate_clear_time_s") is not None]
|
|
||||||
cleared = [int(item["cleared_count"] or 0) for item in items]
|
|
||||||
program = [float(item["program_run_time_s"]) for item in items
|
|
||||||
if item.get("program_run_time_s") is not None]
|
|
||||||
success = [1.0 if item.get("valid") else 0.0 for item in items]
|
|
||||||
result["variants"][variant] = {
|
|
||||||
"episodes": len(items),
|
|
||||||
"valid_episodes": int(sum(success)),
|
|
||||||
"valid_rate": (sum(success) / len(items)) if items else 0.0,
|
|
||||||
"success_rate": (sum(success) / len(items)) if items else 0.0,
|
|
||||||
"cleared_count_mean": statistics.fmean(cleared) if cleared else 0.0,
|
|
||||||
"cleared_count_std": (statistics.pstdev(cleared)
|
|
||||||
if len(cleared) > 1 else 0.0),
|
|
||||||
"virtual_time_mean_s": statistics.fmean(virtual) if virtual else 0.0,
|
|
||||||
"virtual_time_std_s": (statistics.pstdev(virtual)
|
|
||||||
if len(virtual) > 1 else 0.0),
|
|
||||||
"virtual_time_min_s": min(virtual) if virtual else 0.0,
|
|
||||||
"virtual_time_max_s": max(virtual) if virtual else 0.0,
|
|
||||||
"avg_locate_clear_time_mean_s": (statistics.fmean(per_source)
|
|
||||||
if per_source else None),
|
|
||||||
"program_run_time_mean_s": (statistics.fmean(program)
|
|
||||||
if program else None),
|
|
||||||
"total_requests_mean": statistics.fmean(
|
|
||||||
[int(item.get("total_requests") or 0) for item in items]),
|
|
||||||
}
|
|
||||||
if result["variants"][variant]["valid_rate"] < 1.0:
|
|
||||||
result["all_valid"] = False
|
|
||||||
return result
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("--variants", default="hrl_heuristic,bayes,directional",
|
|
||||||
help="逗号分隔：hrl_heuristic,bayes,directional,hrl")
|
|
||||||
parser.add_argument("--manifest", default="output/hrl/manifests/dev.json")
|
|
||||||
parser.add_argument("--output", default="output/hrl/p1-benchmark")
|
|
||||||
parser.add_argument("--problem", type=int, choices=(3, 4), default=3)
|
|
||||||
parser.add_argument("--scenes", type=int, default=6,
|
|
||||||
help="manifest 不存在时生成的场景数")
|
|
||||||
parser.add_argument("--sources", type=int, default=None,
|
|
||||||
help="每局源数；默认按题面 10~16 随机")
|
|
||||||
parser.add_argument("--directional", type=int, default=3,
|
|
||||||
help="问题4定向源数量（生成 manifest 时使用）")
|
|
||||||
parser.add_argument("--start-seed", type=int, default=91_000)
|
|
||||||
parser.add_argument("--hrl-model", default=None)
|
|
||||||
parser.add_argument("--hrl-device", choices=("auto", "cpu", "cuda", "mps"),
|
|
||||||
default="auto")
|
|
||||||
parser.add_argument("--force", action="store_true",
|
|
||||||
help="允许覆盖已有输出目录")
|
|
||||||
parser.add_argument("--log", action="store_true",
|
|
||||||
help="为兼容 CLI 保留；当前始终写 session.jsonl")
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
|
|
||||||
config = ProblemConfig()
|
|
||||||
output = Path(args.output)
|
|
||||||
if output.exists() and any(output.iterdir()) and not args.force:
|
|
||||||
print(f"输出目录已存在且非空：{output}；如需覆盖请加 --force。",
|
|
||||||
file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
output.mkdir(parents=True, exist_ok=True)
|
|
||||||
manifest_path = Path(args.manifest)
|
|
||||||
manifest = load_or_generate_manifest(
|
|
||||||
manifest_path, problem=args.problem, scenes=args.scenes,
|
|
||||||
sources=args.sources, directional=args.directional,
|
|
||||||
start_seed=args.start_seed, config=config)
|
|
||||||
scenes = scenes_from_manifest(manifest)
|
|
||||||
variants = [item.strip() for item in args.variants.split(",") if item.strip()]
|
|
||||||
if not variants:
|
|
||||||
parser.error("--variants 不能为空")
|
|
||||||
if "hrl" in variants and not args.hrl_model:
|
|
||||||
# manifest 中也允许写明模型路径；此处先支持 CLI 显式路径。
|
|
||||||
print("variant=hrl 需要 --hrl-model（P1 冒烟模型路径）。", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
known = {"hrl_heuristic", "bayes", "directional", "hrl"}
|
|
||||||
unknown = sorted(set(variants) - known)
|
|
||||||
if unknown:
|
|
||||||
parser.error(f"未知 variant: {unknown}")
|
|
||||||
|
|
||||||
rows: list[dict[str, Any]] = []
|
|
||||||
runs_root = output / "runs"
|
|
||||||
wall_started = time.monotonic()
|
|
||||||
for scene in scenes:
|
|
||||||
for variant in variants:
|
|
||||||
run_dir = runs_root / scene.scene_id / variant
|
|
||||||
print(json.dumps({"event": "run_start", "scene": scene.scene_id,
|
|
||||||
"variant": variant}, ensure_ascii=False))
|
|
||||||
try:
|
|
||||||
result = run_scene_variant(
|
|
||||||
scene, variant, config=config, hrl_model=args.hrl_model,
|
|
||||||
run_dir=run_dir, hrl_device=args.hrl_device)
|
|
||||||
row = result["row"]
|
|
||||||
except Exception as exc: # 单场异常也进入聚合，all_valid=false
|
|
||||||
row = _error_row(scene, variant, exc)
|
|
||||||
rows.append(row)
|
|
||||||
print(json.dumps({"event": "run_done", **{
|
|
||||||
key: row.get(key) for key in
|
|
||||||
("scene_id", "variant", "cleared_count", "virtual_time_s",
|
|
||||||
"valid", "program_run_time_s", "error")}}, ensure_ascii=False))
|
|
||||||
aggregate = _aggregate(rows)
|
|
||||||
aggregate["wall_seconds"] = round(time.monotonic() - wall_started, 3)
|
|
||||||
aggregate["manifest"] = str(manifest_path)
|
|
||||||
aggregate["output"] = str(output)
|
|
||||||
aggregate["variants_requested"] = variants
|
|
||||||
aggregate["scenes"] = len(scenes)
|
|
||||||
(output / "results.json").write_text(
|
|
||||||
json.dumps(rows, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
||||||
(output / "manifest.json").write_text(
|
|
||||||
json.dumps(manifest, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
||||||
(output / "aggregate.json").write_text(
|
|
||||||
json.dumps(aggregate, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
||||||
fieldnames = list(rows[0].keys())
|
|
||||||
for row in rows:
|
|
||||||
for key in row:
|
|
||||||
if key not in fieldnames:
|
|
||||||
fieldnames.append(key)
|
|
||||||
with (output / "aggregate.csv").open("w", newline="", encoding="utf-8") as handle:
|
|
||||||
writer = csv.DictWriter(handle, fieldnames=fieldnames)
|
|
||||||
writer.writeheader()
|
|
||||||
writer.writerows(rows)
|
|
||||||
print(json.dumps(aggregate, ensure_ascii=False, indent=2))
|
|
||||||
return 0 if aggregate.get("all_valid") else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,128 +0,0 @@
|
|||||||
"""问题3/4 RL 策略与既有策略的配对自建消融。
|
|
||||||
|
|
||||||
同一种子、同一场景分别跑基线（Q3 bayes / Q4 directional）与 RL 变体，
|
|
||||||
比较全部清除成功、虚拟总时间、检测次数与请求总数。结果写入
|
|
||||||
output/rl/benchmark.json。
|
|
||||||
|
|
||||||
用法：
|
|
||||||
uv sync --group rl --group dev
|
|
||||||
uv run python scripts/benchmark_rl.py --seeds 5
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import statistics
|
|
||||||
import time
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.evaluation import recompute_virtual_time
|
|
||||||
from mcm26b.harness import (LocalSimTransport, mixed_scene, random_omni_scene,
|
|
||||||
referee_spec)
|
|
||||||
from mcm26b.planner import DirectionalExtension
|
|
||||||
from mcm26b.planner.bayes_loop import BayesLoop
|
|
||||||
from mcm26b.planner.rl_loop import RLBayesLoop, RLDirectionalLoop
|
|
||||||
from mcm26b.protocol.client import SimulatorClient
|
|
||||||
from referee import LocalSimulator
|
|
||||||
|
|
||||||
|
|
||||||
def run_one(loop_cls, sources, *, config, robot_id: str, **kwargs) -> dict:
|
|
||||||
simulator = LocalSimulator(sources, robot_id=robot_id,
|
|
||||||
spec=referee_spec(config, max_real_duration_s=3600.0),
|
|
||||||
stress_mode=True)
|
|
||||||
client = SimulatorClient(robot_id, config=config,
|
|
||||||
transport=LocalSimTransport(simulator))
|
|
||||||
started = time.monotonic()
|
|
||||||
summary = loop_cls(client, config, **kwargs).run() \
|
|
||||||
if kwargs else loop_cls(client, config).run()
|
|
||||||
events = client.log.action_events()
|
|
||||||
successful = {e['request']['channel'] for e in events
|
|
||||||
if e['response'].get('clear_result') == 'success'}
|
|
||||||
truth = {s.channel for s in sources}
|
|
||||||
return {
|
|
||||||
"completed": bool(summary["completed"]),
|
|
||||||
"all_cleared": successful == truth,
|
|
||||||
"correct_absence": not bool(set(summary['absent_channels']) & truth),
|
|
||||||
"anomalies": summary['anomalies'],
|
|
||||||
"timing_mismatches": len(recompute_virtual_time(events)['mismatches']),
|
|
||||||
"normal_exit": summary['exit_reason'] == 'user_exit',
|
|
||||||
"virtual_time_s": float(summary["virtual_time_s"]),
|
|
||||||
"measures": int(summary["measures"]),
|
|
||||||
"total_requests": int(summary["total_requests"]),
|
|
||||||
"move_distance_m": float(summary["move_distance_m"]),
|
|
||||||
"wall_s": round(time.monotonic() - started, 3),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("--seeds", type=int, default=5)
|
|
||||||
parser.add_argument("--sources", type=int, default=10)
|
|
||||||
parser.add_argument("--directional", type=int, default=3)
|
|
||||||
parser.add_argument("--output", default="output/rl/benchmark.json")
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
config = ProblemConfig()
|
|
||||||
robot_id = "2026001"
|
|
||||||
report = {"config": {"seeds": args.seeds, "sources": args.sources,
|
|
||||||
"directional": args.directional},
|
|
||||||
"q3": [], "q4": []}
|
|
||||||
|
|
||||||
for seed in range(20260910, 20260910 + args.seeds):
|
|
||||||
omni = random_omni_scene(seed=seed, count=args.sources, config=config)
|
|
||||||
mixed = mixed_scene(seed=seed, count=args.sources,
|
|
||||||
directional=args.directional, config=config)
|
|
||||||
q3_base = run_one(BayesLoop, omni, config=config, robot_id=robot_id)
|
|
||||||
q3_rl = run_one(RLBayesLoop, omni, config=config, robot_id=robot_id)
|
|
||||||
q4_base = run_one(DirectionalExtension, mixed, config=config,
|
|
||||||
robot_id=robot_id)
|
|
||||||
q4_rl = run_one(RLDirectionalLoop, mixed, config=config,
|
|
||||||
robot_id=robot_id)
|
|
||||||
report["q3"].append({"seed": seed, "bayes": q3_base, "rl": q3_rl})
|
|
||||||
report["q4"].append({"seed": seed, "directional": q4_base, "rl": q4_rl})
|
|
||||||
print(json.dumps({"event": "seed_done", "seed": seed,
|
|
||||||
"q3_base_s": q3_base["virtual_time_s"],
|
|
||||||
"q3_rl_s": q3_rl["virtual_time_s"],
|
|
||||||
"q4_base_s": q4_base["virtual_time_s"],
|
|
||||||
"q4_rl_s": q4_rl["virtual_time_s"]},
|
|
||||||
ensure_ascii=False))
|
|
||||||
|
|
||||||
for key, baseline in (("q3", "bayes"), ("q4", "directional")):
|
|
||||||
rows = report[key]
|
|
||||||
deltas = [row["rl"]["virtual_time_s"] - row[baseline]["virtual_time_s"]
|
|
||||||
for row in rows]
|
|
||||||
cleared = all(row["rl"]["all_cleared"] and row[baseline]["all_cleared"]
|
|
||||||
for row in rows)
|
|
||||||
valid = all(part['completed'] and part['all_cleared'] and part['correct_absence']
|
|
||||||
and part['normal_exit'] and not part['anomalies']
|
|
||||||
and not part['timing_mismatches']
|
|
||||||
for row in rows for part in (row[baseline],row['rl']))
|
|
||||||
report[key + "_summary"] = {
|
|
||||||
"mean_virtual_s": {
|
|
||||||
baseline: statistics.mean(row[baseline]["virtual_time_s"] for row in rows),
|
|
||||||
"rl": statistics.mean(row["rl"]["virtual_time_s"] for row in rows),
|
|
||||||
},
|
|
||||||
"mean_delta_s": statistics.mean(deltas),
|
|
||||||
"relative_delta_pct": 100.0 * statistics.mean(deltas)
|
|
||||||
/ statistics.mean(row[baseline]["virtual_time_s"] for row in rows),
|
|
||||||
"mean_measures": {
|
|
||||||
baseline: statistics.mean(row[baseline]["measures"] for row in rows),
|
|
||||||
"rl": statistics.mean(row["rl"]["measures"] for row in rows),
|
|
||||||
},
|
|
||||||
"all_cleared": cleared,
|
|
||||||
"all_valid": valid,
|
|
||||||
}
|
|
||||||
output = Path(args.output)
|
|
||||||
output.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
output.write_text(json.dumps(report, ensure_ascii=False, indent=2),
|
|
||||||
encoding="utf-8")
|
|
||||||
print(json.dumps({"event": "benchmark_done", "path": str(output),
|
|
||||||
"q3": report["q3_summary"],
|
|
||||||
"q4": report["q4_summary"]},
|
|
||||||
ensure_ascii=False, indent=2))
|
|
||||||
return 0 if report['q3_summary']['all_valid'] and report['q4_summary']['all_valid'] else 1
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,125 +0,0 @@
|
|||||||
"""同场景配对比较手写策略、旧PPO及修订模型；真值仅在评测端使用。"""
|
|
||||||
import argparse
|
|
||||||
from dataclasses import asdict
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import math
|
|
||||||
from pathlib import Path
|
|
||||||
from statistics import mean, median
|
|
||||||
from time import perf_counter
|
|
||||||
|
|
||||||
import torch
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.evaluation import recompute_virtual_time
|
|
||||||
from mcm26b.harness import LocalSimTransport, mixed_scene, random_omni_scene, referee_spec
|
|
||||||
from mcm26b.planner import DirectionalExtension
|
|
||||||
from mcm26b.planner.bayes_loop import BayesLoop
|
|
||||||
from mcm26b.planner.rl_loop import RLBayesLoop, RLDirectionalLoop
|
|
||||||
from mcm26b.protocol.client import SimulatorClient
|
|
||||||
from referee import LocalSimulator, Source
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument('--split', choices=('development','validation','stress'),
|
|
||||||
default='development')
|
|
||||||
parser.add_argument('--variants', default='baseline,legacy,revised')
|
|
||||||
parser.add_argument('--problems', default='3,4')
|
|
||||||
parser.add_argument('--model-dir', type=Path, default=Path('output/rl/review-20260911'))
|
|
||||||
parser.add_argument('--label', required=True)
|
|
||||||
args = parser.parse_args()
|
|
||||||
torch.set_num_threads(1)
|
|
||||||
config = ProblemConfig()
|
|
||||||
output = args.model_dir / 'evaluation' / args.split / args.label
|
|
||||||
output.mkdir(parents=True, exist_ok=False)
|
|
||||||
variants = args.variants.split(',')
|
|
||||||
rows = []
|
|
||||||
def save():
|
|
||||||
aggregates = {}
|
|
||||||
for problem in (3,4):
|
|
||||||
for variant in variants:
|
|
||||||
part = [r for r in rows if r['problem']==problem and r['variant']==variant]
|
|
||||||
if part:
|
|
||||||
aggregates[f'q{problem}-{variant}'] = dict(
|
|
||||||
runs=len(part), valid=sum(r['valid'] for r in part),
|
|
||||||
mean_virtual_s=mean(r['virtual_s'] for r in part),
|
|
||||||
median_virtual_s=median(r['virtual_s'] for r in part),
|
|
||||||
max_virtual_s=max(r['virtual_s'] for r in part),
|
|
||||||
failed_clears=sum(r['failed_clears'] for r in part),
|
|
||||||
repeated_measures=sum(r['repeated_measures'] for r in part))
|
|
||||||
result = dict(source='local_referee_only', config=asdict(config), split=args.split,
|
|
||||||
rows=rows, aggregates=aggregates)
|
|
||||||
(output/'results.json').write_text(
|
|
||||||
json.dumps(result,ensure_ascii=False,indent=2),encoding='utf-8')
|
|
||||||
return aggregates
|
|
||||||
for problem in map(int, args.problems.split(',')):
|
|
||||||
scenes = []
|
|
||||||
if args.split == 'stress':
|
|
||||||
for count in (10,16):
|
|
||||||
scenes.append((f'boundary-{count}', count, 99000+count, [
|
|
||||||
Source(c+1,1800*math.cos(2*math.pi*c/count),1800*math.sin(2*math.pi*c/count),
|
|
||||||
1000,directional=(problem==4 and c%2==0),
|
|
||||||
heading_deg=360*c/count if problem==4 and c%2==0 else 0)
|
|
||||||
for c in range(count)]))
|
|
||||||
else:
|
|
||||||
seeds = range(700,703) if args.split=='development' else range(91000,91006)
|
|
||||||
counts = (10,16) if args.split=='development' else (10,13,16)
|
|
||||||
for count in counts:
|
|
||||||
for seed in seeds:
|
|
||||||
directional = (1, count//2, count-1)[seed%3]
|
|
||||||
sources = (random_omni_scene(seed, count, config) if problem==3 else
|
|
||||||
mixed_scene(seed, count, directional, config))
|
|
||||||
scenes.append((f'{count}-{seed}',count,seed,sources))
|
|
||||||
for name,count,seed,sources in scenes:
|
|
||||||
scene_data = dict(sources=[asdict(s) for s in sources], error_seed=seed+2000000)
|
|
||||||
scene_path = output/f'q{problem}-{name}-scene.json'
|
|
||||||
scene_path.write_text(json.dumps(scene_data),encoding='utf-8')
|
|
||||||
for variant in variants:
|
|
||||||
model = None
|
|
||||||
cls = BayesLoop if problem==3 else DirectionalExtension
|
|
||||||
kwargs = {}
|
|
||||||
if variant != 'baseline':
|
|
||||||
cls = RLBayesLoop if problem==3 else RLDirectionalLoop
|
|
||||||
model = (Path(f'output/rl/q{problem}_ppo.zip') if variant=='legacy'
|
|
||||||
else args.model_dir/f'q{problem}_masked.zip')
|
|
||||||
kwargs = dict(model_path=model, device='cpu')
|
|
||||||
sim = LocalSimulator(sources,robot_id='RL-AUDIT',spec=referee_spec(config),
|
|
||||||
error_seed=seed+2000000)
|
|
||||||
client = SimulatorClient('RL-AUDIT',config=config,transport=LocalSimTransport(sim))
|
|
||||||
start = perf_counter()
|
|
||||||
summary = cls(client,config,**kwargs).run()
|
|
||||||
events = client.log.action_events()
|
|
||||||
truth = {s.channel for s in sources}
|
|
||||||
successful = {e['request']['channel'] for e in events
|
|
||||||
if e['response'].get('clear_result')=='success'}
|
|
||||||
seen = set()
|
|
||||||
repeats = 0
|
|
||||||
for e in events:
|
|
||||||
if e['path']=='/measure':
|
|
||||||
p=e['request']['position']
|
|
||||||
tag=(e['request']['channel'],round(p['x'],6),round(p['y'],6))
|
|
||||||
repeats += tag in seen
|
|
||||||
seen.add(tag)
|
|
||||||
mismatches = recompute_virtual_time(events)['mismatches']
|
|
||||||
valid = (successful==truth and summary['completed'] and not summary['anomalies']
|
|
||||||
and not set(summary['absent_channels']) & truth
|
|
||||||
and summary['exit_reason']=='user_exit' and not mismatches)
|
|
||||||
row = dict(case=name,problem=problem,variant=variant,valid=valid,
|
|
||||||
virtual_s=client.state.virtual_time_s,all_cleared=successful==truth,
|
|
||||||
measures=summary['measures'],
|
|
||||||
failed_clears=summary['clears']-len(successful),
|
|
||||||
repeated_measures=repeats,wall_s=perf_counter()-start,
|
|
||||||
model_sha256=(hashlib.sha256(model.read_bytes()).hexdigest()
|
|
||||||
if model else None))
|
|
||||||
rows.append(row)
|
|
||||||
(output/f'q{problem}-{name}-{variant}.json').write_text(json.dumps(
|
|
||||||
dict(row=row,summary=summary,events=events,timing_mismatches=mismatches),
|
|
||||||
ensure_ascii=False),encoding='utf-8')
|
|
||||||
save()
|
|
||||||
print(json.dumps(row),flush=True)
|
|
||||||
print(json.dumps(save(),indent=2),flush=True)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__=='__main__':
|
|
||||||
main()
|
|
||||||
@@ -1,24 +0,0 @@
|
|||||||
"""高层专家轨迹预训练入口（P2 待实现）。
|
|
||||||
|
|
||||||
按 docs/hrl-sol-execution.md，P2 才运行 `hrl_heuristic` 收集
|
|
||||||
(episode_id, state, candidates, mask, action)，再按 episode 划分训练/验证集做 BC。
|
|
||||||
本文件在 P0+P1 阶段只提供命令行占位，避免把未实现的 BC 说成已交付。
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("--problem", type=int, choices=(3, 4), default=3)
|
|
||||||
parser.add_argument("--episodes", type=int, default=3000)
|
|
||||||
parser.add_argument("--output", default="output/hrl/bc_high")
|
|
||||||
parser.parse_args(argv)
|
|
||||||
raise NotImplementedError(
|
|
||||||
"P2 尚未实现：专家轨迹收集/BC 需要先通过 P1 smoke；"
|
|
||||||
"当前请使用 scripts/benchmark_hrl.py --variants hrl_heuristic 做启发式骨架验证。")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,255 +0,0 @@
|
|||||||
"""用自建模拟器的既有专家逻辑做行为克隆，初始化 PPO 策略网络。
|
|
||||||
|
|
||||||
思路：自建 referee 的观测模型与真值逻辑既快又可复现。先在它上面用
|
|
||||||
手写专家（Q3: worst-posterior 选点器；Q4: 遮挡梯子选点器）生成
|
|
||||||
(观测, 专家动作) 数据集，训练一个与 PPO policy_net 同构的 MLP 分类器，
|
|
||||||
再把权重拷入 PPO 模型作为初始化；随后 scripts/train_rl.py --init 继续
|
|
||||||
PPO 微调。这样 RL 从强先验出发，而不是从随机策略开始。
|
|
||||||
|
|
||||||
产物：
|
|
||||||
output/rl/bc/q3_expert.npz output/rl/bc/q3_bc.pt output/rl/q3_ppo_bc_init.zip
|
|
||||||
output/rl/bc/q4_expert.npz output/rl/bc/q4_bc.pt output/rl/q4_ppo_bc_init.zip
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import math
|
|
||||||
import time
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
import torch
|
|
||||||
from stable_baselines3 import PPO
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.planner import safe_disk_radius, select_measurement_point
|
|
||||||
from mcm26b.rl._constants import NUM_ACTIONS, OBS_DIM
|
|
||||||
from mcm26b.rl.env import LocalizationEnv
|
|
||||||
from mcm26b.rl.features import candidate_actions
|
|
||||||
|
|
||||||
HIDDEN = (128, 128)
|
|
||||||
|
|
||||||
|
|
||||||
def expert_action(env: LocalizationEnv) -> int:
|
|
||||||
"""当前状态的专家动作编号（与训练环境候选同源，就近映射）。"""
|
|
||||||
config = env.config
|
|
||||||
channel = env._channel
|
|
||||||
region = env._belief.candidate_set(channel)
|
|
||||||
position = env._client.state.position
|
|
||||||
blocked = env._blocked
|
|
||||||
candidates = candidate_actions(region, position, config, blocked)
|
|
||||||
|
|
||||||
if env.problem == 3:
|
|
||||||
expert_point = select_measurement_point(
|
|
||||||
region, position, config).position
|
|
||||||
else:
|
|
||||||
center, radius = region.mec()
|
|
||||||
safe = safe_disk_radius(region, config)
|
|
||||||
if not blocked:
|
|
||||||
expert_point = select_measurement_point(
|
|
||||||
region, position, config).position
|
|
||||||
elif len(blocked) == 1 and safe >= 0:
|
|
||||||
first = blocked[0]
|
|
||||||
dx, dy = center[0] - first[0], center[1] - first[1]
|
|
||||||
norm = math.hypot(dx, dy)
|
|
||||||
if norm <= 1e-9:
|
|
||||||
expert_point = (center[0] + safe, center[1])
|
|
||||||
else:
|
|
||||||
scale = min(1.0, safe / norm)
|
|
||||||
expert_point = (center[0] + dx * scale,
|
|
||||||
center[1] + dy * scale)
|
|
||||||
else:
|
|
||||||
first = blocked[0]
|
|
||||||
base = math.atan2(first[1] - center[1], first[0] - center[0])
|
|
||||||
angle = base + math.radians(45.0 * (len(blocked) - 1))
|
|
||||||
expert_point = (center[0] + radius * math.cos(angle),
|
|
||||||
center[1] + radius * math.sin(angle))
|
|
||||||
return min(range(len(candidates)),
|
|
||||||
key=lambda k: math.dist(candidates[k], expert_point))
|
|
||||||
|
|
||||||
|
|
||||||
class BCMlp(torch.nn.Module):
|
|
||||||
def __init__(self):
|
|
||||||
super().__init__()
|
|
||||||
self.net = torch.nn.Sequential(
|
|
||||||
torch.nn.Linear(OBS_DIM, HIDDEN[0]), torch.nn.Tanh(),
|
|
||||||
torch.nn.Linear(HIDDEN[0], HIDDEN[1]), torch.nn.Tanh(),
|
|
||||||
torch.nn.Linear(HIDDEN[1], NUM_ACTIONS))
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
return self.net(x)
|
|
||||||
|
|
||||||
|
|
||||||
def collect(problem: int, episodes: int, seed: int, config: ProblemConfig):
|
|
||||||
env = LocalizationEnv(
|
|
||||||
config, problem=problem,
|
|
||||||
directional_probability=1.0 if problem == 4 else 0.0,
|
|
||||||
max_episode_steps=16)
|
|
||||||
observations, actions, groups = [], [], []
|
|
||||||
for episode in range(episodes):
|
|
||||||
obs, _info = env.reset(seed=seed + episode)
|
|
||||||
for _step in range(env.max_episode_steps + 2):
|
|
||||||
action = expert_action(env)
|
|
||||||
observations.append(obs)
|
|
||||||
actions.append(action)
|
|
||||||
groups.append(episode)
|
|
||||||
obs, _reward, terminated, truncated, _info = env.step(action)
|
|
||||||
if terminated or truncated:
|
|
||||||
break
|
|
||||||
return np.asarray(observations, dtype=np.float32), \
|
|
||||||
np.asarray(actions, dtype=np.int64), np.asarray(groups, dtype=np.int64)
|
|
||||||
|
|
||||||
|
|
||||||
def train_bc(x, y, *, epochs: int, batch_size: int, lr: float,
|
|
||||||
device: str, groups, valid_frac: float = 0.1):
|
|
||||||
model = BCMlp().to(device)
|
|
||||||
optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
|
|
||||||
episodes = np.unique(groups)
|
|
||||||
if len(episodes) < 2:
|
|
||||||
raise ValueError('行为克隆至少需要两个独立episode才能划分验证集')
|
|
||||||
valid_groups = np.random.default_rng(0).permutation(episodes)
|
|
||||||
valid_groups = valid_groups[:max(1, int(len(episodes)*valid_frac))]
|
|
||||||
valid = torch.as_tensor(np.isin(groups, valid_groups), device=device)
|
|
||||||
x_t, y_t = torch.as_tensor(x, device=device), torch.as_tensor(y, device=device)
|
|
||||||
x_v, y_v = x_t[valid], y_t[valid]
|
|
||||||
x_tr, y_tr = x_t[~valid], y_t[~valid]
|
|
||||||
loss_fn = torch.nn.CrossEntropyLoss()
|
|
||||||
for epoch in range(epochs):
|
|
||||||
order = torch.randperm(len(x_tr), device=device)
|
|
||||||
total = correct = 0.0
|
|
||||||
for start in range(0, len(order), batch_size):
|
|
||||||
idx = order[start:start + batch_size]
|
|
||||||
logits = model(x_tr[idx])
|
|
||||||
loss = loss_fn(logits, y_tr[idx])
|
|
||||||
optimizer.zero_grad()
|
|
||||||
loss.backward()
|
|
||||||
optimizer.step()
|
|
||||||
total += float(loss.item()) * len(idx)
|
|
||||||
correct += int((logits.argmax(1) == y_tr[idx]).sum().item())
|
|
||||||
train_acc = correct / len(x_tr)
|
|
||||||
with torch.no_grad():
|
|
||||||
valid_acc = float((model(x_v).argmax(1) == y_v).float().mean().item())
|
|
||||||
if epoch == 0 or (epoch + 1) % 5 == 0:
|
|
||||||
print(json.dumps({"event": "bc_epoch", "epoch": epoch + 1,
|
|
||||||
"loss": round(total / len(x_tr), 4),
|
|
||||||
"train_acc": round(train_acc, 4),
|
|
||||||
"valid_acc": round(valid_acc, 4)},
|
|
||||||
ensure_ascii=False))
|
|
||||||
return model, float(valid_acc)
|
|
||||||
|
|
||||||
|
|
||||||
def copy_into_ppo(problem: int, model: BCMlp, device: str) -> Path | None:
|
|
||||||
"""把 BC 网络权重按 SB3 的 policy_net/action_net 拆分拷入 PPO。"""
|
|
||||||
# 当前奖励/特征已修订，重新初始化critic，不能沿用旧任务的价值预测。
|
|
||||||
env = LocalizationEnv(problem=problem)
|
|
||||||
ppo = PPO('MlpPolicy', env, policy_kwargs={'net_arch': list(HIDDEN)},
|
|
||||||
n_steps=512, batch_size=128, gamma=1.0, device=device)
|
|
||||||
state = {key.removeprefix("net."): value
|
|
||||||
for key, value in model.state_dict().items()}
|
|
||||||
# SB3 MlpPolicy：policy_net 只含前两个 Linear（Tanh 激活），
|
|
||||||
# 输出层在 action_net。BC 的 Sequential 键为 0/2/4。
|
|
||||||
policy_state = {key: value for key, value in state.items()
|
|
||||||
if key in ("0.weight", "0.bias", "2.weight", "2.bias")}
|
|
||||||
action_state = {
|
|
||||||
"weight": state["4.weight"], "bias": state["4.bias"]}
|
|
||||||
ppo.policy.mlp_extractor.policy_net.load_state_dict(policy_state)
|
|
||||||
ppo.policy.action_net.load_state_dict(action_state)
|
|
||||||
# 数值检查整个前向映射，而不只是线性层形状。
|
|
||||||
probe = torch.randn((8, OBS_DIM), device=device)
|
|
||||||
with torch.no_grad():
|
|
||||||
copied = ppo.policy.action_net(ppo.policy.mlp_extractor.policy_net(probe))
|
|
||||||
if not torch.allclose(model(probe), copied, atol=1e-6):
|
|
||||||
raise RuntimeError('行为克隆与PPO激活函数/前向映射不一致')
|
|
||||||
init_path = Path(f"output/rl/q{problem}_ppo_bc_init.zip")
|
|
||||||
ppo.save(str(init_path))
|
|
||||||
env.close()
|
|
||||||
init_path.with_suffix('.json').write_text(json.dumps(
|
|
||||||
dict(algorithm='ppo', feature_version=2, problem=problem)), encoding='utf-8')
|
|
||||||
return init_path
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("--problem", type=int, choices=(3, 4), required=True)
|
|
||||||
parser.add_argument("--episodes", type=int, default=3000)
|
|
||||||
parser.add_argument("--epochs", type=int, default=20)
|
|
||||||
parser.add_argument("--batch-size", type=int, default=256)
|
|
||||||
parser.add_argument("--lr", type=float, default=3e-4)
|
|
||||||
parser.add_argument("--seed", type=int, default=20260911)
|
|
||||||
parser.add_argument("--device", default="auto")
|
|
||||||
parser.add_argument("--copy-only", action="store_true",
|
|
||||||
help="跳过采集/训练，只把已有 q<problem>_bc.pt 拷入 PPO 初始化")
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
|
|
||||||
if args.device == "auto":
|
|
||||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
|
||||||
else:
|
|
||||||
device = args.device
|
|
||||||
print(json.dumps({"event": "bc_start", "problem": args.problem,
|
|
||||||
"episodes": args.episodes, "device": device,
|
|
||||||
"cuda": torch.cuda.is_available()},
|
|
||||||
ensure_ascii=False))
|
|
||||||
|
|
||||||
if args.copy_only:
|
|
||||||
bc_dir = Path("output/rl/bc")
|
|
||||||
checkpoint = torch.load(bc_dir / f"q{args.problem}_bc.pt",
|
|
||||||
map_location="cpu")
|
|
||||||
if checkpoint.get('activation') != 'Tanh' or checkpoint.get('feature_version') != 2:
|
|
||||||
parser.error('旧ReLU/旧特征BC检查点不能直接拷贝，请重新采集并训练')
|
|
||||||
model = BCMlp()
|
|
||||||
model.load_state_dict(checkpoint["state_dict"])
|
|
||||||
init_path = copy_into_ppo(args.problem, model, device)
|
|
||||||
dataset = np.load(bc_dir / f"q{args.problem}_expert.npz",
|
|
||||||
allow_pickle=False)
|
|
||||||
report = {
|
|
||||||
"problem": args.problem,
|
|
||||||
"samples": int(len(dataset["obs"])),
|
|
||||||
"valid_accuracy": checkpoint.get("valid_acc"),
|
|
||||||
"dataset": str(bc_dir / f"q{args.problem}_expert.npz"),
|
|
||||||
"bc_model": str(bc_dir / f"q{args.problem}_bc.pt"),
|
|
||||||
"ppo_init": str(init_path) if init_path is not None else None,
|
|
||||||
}
|
|
||||||
(bc_dir / f"q{args.problem}_bc.json").write_text(
|
|
||||||
json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
||||||
print(json.dumps({"event": "bc_copy", "problem": args.problem,
|
|
||||||
"ppo_init": str(init_path), **report},
|
|
||||||
ensure_ascii=False, indent=2))
|
|
||||||
return 0 if init_path is not None else 2
|
|
||||||
|
|
||||||
config = ProblemConfig()
|
|
||||||
started = time.monotonic()
|
|
||||||
torch.manual_seed(args.seed)
|
|
||||||
x, y, groups = collect(args.problem, args.episodes, args.seed, config)
|
|
||||||
collect_s = time.monotonic() - started
|
|
||||||
bc_dir = Path("output/rl/bc")
|
|
||||||
bc_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
np.savez_compressed(bc_dir / f"q{args.problem}_expert.npz", obs=x, action=y, episode=groups)
|
|
||||||
|
|
||||||
model, valid_acc = train_bc(x, y, epochs=args.epochs,
|
|
||||||
batch_size=args.batch_size, lr=args.lr,
|
|
||||||
device=device, groups=groups)
|
|
||||||
torch.save({"state_dict": model.state_dict(), "obs_dim": OBS_DIM,
|
|
||||||
"actions": NUM_ACTIONS, "valid_acc": valid_acc,
|
|
||||||
"activation": "Tanh", "feature_version": 2},
|
|
||||||
bc_dir / f"q{args.problem}_bc.pt")
|
|
||||||
|
|
||||||
# 拷入现有 PPO 的 policy_net/action_net 作为初始化（critic 保持原权重）。
|
|
||||||
init_path = copy_into_ppo(args.problem, model, device)
|
|
||||||
report = {
|
|
||||||
"problem": args.problem, "episodes": args.episodes,
|
|
||||||
"samples": int(len(x)), "valid_accuracy": valid_acc,
|
|
||||||
"collect_seconds": round(collect_s, 1),
|
|
||||||
"train_seconds": round(time.monotonic() - started - collect_s, 1),
|
|
||||||
"dataset": str(bc_dir / f"q{args.problem}_expert.npz"),
|
|
||||||
"bc_model": str(bc_dir / f"q{args.problem}_bc.pt"),
|
|
||||||
"ppo_init": str(init_path) if init_path is not None else None,
|
|
||||||
}
|
|
||||||
(bc_dir / f"q{args.problem}_bc.json").write_text(
|
|
||||||
json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
||||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,36 +0,0 @@
|
|||||||
"""从 Tee 输出的 CLI 文本中提取末尾 JSON，落盘为 summary.json。
|
|
||||||
|
|
||||||
官方演练目录由 run 命令产生 cli-output.txt（前面可能有 CUDA 警告），
|
|
||||||
本脚本只提取最后一个完整 JSON 对象，并核对 adjacent 的官方 result.json。
|
|
||||||
"""
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
def extract(text: str) -> dict:
|
|
||||||
start = text.rfind("\n{")
|
|
||||||
if start < 0:
|
|
||||||
raise ValueError("未找到 JSON 对象")
|
|
||||||
return json.loads(text[start + 1:])
|
|
||||||
|
|
||||||
|
|
||||||
def main() -> int:
|
|
||||||
parser = argparse.ArgumentParser()
|
|
||||||
parser.add_argument("folder", type=Path)
|
|
||||||
args = parser.parse_args()
|
|
||||||
folder = args.folder
|
|
||||||
summary = extract((folder / "cli-output.txt").read_text(encoding="utf-8"))
|
|
||||||
(folder / "summary.json").write_text(
|
|
||||||
json.dumps(summary, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
|
|
||||||
print(json.dumps({
|
|
||||||
"folder": str(folder), "completed": summary["completed"],
|
|
||||||
"virtual_time_s": summary["virtual_time_s"],
|
|
||||||
"cleared_channels": summary["cleared_channels"],
|
|
||||||
"total_requests": summary["total_requests"],
|
|
||||||
}, ensure_ascii=False))
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,78 +0,0 @@
|
|||||||
"""汇总配对置信区间、历史官方证据与BC/特征错位探针，不连接模拟器。"""
|
|
||||||
from pathlib import Path
|
|
||||||
import json
|
|
||||||
import hashlib
|
|
||||||
import numpy as np
|
|
||||||
import torch
|
|
||||||
|
|
||||||
import verify_rl_practice
|
|
||||||
from mcm26b.rl.env import LocalizationEnv
|
|
||||||
from mcm26b.rl.features import build_observation
|
|
||||||
|
|
||||||
|
|
||||||
def main():
|
|
||||||
root = Path('output/rl/review-20260911')
|
|
||||||
records = json.loads((root/'evaluation/validation/final/results.json')
|
|
||||||
.read_text(encoding='utf-8'))
|
|
||||||
rng = np.random.default_rng(20260911)
|
|
||||||
comparisons = {}
|
|
||||||
for problem in (3,4):
|
|
||||||
part = [r for r in records['rows'] if r['problem']==problem]
|
|
||||||
for label in ('baseline','legacy'):
|
|
||||||
baseline = {r['case']:r for r in part if r['variant']==label}
|
|
||||||
selected = sorted((r for r in part if r['variant']=='revised'),key=lambda r:r['case'])
|
|
||||||
a = np.array([baseline[r['case']]['virtual_s'] for r in selected])
|
|
||||||
b = np.array([r['virtual_s'] for r in selected])
|
|
||||||
samples = rng.integers(0,len(a),(10000,len(a)))
|
|
||||||
changes = (b[samples].mean(axis=1)/a[samples].mean(axis=1)-1)*100
|
|
||||||
comparisons[f'q{problem}-revised-vs-{label}'] = dict(
|
|
||||||
relative_change_pct=(b.mean()/a.mean()-1)*100,
|
|
||||||
paired_bootstrap_95_pct=np.quantile(changes,[.025,.975]).tolist(),
|
|
||||||
faster_cases=int(sum(b<a)),runs=len(a))
|
|
||||||
# 重新计算原始官方日志的检查，派生报告单独保存，原始归档不写入。
|
|
||||||
verify_rl_practice.save_json = lambda path,value: None
|
|
||||||
official = [verify_rl_practice.audit(p.parent) for p in sorted(
|
|
||||||
Path('output/rl/official-practice').rglob('*.result.json'))]
|
|
||||||
env = LocalizationEnv(handoff_probability=0,boundary_probability=0)
|
|
||||||
fixed,_ = env.reset(seed=1)
|
|
||||||
# 复现旧版把整个BeliefState作为单频道对象传入的错误。
|
|
||||||
wrong = build_observation(env._belief.candidate_set(env._channel),env._client.state.position,
|
|
||||||
env._channel,env.config,env.problem,env._blocked,env._belief,
|
|
||||||
virtual_time_s=env._client.state.virtual_time_s)
|
|
||||||
features = dict(mismatched_indices=np.flatnonzero(fixed!=wrong).tolist(),
|
|
||||||
fixed_history_count=float(fixed[7]),old_history_count=float(wrong[7]))
|
|
||||||
env.close()
|
|
||||||
torch.set_num_threads(1)
|
|
||||||
torch.manual_seed(0)
|
|
||||||
bc = {}
|
|
||||||
for problem in (3,4):
|
|
||||||
# 使用旧检查点和原始专家输入，比对原ReLU网络与拷贝后的Tanh网络。
|
|
||||||
checkpoint = torch.load(f'output/rl/bc/q{problem}_bc.pt',map_location='cpu',
|
|
||||||
weights_only=True)
|
|
||||||
data = np.load(f'output/rl/bc/q{problem}_expert.npz',allow_pickle=False)
|
|
||||||
x = torch.from_numpy(data['obs'][::max(1,len(data['obs'])//1000)].copy())
|
|
||||||
|
|
||||||
def network(activation, checkpoint=checkpoint):
|
|
||||||
net = torch.nn.Sequential(torch.nn.Linear(401,128),activation(),
|
|
||||||
torch.nn.Linear(128,128),activation(),torch.nn.Linear(128,60))
|
|
||||||
state = {k.removeprefix('net.'):v for k,v in checkpoint['state_dict'].items()}
|
|
||||||
net.load_state_dict(state)
|
|
||||||
return net
|
|
||||||
with torch.no_grad():
|
|
||||||
before,after=network(torch.nn.ReLU)(x),network(torch.nn.Tanh)(x)
|
|
||||||
bc[f'q{problem}'] = dict(samples=len(x),
|
|
||||||
different_argmax_fraction=float((before.argmax(1)!=after.argmax(1)).float().mean()),
|
|
||||||
max_logit_difference=float((before-after).abs().max()))
|
|
||||||
report = dict(comparisons=comparisons,official=official,feature_probe=features,bc_probe=bc,
|
|
||||||
limitations='置信区间仅重采样固定模型的场景，不覆盖训练随机种子不确定性。',
|
|
||||||
code_sha256={str(p):hashlib.sha256(p.read_bytes()).hexdigest()
|
|
||||||
for p in sorted(Path('src/mcm26b/rl').glob('*.py'))})
|
|
||||||
(root/'review_evidence.json').write_text(
|
|
||||||
json.dumps(report,ensure_ascii=False,indent=2),encoding='utf-8')
|
|
||||||
print(json.dumps(dict(comparisons=comparisons,feature_probe=features,bc_probe=bc,
|
|
||||||
official=[dict(case=r['case_code'],passed=r['passed'])
|
|
||||||
for r in official]),indent=2))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__=='__main__':
|
|
||||||
main()
|
|
||||||
@@ -1,987 +0,0 @@
|
|||||||
"""高层 MaskablePPO 训练（P1）。
|
|
||||||
|
|
||||||
架构与边界：
|
|
||||||
- 环境 = :class:`mcm26b.hrl.hierarchical_env.HierarchicalEnv`，每一步是一个高层
|
|
||||||
宏动作（选择 0–119 槽位：clear/locate/station/fallback/保留）；
|
|
||||||
- 训练/部署共用 :mod:`mcm26b.hrl.scheduler` 的候选、观测和动作掩码；
|
|
||||||
- 低层默认使用几何选点；``--local-selector rl`` 仅作为接口预留并记录，
|
|
||||||
当前 HTTP 执行路径仍由高层宏动作终点决定，不把八方向推演当正式收益。
|
|
||||||
|
|
||||||
落盘保证（长训必需）：
|
|
||||||
- **急停可恢复**：``--recovery-every``（默认 2000 环境步）把一份滚动恢复点
|
|
||||||
``<模型目录>/checkpoints/<stem>_latest.zip`` 原子覆盖写；训练开始前先写一份
|
|
||||||
起步基线。因此即使进程被 SIGKILL / ``taskkill /F`` 强杀（没有任何 Python
|
|
||||||
收尾机会），也**最多只丢 ``--recovery-every`` 个环境步**，且磁盘上始终有一份
|
|
||||||
完整、可加载、可 ``--resume`` 的产物；
|
|
||||||
- **原子落盘**：模型 zip、旁挂 json 与 ``<run_dir>/resume.json`` 都是"临时文件 +
|
|
||||||
``os.replace``"，保存途中被杀不会留下半截文件，也不会破坏上一份完整快照；
|
|
||||||
- ``--checkpoint-every`` 每 N 个环境步在 ``<模型目录>/checkpoints/`` 另留一份历史
|
|
||||||
checkpoint（``<stem>_<步数>.zip``），只保留最近 ``--checkpoint-keep`` 个，供
|
|
||||||
返祖对比与多档恢复；
|
|
||||||
- ``<run_dir>/resume.json`` 是恢复的**权威指针**：无论上一刻是滚动恢复点、历史
|
|
||||||
checkpoint、优雅收尾还是异常兜底，它总指向一份可续训产物，并记录增量语义
|
|
||||||
（``timesteps`` 是追加步数）与生效过的 PPO 超参；
|
|
||||||
- ``--control-file`` 是训练前端的**跨平台**控制通道：前端原子写 JSON
|
|
||||||
（``paused`` / ``stop``），本脚本每个 step 轮询。暂停时训练在 step 边界挂起，
|
|
||||||
停止时先保存模型与元数据再退出；
|
|
||||||
- 信号（SIGTERM/SIGINT，Windows 上另有 SIGBREAK）仍可用：直接 kill 本脚本或
|
|
||||||
前端发进程组信号时同样走优雅收尾；subproc worker 在 :func:`_make_env` 里忽略
|
|
||||||
这些信号，保证它们活到主进程 ``vec_env.close()`` 之后再退出；
|
|
||||||
- 任何异常都会先落一份 ``*_emergency.zip`` + ``resume.json`` 再抛出，不再出现
|
|
||||||
"跑了很久什么都没有"。
|
|
||||||
|
|
||||||
为什么暂停/停止不靠信号：Windows 没有 SIGSTOP，且 ``os.kill(pid, SIGTERM)`` 会
|
|
||||||
退化成 ``TerminateProcess`` 硬杀——收尾分支根本不会执行。控制文件两端都只是
|
|
||||||
普通文件 IO，POSIX 与 Windows 行为一致（见 :mod:`mcm26b.process_control`）。
|
|
||||||
|
|
||||||
P1 smoke 示例：
|
|
||||||
python scripts/train_hrl_high.py \
|
|
||||||
--problem 3 --stage 1 --timesteps 2048 --seed 320001 \
|
|
||||||
--envs 1 --vec-env dummy --device mps \
|
|
||||||
--config-profile production --model output/hrl/p1-smoke/q3.zip \
|
|
||||||
--eval-episodes 2 --verbose 0
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import math
|
|
||||||
import os
|
|
||||||
import platform
|
|
||||||
import random
|
|
||||||
import signal
|
|
||||||
import sys
|
|
||||||
import time
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.hrl.curriculum import STAGES
|
|
||||||
from mcm26b.hrl.constants import (ACTION_LAYOUT_VERSION, FEATURE_VERSION,
|
|
||||||
HIGH_OBS_DIM, N_CANDIDATES, REWARD_VERSION)
|
|
||||||
from mcm26b.process_control import ControlState, read_control
|
|
||||||
|
|
||||||
|
|
||||||
def _stage_from_index(index: int):
|
|
||||||
if not 1 <= int(index) <= len(STAGES):
|
|
||||||
raise ValueError(f"stage 必须在 1..{len(STAGES)}")
|
|
||||||
return STAGES[int(index) - 1]
|
|
||||||
|
|
||||||
|
|
||||||
def _profile_fidelity(profile: str) -> str:
|
|
||||||
if profile == "production":
|
|
||||||
return "production"
|
|
||||||
if profile == "coarse-pretrain":
|
|
||||||
return "fast"
|
|
||||||
raise ValueError(f"未知 config-profile: {profile}")
|
|
||||||
|
|
||||||
|
|
||||||
def _reward_from_profile(profile: str):
|
|
||||||
from mcm26b.hrl.hierarchical_env import HRLReward
|
|
||||||
|
|
||||||
if profile == "time":
|
|
||||||
return HRLReward.time_only()
|
|
||||||
if profile == "prompt":
|
|
||||||
return HRLReward()
|
|
||||||
raise ValueError(f"未知 reward-profile: {profile}")
|
|
||||||
|
|
||||||
|
|
||||||
def _stop_signals() -> list[int]:
|
|
||||||
"""本平台可用于"请求收尾"的信号集合。
|
|
||||||
|
|
||||||
Windows 上没有 SIGTERM 的软语义（``os.kill`` 直接 TerminateProcess），但
|
|
||||||
``SIGBREAK`` 可以由 ``CTRL_BREAK_EVENT`` 触发并进入 Python 处理器；POSIX 上
|
|
||||||
SIGTERM/SIGINT 均可捕获。控制文件才是主通道，这里只是让"直接 kill 脚本"
|
|
||||||
也尽量走优雅收尾。
|
|
||||||
"""
|
|
||||||
signals = [signal.SIGTERM, signal.SIGINT]
|
|
||||||
sigbreak = getattr(signal, "SIGBREAK", None)
|
|
||||||
if sigbreak is not None:
|
|
||||||
signals.append(sigbreak)
|
|
||||||
return signals
|
|
||||||
|
|
||||||
|
|
||||||
def _ignore_stop_signals_in_worker() -> None:
|
|
||||||
"""subproc worker 忽略收尾信号，保证主进程能优雅落盘。
|
|
||||||
|
|
||||||
无论信号来自进程组 SIGTERM 还是 Windows ``CTRL_BREAK_EVENT``，若 worker
|
|
||||||
同时退出，主进程会在保存模型之前就拿到断管/EOF 错误，整局训练白跑。worker
|
|
||||||
活在管道后面，主进程收尾时通过 ``vec_env.close()`` 的 ``close`` 指令正常
|
|
||||||
结束它们。
|
|
||||||
"""
|
|
||||||
try:
|
|
||||||
import multiprocessing
|
|
||||||
if multiprocessing.current_process().name == "MainProcess":
|
|
||||||
return
|
|
||||||
for signum in _stop_signals():
|
|
||||||
signal.signal(signum, signal.SIG_IGN)
|
|
||||||
except (ImportError, ValueError, OSError): # 非主线程/平台不支持：不影响训练
|
|
||||||
return
|
|
||||||
|
|
||||||
|
|
||||||
class _ControlPoll:
|
|
||||||
"""控制文件轮询状态机（纯标准库，可脱离 SB3 单测）。
|
|
||||||
|
|
||||||
``poll()`` 每 ``poll_every`` 次调用才真正读盘，避免每个 env step 都做一次
|
|
||||||
文件 IO；``wait_while_paused()`` 阻塞到暂停解除或收到 stop，并累计暂停秒数
|
|
||||||
（``--max-wall-s`` 要把这段扣掉，否则"挂起一小时回来就判超时"）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, path: Path, *, poll_every: int = 32,
|
|
||||||
poll_s: float = 0.2):
|
|
||||||
self.path = Path(path)
|
|
||||||
self.poll_every = max(1, int(poll_every))
|
|
||||||
self.poll_s = max(0.02, float(poll_s))
|
|
||||||
self.paused_total_s = 0.0
|
|
||||||
self._calls = 0
|
|
||||||
self._state = ControlState()
|
|
||||||
|
|
||||||
def poll(self) -> ControlState:
|
|
||||||
"""按抽样间隔返回最近一次控制状态。"""
|
|
||||||
self._calls += 1
|
|
||||||
if self._calls % self.poll_every:
|
|
||||||
return self._state
|
|
||||||
self._state = read_control(self.path)
|
|
||||||
return self._state
|
|
||||||
|
|
||||||
def wait_while_paused(self, on_pause, on_resume) -> bool:
|
|
||||||
"""阻塞到继续或停止；返回 True 表示期间收到了 stop。
|
|
||||||
|
|
||||||
``on_pause`` / ``on_resume`` 用来向前端回显事件（训练脚本的 JSON 行）。
|
|
||||||
"""
|
|
||||||
on_pause()
|
|
||||||
started = time.monotonic()
|
|
||||||
stopped = False
|
|
||||||
while True:
|
|
||||||
time.sleep(self.poll_s)
|
|
||||||
self._state = read_control(self.path)
|
|
||||||
if self._state.stop:
|
|
||||||
stopped = True
|
|
||||||
break
|
|
||||||
if not self._state.paused:
|
|
||||||
break
|
|
||||||
self.paused_total_s += time.monotonic() - started
|
|
||||||
on_resume()
|
|
||||||
return stopped
|
|
||||||
|
|
||||||
|
|
||||||
def _make_env(problem: int, stage_index: int,
|
|
||||||
monitor_path: str | None, profile: str,
|
|
||||||
reward_profile: str):
|
|
||||||
"""顶层工厂：供 DummyVecEnv / SubprocVecEnv 调用（必须是可 pickle 的）。"""
|
|
||||||
from stable_baselines3.common.monitor import Monitor
|
|
||||||
|
|
||||||
from mcm26b.hrl.hierarchical_env import HierarchicalEnv
|
|
||||||
|
|
||||||
_ignore_stop_signals_in_worker()
|
|
||||||
stage = _stage_from_index(stage_index)
|
|
||||||
env = HierarchicalEnv(
|
|
||||||
problem=problem,
|
|
||||||
stage=stage,
|
|
||||||
fidelity=_profile_fidelity(profile),
|
|
||||||
max_episode_steps=300,
|
|
||||||
real_budget_s=1800.0,
|
|
||||||
reward=_reward_from_profile(reward_profile),
|
|
||||||
)
|
|
||||||
return Monitor(env, filename=monitor_path)
|
|
||||||
|
|
||||||
|
|
||||||
def _build_vec_env(args, monitor_paths: list[str]):
|
|
||||||
from stable_baselines3.common.vec_env import (DummyVecEnv, SubprocVecEnv)
|
|
||||||
|
|
||||||
from functools import partial
|
|
||||||
|
|
||||||
factories = [
|
|
||||||
partial(_make_env, args.problem, args.stage,
|
|
||||||
monitor_paths[rank], args.config_profile, args.reward_profile)
|
|
||||||
for rank in range(args.envs)
|
|
||||||
]
|
|
||||||
if args.vec_env == "subproc":
|
|
||||||
# 每个 worker 独立 Monitor 文件；spawn 入口在 __main__ 保护内。
|
|
||||||
return SubprocVecEnv(factories, start_method="spawn")
|
|
||||||
return DummyVecEnv(factories)
|
|
||||||
|
|
||||||
|
|
||||||
def _hash_file(path: Path) -> str:
|
|
||||||
return hashlib.sha256(path.read_bytes()).hexdigest()
|
|
||||||
|
|
||||||
|
|
||||||
def _policy_hyperparams(model) -> dict[str, Any]:
|
|
||||||
"""记录**实际生效**的 PPO 超参。
|
|
||||||
|
|
||||||
``--resume``/``--init`` 时 ``MaskablePPO.load`` 会一并恢复 checkpoint 内的
|
|
||||||
n_steps/batch_size/n_epochs 等，命令行同名参数被忽略；把它们写进元数据，
|
|
||||||
事后才能核对一局到底用什么配置训的。
|
|
||||||
"""
|
|
||||||
|
|
||||||
def _scalar(value):
|
|
||||||
# SB3 2.9 的 learning_rate/clip_range 可能是 Schedule 对象
|
|
||||||
current = getattr(value, "current_value", None)
|
|
||||||
if callable(current):
|
|
||||||
try:
|
|
||||||
value = current()
|
|
||||||
except Exception:
|
|
||||||
return str(value)
|
|
||||||
try:
|
|
||||||
return float(value)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return str(value)
|
|
||||||
|
|
||||||
return {
|
|
||||||
"n_steps": int(getattr(model, "n_steps", 0) or 0),
|
|
||||||
"batch_size": int(getattr(model, "batch_size", 0) or 0),
|
|
||||||
"n_epochs": int(getattr(model, "n_epochs", 0) or 0),
|
|
||||||
"gamma": _scalar(getattr(model, "gamma", 0.0)),
|
|
||||||
"gae_lambda": _scalar(getattr(model, "gae_lambda", 0.0)),
|
|
||||||
"learning_rate": _scalar(getattr(model, "learning_rate", 0.0)),
|
|
||||||
"clip_range": _scalar(getattr(model, "clip_range", 0.0)),
|
|
||||||
"ent_coef": _scalar(getattr(model, "ent_coef", 0.0)),
|
|
||||||
"vf_coef": _scalar(getattr(model, "vf_coef", 0.0)),
|
|
||||||
"max_grad_norm": _scalar(getattr(model, "max_grad_norm", 0.0)),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _progress_payload(model, start: int) -> dict[str, Any]:
|
|
||||||
"""组装 ``--progress-json`` 的单行负载。
|
|
||||||
|
|
||||||
纯函数（只读 model 的 logger / ep_info_buffer），便于脱离训练循环测试；
|
|
||||||
NaN / ±inf 一律折算成 None，保证 ``json.dumps(allow_nan=False)`` 不会炸。
|
|
||||||
"""
|
|
||||||
values = getattr(model.logger, "name_to_value", {}) or {}
|
|
||||||
|
|
||||||
def _num(key, default=None):
|
|
||||||
value = values.get(key)
|
|
||||||
if value is None:
|
|
||||||
return default
|
|
||||||
try:
|
|
||||||
value = float(value)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return default
|
|
||||||
if not math.isfinite(value):
|
|
||||||
return default
|
|
||||||
return value
|
|
||||||
|
|
||||||
ep_rewards: list[float] = []
|
|
||||||
ep_lengths: list[float] = []
|
|
||||||
for item in getattr(model, "ep_info_buffer", []) or []:
|
|
||||||
try:
|
|
||||||
ep_rewards.append(float(item.get("r")))
|
|
||||||
ep_lengths.append(float(item.get("l")))
|
|
||||||
except (TypeError, ValueError, AttributeError):
|
|
||||||
continue
|
|
||||||
return {
|
|
||||||
"event": "progress",
|
|
||||||
"timesteps": int(model.num_timesteps),
|
|
||||||
"start_timesteps": int(start),
|
|
||||||
"n_updates": int(getattr(model, "_n_updates", 0) or 0),
|
|
||||||
"ep_rew_mean": (sum(ep_rewards) / len(ep_rewards)
|
|
||||||
if ep_rewards else None),
|
|
||||||
"ep_len_mean": (sum(ep_lengths) / len(ep_lengths)
|
|
||||||
if ep_lengths else None),
|
|
||||||
"value_loss": _num("train/value_loss"),
|
|
||||||
"policy_gradient_loss": _num("train/policy_gradient_loss"),
|
|
||||||
"approx_kl": _num("train/approx_kl"),
|
|
||||||
"entropy_loss": _num("train/entropy_loss"),
|
|
||||||
"explained_variance": _num("train/explained_variance"),
|
|
||||||
"fps": _num("time/fps"),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _model_metadata(*, args, wall_s: float, n_envs: int,
|
|
||||||
actual_timesteps: int) -> dict[str, Any]:
|
|
||||||
import numpy
|
|
||||||
import stable_baselines3
|
|
||||||
import sb3_contrib
|
|
||||||
import torch
|
|
||||||
|
|
||||||
stage = _stage_from_index(args.stage)
|
|
||||||
return {
|
|
||||||
"level": "high",
|
|
||||||
"problem": int(args.problem),
|
|
||||||
"stage": int(args.stage),
|
|
||||||
"stage_name": stage.name,
|
|
||||||
"algorithm": "maskable_ppo",
|
|
||||||
"feature_version": FEATURE_VERSION,
|
|
||||||
"action_layout_version": ACTION_LAYOUT_VERSION,
|
|
||||||
"reward_version": REWARD_VERSION,
|
|
||||||
"obs_dim": HIGH_OBS_DIM,
|
|
||||||
"act_dim": N_CANDIDATES,
|
|
||||||
"seed": int(args.seed),
|
|
||||||
"requested_timesteps": int(args.timesteps),
|
|
||||||
"timesteps_semantics": ("additional" if (args.resume or args.init)
|
|
||||||
else "total"),
|
|
||||||
"actual_timesteps": int(actual_timesteps),
|
|
||||||
"n_envs": int(n_envs),
|
|
||||||
"vec_env": args.vec_env,
|
|
||||||
"config_profile": args.config_profile,
|
|
||||||
"fidelity": _profile_fidelity(args.config_profile),
|
|
||||||
"reward_profile": args.reward_profile,
|
|
||||||
"local_selector": args.local_selector,
|
|
||||||
"local_model": args.local_model,
|
|
||||||
"checkpoint_every": int(getattr(args, "checkpoint_every", 0) or 0),
|
|
||||||
"checkpoint_keep": int(getattr(args, "checkpoint_keep", 0) or 0),
|
|
||||||
"checkpoint_dir": getattr(args, "checkpoint_dir", None),
|
|
||||||
"recovery_every": int(getattr(args, "recovery_every", 0) or 0),
|
|
||||||
"device": args.device,
|
|
||||||
"wall_seconds": round(wall_s, 3),
|
|
||||||
"max_wall_s": float(args.max_wall_s),
|
|
||||||
"created_unix_s": time.time(),
|
|
||||||
"python": sys.version.split()[0],
|
|
||||||
"platform": platform.platform(),
|
|
||||||
"torch": torch.__version__,
|
|
||||||
"numpy": numpy.__version__,
|
|
||||||
"stable_baselines3": stable_baselines3.__version__,
|
|
||||||
"sb3_contrib": sb3_contrib.__version__,
|
|
||||||
"mps_available": bool(getattr(getattr(torch.backends, "mps", None),
|
|
||||||
"is_available", lambda: False)()),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _write_json_atomic(path: Path, payload: dict[str, Any]) -> None:
|
|
||||||
"""原子写 JSON：同目录 ``.tmp-`` 前缀临时文件 + ``os.replace``。
|
|
||||||
|
|
||||||
急停（SIGKILL / ``taskkill /F``）会绕过所有 Python 清理逻辑，保存途中被杀
|
|
||||||
是完全可能的。原子替换保证**真实文件名下要么是旧的完整文件、要么是新的
|
|
||||||
完整文件**，不会出现半截 JSON 让续训读到一半就崩。
|
|
||||||
"""
|
|
||||||
path = Path(path)
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
tmp = path.with_name(f".tmp-{path.name}")
|
|
||||||
tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2),
|
|
||||||
encoding="utf-8")
|
|
||||||
os.replace(tmp, path)
|
|
||||||
|
|
||||||
|
|
||||||
def _save_model_atomic(model, target: Path) -> Path:
|
|
||||||
"""把 SB3 模型原子地写到 ``target``，返回实际落盘路径。
|
|
||||||
|
|
||||||
先写 ``.tmp-<name>``（同样以 ``.zip`` 结尾，SB3 不会再加后缀），完整写完后
|
|
||||||
``os.replace`` 覆盖目标。于是：
|
|
||||||
|
|
||||||
- 保存途中被强杀 → 目标路径仍是上一份**完整**的快照，可继续复训；
|
|
||||||
- 保存成功 → 目标路径立即变成新快照，不存在"看起来存在但打不开"的窗口。
|
|
||||||
|
|
||||||
临时名带 ``.tmp-`` 前缀，因此不会被 ``*_ckpt_*.zip`` 的清理逻辑误当成正式
|
|
||||||
checkpoint。
|
|
||||||
"""
|
|
||||||
target = Path(target)
|
|
||||||
target.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
tmp = target.with_name(f".tmp-{target.name}")
|
|
||||||
model.save(str(tmp))
|
|
||||||
if not tmp.exists(): # SB3 对未知后缀可能自行补 .zip
|
|
||||||
candidate = tmp.with_name(tmp.name + ".zip")
|
|
||||||
if candidate.exists():
|
|
||||||
tmp = candidate
|
|
||||||
os.replace(tmp, target)
|
|
||||||
return target
|
|
||||||
|
|
||||||
|
|
||||||
def _write_resume_hint(*, run_dir: Path, model_path: Path, timesteps: int,
|
|
||||||
reason: str, model=None) -> dict[str, Any]:
|
|
||||||
"""把"从哪继续"写进 ``<run_dir>/resume.json``。
|
|
||||||
|
|
||||||
这是急停恢复的**权威指针**：无论上一刻是周期 checkpoint、滚动恢复点、
|
|
||||||
优雅收尾还是异常兜底，这个文件总指向一份完整、可 ``--resume`` 的产物，
|
|
||||||
并带上续训语义（``timesteps`` 是增量步数）与生效过的 PPO 超参。
|
|
||||||
"""
|
|
||||||
hint: dict[str, Any] = {
|
|
||||||
"event": "resume_hint",
|
|
||||||
"model": str(model_path),
|
|
||||||
"timesteps": int(timesteps),
|
|
||||||
"timesteps_semantics": "additional",
|
|
||||||
"reason": str(reason),
|
|
||||||
"created_unix_s": time.time(),
|
|
||||||
"resume_args": ["--resume", "--model", str(model_path)],
|
|
||||||
}
|
|
||||||
if model is not None:
|
|
||||||
hint["ppo_hyperparams"] = _policy_hyperparams(model)
|
|
||||||
_write_json_atomic(run_dir / "resume.json", hint)
|
|
||||||
return hint
|
|
||||||
|
|
||||||
|
|
||||||
def _write_metadata(*, model_path: Path, args, wall_s: float, n_envs: int,
|
|
||||||
actual_timesteps: int, monitor_paths: list[str],
|
|
||||||
model=None,
|
|
||||||
extra: dict[str, Any] | None = None) -> dict[str, Any]:
|
|
||||||
"""写模型旁挂元数据（含 sha256），训练前端/审计都读它。"""
|
|
||||||
metadata = _model_metadata(args=args, wall_s=wall_s, n_envs=n_envs,
|
|
||||||
actual_timesteps=actual_timesteps)
|
|
||||||
if model is not None:
|
|
||||||
metadata["ppo_hyperparams"] = _policy_hyperparams(model)
|
|
||||||
metadata["model_sha256"] = _hash_file(model_path)
|
|
||||||
metadata["monitor_files"] = list(monitor_paths)
|
|
||||||
metadata["reward_note"] = (
|
|
||||||
"reward_profile=time 使用 HRLReward.time_only（P0.4 纯时间口径）；"
|
|
||||||
"reward_profile=prompt 使用原始 §5.2 塑形。模型元数据已固化口径。")
|
|
||||||
if extra:
|
|
||||||
metadata.update(extra)
|
|
||||||
_write_json_atomic(model_path.with_suffix(".json"), metadata)
|
|
||||||
return metadata
|
|
||||||
|
|
||||||
|
|
||||||
def _prune_checkpoints(directory: Path, stem: str, keep: int) -> list[str]:
|
|
||||||
"""只保留最近 ``keep`` 个 checkpoint（zip+json 同删），返回被删的 zip 路径。
|
|
||||||
|
|
||||||
只匹配 ``<stem>_<步数>.zip``；滚动恢复点 ``<stem>_latest.zip`` 与临时文件
|
|
||||||
``.tmp-*.zip`` 都不匹配，因此永远不会被历史清理误删。
|
|
||||||
"""
|
|
||||||
files = sorted((path for path in directory.glob(f"{stem}_[0-9]*.zip")
|
|
||||||
if path.stem.rsplit("_", 1)[-1].isdigit()),
|
|
||||||
key=lambda path: path.stat().st_mtime, reverse=True)
|
|
||||||
removed: list[str] = []
|
|
||||||
for stale in files[max(1, int(keep)):]:
|
|
||||||
for candidate in (stale, stale.with_suffix(".json")):
|
|
||||||
if candidate.exists():
|
|
||||||
candidate.unlink()
|
|
||||||
removed.append(str(stale))
|
|
||||||
return removed
|
|
||||||
|
|
||||||
|
|
||||||
def _evaluate_env(model, *, problem: int, stage_index: int,
|
|
||||||
episodes: int, seed: int, profile: str,
|
|
||||||
reward_profile: str) -> dict[str, Any]:
|
|
||||||
from mcm26b.hrl.hierarchical_env import HierarchicalEnv
|
|
||||||
|
|
||||||
stage = _stage_from_index(stage_index)
|
|
||||||
env = HierarchicalEnv(problem=problem, stage=stage,
|
|
||||||
fidelity=_profile_fidelity(profile),
|
|
||||||
max_episode_steps=300, real_budget_s=1800.0,
|
|
||||||
reward=_reward_from_profile(reward_profile))
|
|
||||||
wins = 0
|
|
||||||
rows = []
|
|
||||||
if episodes <= 0:
|
|
||||||
env.close()
|
|
||||||
return {"episodes": 0, "successes": 0, "success_rate": None, "rows": []}
|
|
||||||
for episode in range(episodes):
|
|
||||||
obs, info = env.reset(seed=seed + episode)
|
|
||||||
done = False
|
|
||||||
steps = 0
|
|
||||||
while not done:
|
|
||||||
action, _state = model.predict(
|
|
||||||
obs, action_masks=env.action_masks(), deterministic=True)
|
|
||||||
obs, _, terminated, truncated, info = env.step(int(action))
|
|
||||||
done = bool(terminated or truncated)
|
|
||||||
steps += 1
|
|
||||||
# 课程阶段的小场景只需证明全部频道 accounted；完整阶段至少应清掉
|
|
||||||
# 场景最少的题面源数。这里不读取真值，只使用环境自身统计。
|
|
||||||
success = bool(
|
|
||||||
info.get("accounted", 0) >= 20
|
|
||||||
and info.get("cleared", 0) >= stage.sources_min
|
|
||||||
and not info.get("anomalies")
|
|
||||||
)
|
|
||||||
wins += int(success)
|
|
||||||
rows.append({
|
|
||||||
"episode": episode,
|
|
||||||
"seed": seed + episode,
|
|
||||||
"steps": steps,
|
|
||||||
"cleared": info.get("cleared", 0),
|
|
||||||
"accounted": info.get("accounted", 0),
|
|
||||||
"virtual_time_s": info.get("episode_virtual_cost_s"),
|
|
||||||
"stop_reason": info.get("stop_reason"),
|
|
||||||
"anomalies": info.get("anomalies"),
|
|
||||||
"success": success,
|
|
||||||
})
|
|
||||||
env.close()
|
|
||||||
return {
|
|
||||||
"episodes": len(rows),
|
|
||||||
"successes": wins,
|
|
||||||
"success_rate": wins / len(rows) if rows else 0.0,
|
|
||||||
"rows": rows,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _evaluate_manifest(model_path: Path, manifest_path: str, *,
|
|
||||||
limit: int) -> dict[str, Any]:
|
|
||||||
"""可选：用 benchmark_hrl 的同一套真值隔离评测在固定 manifest 上跑已训模型。"""
|
|
||||||
scripts_dir = Path(__file__).resolve().parent
|
|
||||||
if str(scripts_dir) not in sys.path:
|
|
||||||
sys.path.insert(0, str(scripts_dir))
|
|
||||||
import benchmark_hrl as bench # scripts/benchmark_hrl.py
|
|
||||||
|
|
||||||
manifest = json.loads(Path(manifest_path).read_text(encoding="utf-8"))
|
|
||||||
scenes = bench.scenes_from_manifest(manifest)
|
|
||||||
if limit > 0:
|
|
||||||
scenes = scenes[:limit]
|
|
||||||
rows = []
|
|
||||||
for scene in scenes:
|
|
||||||
run_dir = Path(model_path).parent / "eval-manifest" / scene.scene_id
|
|
||||||
result = bench.run_scene_variant(
|
|
||||||
scene, "hrl", config=ProblemConfig(),
|
|
||||||
hrl_model=str(model_path), run_dir=run_dir)
|
|
||||||
rows.append(result["row"])
|
|
||||||
aggregate = bench._aggregate(rows)
|
|
||||||
return aggregate
|
|
||||||
|
|
||||||
|
|
||||||
def run_training(args) -> int:
|
|
||||||
import numpy as np
|
|
||||||
import torch
|
|
||||||
from sb3_contrib import MaskablePPO
|
|
||||||
from stable_baselines3.common.callbacks import BaseCallback
|
|
||||||
from stable_baselines3.common.utils import set_random_seed
|
|
||||||
|
|
||||||
torch.set_num_threads(1)
|
|
||||||
set_random_seed(args.seed)
|
|
||||||
random.seed(args.seed)
|
|
||||||
np.random.seed(args.seed)
|
|
||||||
if torch.cuda.is_available():
|
|
||||||
torch.cuda.manual_seed_all(args.seed)
|
|
||||||
|
|
||||||
model_path = Path(args.model)
|
|
||||||
if model_path.exists() and not (args.resume or args.force):
|
|
||||||
print(f"模型已存在：{model_path}；如需覆盖请加 --force，续训请加 --resume。",
|
|
||||||
file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
model_path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
run_dir = model_path.parent
|
|
||||||
monitor_paths = [
|
|
||||||
str(run_dir / f"high_curve_{model_path.stem}_env{rank}.monitor.csv")
|
|
||||||
for rank in range(args.envs)
|
|
||||||
]
|
|
||||||
for path in monitor_paths:
|
|
||||||
Path(path).parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
|
|
||||||
vec_env = _build_vec_env(args, monitor_paths)
|
|
||||||
|
|
||||||
if args.resume:
|
|
||||||
if not model_path.exists():
|
|
||||||
print(f"--resume 指定的模型不存在：{model_path}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
model = MaskablePPO.load(str(model_path), env=vec_env,
|
|
||||||
device=args.device,
|
|
||||||
print_system_info=False)
|
|
||||||
elif args.init:
|
|
||||||
init_path = Path(args.init)
|
|
||||||
if not init_path.exists():
|
|
||||||
print(f"--init 指定的模型不存在：{init_path}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
model = MaskablePPO.load(str(init_path), env=vec_env,
|
|
||||||
device=args.device,
|
|
||||||
print_system_info=False)
|
|
||||||
else:
|
|
||||||
model = MaskablePPO(
|
|
||||||
"MlpPolicy",
|
|
||||||
vec_env,
|
|
||||||
policy_kwargs={"net_arch": [128, 128]},
|
|
||||||
n_steps=args.rollout_steps,
|
|
||||||
batch_size=args.batch_size,
|
|
||||||
n_epochs=args.n_epochs,
|
|
||||||
gamma=1.0,
|
|
||||||
gae_lambda=0.95,
|
|
||||||
learning_rate=3e-4,
|
|
||||||
clip_range=0.2,
|
|
||||||
ent_coef=0.005,
|
|
||||||
vf_coef=0.5,
|
|
||||||
max_grad_norm=0.5,
|
|
||||||
seed=args.seed,
|
|
||||||
device=args.device,
|
|
||||||
verbose=args.verbose,
|
|
||||||
)
|
|
||||||
if args.init:
|
|
||||||
pass # 已在上面处理；此分支仅保持语义清晰
|
|
||||||
|
|
||||||
# resume/init 会恢复 checkpoint 内的 PPO 超参，命令行同名三项目前被忽略；
|
|
||||||
# 显式说出差异，避免"以为换了 n_steps 其实没换"。
|
|
||||||
if args.resume or args.init:
|
|
||||||
effective = _policy_hyperparams(model)
|
|
||||||
requested = {"n_steps": int(args.rollout_steps),
|
|
||||||
"batch_size": int(args.batch_size),
|
|
||||||
"n_epochs": int(args.n_epochs)}
|
|
||||||
if {key: effective[key] for key in requested} != requested:
|
|
||||||
print(json.dumps({
|
|
||||||
"event": "hyperparams_from_checkpoint",
|
|
||||||
"effective": {key: effective[key] for key in requested},
|
|
||||||
"requested": requested,
|
|
||||||
"note": ("resume/init 沿用 checkpoint 内的 PPO 超参，"
|
|
||||||
"命令行 rollout-steps/batch-size/n-epochs 被忽略")},
|
|
||||||
ensure_ascii=False), file=sys.stderr, flush=True)
|
|
||||||
|
|
||||||
# ------------------------------------------------------ 优雅暂停 / 停止
|
|
||||||
# 主通道是控制文件（跨平台）；信号是兜底：直接 kill 本进程时也走同一段收尾。
|
|
||||||
stop_state: dict[str, Any] = {"requested": False, "signal": None}
|
|
||||||
previous_handlers: dict[int, Any] = {}
|
|
||||||
|
|
||||||
def _request_stop(source: str) -> None:
|
|
||||||
if stop_state["requested"]:
|
|
||||||
return
|
|
||||||
stop_state["requested"] = True
|
|
||||||
stop_state["signal"] = source
|
|
||||||
print(json.dumps({"event": "stop_requested", "signal": source}), flush=True)
|
|
||||||
|
|
||||||
def _on_stop_signal(signum, _frame):
|
|
||||||
_request_stop(signal.Signals(signum).name)
|
|
||||||
|
|
||||||
for _signum in _stop_signals():
|
|
||||||
try:
|
|
||||||
previous_handlers[_signum] = signal.signal(_signum, _on_stop_signal)
|
|
||||||
except (ValueError, OSError): # 非主线程/平台不支持：不阻塞训练
|
|
||||||
pass
|
|
||||||
|
|
||||||
class _ControlChannel(BaseCallback):
|
|
||||||
"""把 :class:`_ControlPoll` 接到 SB3 的 step 回调上。
|
|
||||||
|
|
||||||
为什么不用信号：Windows 没有 SIGSTOP，且 ``os.kill(SIGTERM)`` 会退化成
|
|
||||||
``TerminateProcess`` 硬杀，收尾分支根本不会执行。控制文件在 POSIX 与
|
|
||||||
Windows 上都只是普通文件 IO，语义完全一致。
|
|
||||||
|
|
||||||
暂停是**协作式**的：在 ``_on_step`` 内阻塞轮询，训练因此停在 step 边界，
|
|
||||||
进程仍存活、显存仍占用，但不消耗算力；收到 stop 立刻转入收尾落盘。
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, path: Path, *, poll_every: int = 32,
|
|
||||||
poll_s: float = 0.2):
|
|
||||||
super().__init__()
|
|
||||||
self.poller = _ControlPoll(path, poll_every=poll_every,
|
|
||||||
poll_s=poll_s)
|
|
||||||
|
|
||||||
def paused_seconds(self) -> float:
|
|
||||||
return self.poller.paused_total_s
|
|
||||||
|
|
||||||
def _event(self, name: str) -> None:
|
|
||||||
print(json.dumps({"event": name,
|
|
||||||
"timesteps": int(self.model.num_timesteps)}),
|
|
||||||
flush=True)
|
|
||||||
|
|
||||||
def _on_step(self) -> bool:
|
|
||||||
state = self.poller.poll()
|
|
||||||
if state.stop:
|
|
||||||
_request_stop("control_file")
|
|
||||||
return False
|
|
||||||
if state.paused:
|
|
||||||
stopped = self.poller.wait_while_paused(
|
|
||||||
lambda: self._event("paused"),
|
|
||||||
lambda: self._event("resumed"))
|
|
||||||
if stopped:
|
|
||||||
_request_stop("control_file")
|
|
||||||
return False
|
|
||||||
return not stop_state["requested"]
|
|
||||||
return True
|
|
||||||
|
|
||||||
control_path = getattr(args, "control_file", None)
|
|
||||||
control_cb = _ControlChannel(Path(control_path)) if control_path else None
|
|
||||||
|
|
||||||
class _GracefulStop(BaseCallback):
|
|
||||||
"""收到停止请求后在最近一次 step 边界结束 rollout，让模型能落盘。"""
|
|
||||||
|
|
||||||
def _on_step(self) -> bool:
|
|
||||||
if stop_state["requested"]:
|
|
||||||
print(json.dumps({"event": "graceful_stop",
|
|
||||||
"signal": stop_state["signal"],
|
|
||||||
"timesteps": int(self.model.num_timesteps)}),
|
|
||||||
flush=True)
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
class _WallClockStop(BaseCallback):
|
|
||||||
"""墙钟硬上限；暂停时间不计入，避免"挂起一小时回来就判超时"。"""
|
|
||||||
|
|
||||||
def __init__(self, max_wall_s: float, paused_clock=None):
|
|
||||||
super().__init__()
|
|
||||||
self.max_wall_s = max_wall_s
|
|
||||||
self.start = time.monotonic()
|
|
||||||
self.paused_clock = paused_clock
|
|
||||||
|
|
||||||
def paused_seconds(self) -> float:
|
|
||||||
return float(self.paused_clock()) if self.paused_clock else 0.0
|
|
||||||
|
|
||||||
def _on_step(self) -> bool:
|
|
||||||
elapsed = time.monotonic() - self.start - self.paused_seconds()
|
|
||||||
if elapsed >= self.max_wall_s:
|
|
||||||
print(json.dumps({"event": "wall_clock_stop",
|
|
||||||
"elapsed_s": elapsed}))
|
|
||||||
return False
|
|
||||||
return True
|
|
||||||
|
|
||||||
class _ProgressJSON(BaseCallback):
|
|
||||||
"""向训练前端输出单行 JSON 进度；不影响 SB3 自身 logger。
|
|
||||||
|
|
||||||
前端首个 curve 点越早出现越好，因此开头几步按更小间隔发送（前 8 个
|
|
||||||
进度点逐步放大到 ``every_steps``），之后稳定在 ``every_steps``。
|
|
||||||
这样短程 smoke 也能看到完整曲线，长训不会刷屏。
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, every_steps: int):
|
|
||||||
super().__init__()
|
|
||||||
self.every_steps = max(1, int(every_steps))
|
|
||||||
# init/resume 时计数器从上一阶段继续，前端要用增量算进度
|
|
||||||
self.start = int(model.num_timesteps)
|
|
||||||
self.last_emit = self.start
|
|
||||||
self.emitted = 0
|
|
||||||
|
|
||||||
def _next_gap(self) -> int:
|
|
||||||
warmup = max(0.2, 1.0 - self.emitted / 8.0)
|
|
||||||
return max(1, min(self.every_steps, int(round(self.every_steps * warmup))))
|
|
||||||
|
|
||||||
def _on_step(self) -> bool:
|
|
||||||
current = int(self.model.num_timesteps)
|
|
||||||
if current - self.last_emit < self._next_gap():
|
|
||||||
return True
|
|
||||||
self.last_emit = current
|
|
||||||
self.emitted += 1
|
|
||||||
payload = _progress_payload(self.model, self.start)
|
|
||||||
print(json.dumps(payload, ensure_ascii=False, allow_nan=False),
|
|
||||||
flush=True)
|
|
||||||
return True
|
|
||||||
|
|
||||||
started = time.monotonic()
|
|
||||||
checkpoints_dir = (Path(args.checkpoint_dir) if args.checkpoint_dir
|
|
||||||
else run_dir / "checkpoints")
|
|
||||||
checkpoint_stem = f"{model_path.stem}_ckpt"
|
|
||||||
|
|
||||||
class _CheckpointJSON(BaseCallback):
|
|
||||||
"""可续训落盘：滚动恢复点 + 周期 checkpoint + resume.json 指针。
|
|
||||||
|
|
||||||
- **滚动恢复点** ``<stem>_latest.zip``：每 ``recovery_every`` 个环境步
|
|
||||||
原子覆盖写一次，训练开始前也先写一份。它保证"急停"（SIGKILL /
|
|
||||||
``taskkill /F``，没有任何 Python 清理机会）之后**最多只丢
|
|
||||||
``recovery_every`` 个环境步**，而且文件始终完整可加载。
|
|
||||||
- **周期 checkpoint** ``<stem>_<步数>.zip``：每 ``every_steps`` 步留一份，
|
|
||||||
只保留最近 ``keep`` 个，用于返祖对比与多档恢复。
|
|
||||||
- 每次落盘都刷新 ``<run_dir>/resume.json``：无论怎么停，都有一份权威的
|
|
||||||
"从哪继续"指针（含增量语义与生效过的 PPO 超参）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, every_steps: int, keep: int, recovery_every: int):
|
|
||||||
super().__init__()
|
|
||||||
self.every_steps = max(0, int(every_steps))
|
|
||||||
self.keep = max(1, int(keep))
|
|
||||||
self.recovery_every = max(0, int(recovery_every))
|
|
||||||
self.last_saved = int(model.num_timesteps)
|
|
||||||
self.last_recovery = int(model.num_timesteps)
|
|
||||||
self.saved: list[str] = []
|
|
||||||
self.latest_path = checkpoints_dir / f"{checkpoint_stem}_latest.zip"
|
|
||||||
|
|
||||||
def write_recovery(self, reason: str) -> None:
|
|
||||||
"""立刻写一份滚动恢复点；训练开始前调用即"起步基线"。
|
|
||||||
|
|
||||||
用闭包里的 ``model`` 而不是 ``self.model``：``BaseCallback.model``
|
|
||||||
要等 ``learn()`` 时才被赋值，而起步基线恰恰在 ``learn()`` 之前写。
|
|
||||||
"""
|
|
||||||
current = int(model.num_timesteps)
|
|
||||||
self.last_recovery = current
|
|
||||||
self._write_recovery(current, reason)
|
|
||||||
|
|
||||||
def _on_step(self) -> bool:
|
|
||||||
current = int(model.num_timesteps)
|
|
||||||
if (self.recovery_every > 0
|
|
||||||
and current - self.last_recovery >= self.recovery_every):
|
|
||||||
self.last_recovery = current
|
|
||||||
self._write_recovery(current, "interval")
|
|
||||||
if (self.every_steps > 0
|
|
||||||
and current - self.last_saved >= self.every_steps):
|
|
||||||
self.last_saved = current
|
|
||||||
self._write_interval(current)
|
|
||||||
return True
|
|
||||||
|
|
||||||
def _persist(self, target: Path, current: int, reason: str,
|
|
||||||
extra: dict[str, Any]) -> None:
|
|
||||||
"""原子落盘 zip + 元数据，并刷新 resume.json 指针。"""
|
|
||||||
_save_model_atomic(model, target)
|
|
||||||
_write_metadata(
|
|
||||||
model_path=target, args=args,
|
|
||||||
wall_s=time.monotonic() - started, n_envs=args.envs,
|
|
||||||
actual_timesteps=current, monitor_paths=monitor_paths,
|
|
||||||
model=model,
|
|
||||||
extra={"checkpoint_timesteps": current,
|
|
||||||
"checkpoint_reason": reason,
|
|
||||||
"checkpoint_of": str(model_path), **extra})
|
|
||||||
_write_resume_hint(run_dir=run_dir, model_path=target,
|
|
||||||
timesteps=current, reason=reason,
|
|
||||||
model=model)
|
|
||||||
|
|
||||||
def _write_recovery(self, current: int, reason: str) -> None:
|
|
||||||
checkpoints_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
self._persist(self.latest_path, current, reason,
|
|
||||||
{"recovery": True})
|
|
||||||
print(json.dumps({"event": "recovery_checkpoint",
|
|
||||||
"reason": reason, "timesteps": current,
|
|
||||||
"model": str(self.latest_path)},
|
|
||||||
ensure_ascii=False), flush=True)
|
|
||||||
|
|
||||||
def _write_interval(self, current: int) -> None:
|
|
||||||
checkpoints_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
target = checkpoints_dir / f"{checkpoint_stem}_{current:09d}.zip"
|
|
||||||
self._persist(target, current, "interval", {"checkpoint": True})
|
|
||||||
removed = _prune_checkpoints(checkpoints_dir, checkpoint_stem,
|
|
||||||
self.keep)
|
|
||||||
self.saved.append(str(target))
|
|
||||||
print(json.dumps({"event": "checkpoint", "timesteps": current,
|
|
||||||
"model": str(target), "pruned": removed},
|
|
||||||
ensure_ascii=False), flush=True)
|
|
||||||
|
|
||||||
progress_every = (args.progress_every if args.progress_every > 0
|
|
||||||
else args.rollout_steps * args.envs)
|
|
||||||
checkpoint_cb = _CheckpointJSON(args.checkpoint_every, args.checkpoint_keep,
|
|
||||||
args.recovery_every)
|
|
||||||
if args.recovery_every > 0:
|
|
||||||
# 起步基线：训练还没跑就急停，也至少有一份可加载、可续训的产物
|
|
||||||
checkpoint_cb.write_recovery("baseline")
|
|
||||||
# 顺序有语义：控制通道最先（暂停/停止要在本步立刻生效），进度次之，
|
|
||||||
# 墙钟上限要能读到控制通道累计的暂停秒数，checkpoint 放最后。
|
|
||||||
callbacks: list[Any] = []
|
|
||||||
if control_cb is not None:
|
|
||||||
callbacks.append(control_cb)
|
|
||||||
if args.progress_json:
|
|
||||||
callbacks.append(_ProgressJSON(progress_every))
|
|
||||||
callbacks.append(_GracefulStop())
|
|
||||||
callbacks.append(_WallClockStop(
|
|
||||||
args.max_wall_s,
|
|
||||||
paused_clock=(control_cb.paused_seconds if control_cb is not None
|
|
||||||
else None)))
|
|
||||||
callbacks.append(checkpoint_cb)
|
|
||||||
|
|
||||||
actual_timesteps = int(model.num_timesteps)
|
|
||||||
try:
|
|
||||||
model.learn(total_timesteps=args.timesteps,
|
|
||||||
callback=callbacks if len(callbacks) > 1 else callbacks[0],
|
|
||||||
progress_bar=False,
|
|
||||||
reset_num_timesteps=not (args.resume or args.init))
|
|
||||||
except BaseException as exc: # 训练失败也必须留下可续训产物，不静默
|
|
||||||
failure = {
|
|
||||||
"event": "training_failed",
|
|
||||||
"error": f"{type(exc).__name__}: {exc}",
|
|
||||||
"model": str(model_path),
|
|
||||||
"timesteps": int(model.num_timesteps),
|
|
||||||
}
|
|
||||||
emergency = run_dir / f"{model_path.stem}_emergency.zip"
|
|
||||||
try:
|
|
||||||
_save_model_atomic(model, emergency)
|
|
||||||
_write_metadata(model_path=emergency, args=args,
|
|
||||||
wall_s=time.monotonic() - started,
|
|
||||||
n_envs=args.envs,
|
|
||||||
actual_timesteps=int(model.num_timesteps),
|
|
||||||
monitor_paths=monitor_paths, model=model,
|
|
||||||
extra={"emergency": True,
|
|
||||||
"error": failure["error"]})
|
|
||||||
_write_resume_hint(run_dir=run_dir, model_path=emergency,
|
|
||||||
timesteps=int(model.num_timesteps),
|
|
||||||
reason="emergency", model=model)
|
|
||||||
failure["emergency_model"] = str(emergency)
|
|
||||||
except Exception as save_exc: # 落盘失败不能再掩盖原始异常
|
|
||||||
failure["emergency_save_error"] = (
|
|
||||||
f"{type(save_exc).__name__}: {save_exc}")
|
|
||||||
_write_json_atomic(run_dir / "training_failure.json", failure)
|
|
||||||
print(json.dumps(failure, ensure_ascii=False), file=sys.stderr,
|
|
||||||
flush=True)
|
|
||||||
raise
|
|
||||||
wall_s = time.monotonic() - started
|
|
||||||
actual_timesteps = int(model.num_timesteps)
|
|
||||||
stopped_by_signal = bool(stop_state["requested"])
|
|
||||||
_save_model_atomic(model, model_path)
|
|
||||||
metadata = _write_metadata(
|
|
||||||
model_path=model_path, args=args, wall_s=wall_s, n_envs=args.envs,
|
|
||||||
actual_timesteps=actual_timesteps, monitor_paths=monitor_paths,
|
|
||||||
model=model,
|
|
||||||
extra={"stopped_by_signal": stopped_by_signal,
|
|
||||||
"stop_signal": stop_state["signal"],
|
|
||||||
"checkpoint_files": checkpoint_cb.saved,
|
|
||||||
"recovery_model": str(checkpoint_cb.latest_path)})
|
|
||||||
# 最终模型也刷新一次 resume.json：优雅收尾后"从哪继续"指向最终模型
|
|
||||||
_write_resume_hint(run_dir=run_dir, model_path=model_path,
|
|
||||||
timesteps=actual_timesteps,
|
|
||||||
reason=("stopped" if stopped_by_signal else "finished"),
|
|
||||||
model=model)
|
|
||||||
metadata_path = model_path.with_suffix(".json")
|
|
||||||
vec_env.close()
|
|
||||||
# 收尾阶段恢复默认处理器：再次 Ctrl+C / 停止可立即强杀，不再等待评估。
|
|
||||||
for _signum, _handler in previous_handlers.items():
|
|
||||||
try:
|
|
||||||
signal.signal(_signum, _handler)
|
|
||||||
except (ValueError, OSError):
|
|
||||||
pass
|
|
||||||
if stopped_by_signal:
|
|
||||||
print(json.dumps({"event": "saved_after_stop",
|
|
||||||
"model": str(model_path),
|
|
||||||
"timesteps": actual_timesteps}), flush=True)
|
|
||||||
|
|
||||||
evaluation = _evaluate_env(
|
|
||||||
model, problem=args.problem, stage_index=args.stage,
|
|
||||||
episodes=args.eval_episodes, seed=args.seed + 1_000_000,
|
|
||||||
profile=args.config_profile, reward_profile=args.reward_profile)
|
|
||||||
manifest_eval = None
|
|
||||||
if args.eval_manifest:
|
|
||||||
try:
|
|
||||||
manifest_eval = _evaluate_manifest(
|
|
||||||
model_path, args.eval_manifest, limit=args.eval_manifest_limit)
|
|
||||||
except Exception as exc:
|
|
||||||
manifest_eval = {"error": f"{type(exc).__name__}: {exc}"}
|
|
||||||
metadata["evaluation"] = evaluation
|
|
||||||
metadata["eval_manifest"] = manifest_eval
|
|
||||||
metadata_path.write_text(json.dumps(metadata, ensure_ascii=False, indent=2),
|
|
||||||
encoding="utf-8")
|
|
||||||
print(json.dumps(metadata, ensure_ascii=False, indent=2))
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("--problem", type=int, choices=(3, 4), required=True)
|
|
||||||
parser.add_argument("--stage", type=int, choices=(1, 2, 3, 4),
|
|
||||||
default=4)
|
|
||||||
parser.add_argument("--timesteps", type=int, required=True)
|
|
||||||
parser.add_argument("--seed", type=int, default=320001)
|
|
||||||
parser.add_argument("--envs", type=int, default=1)
|
|
||||||
parser.add_argument("--vec-env", choices=("dummy", "subproc"), default="dummy")
|
|
||||||
parser.add_argument("--device", choices=("cpu", "cuda", "mps", "auto"),
|
|
||||||
default="auto")
|
|
||||||
parser.add_argument("--config-profile",
|
|
||||||
choices=("production", "coarse-pretrain"),
|
|
||||||
default="production")
|
|
||||||
parser.add_argument("--model", required=True)
|
|
||||||
parser.add_argument("--resume", action="store_true",
|
|
||||||
help="从 --model 续训（沿用 checkpoint 内的 PPO 超参）")
|
|
||||||
parser.add_argument("--init", default=None,
|
|
||||||
help="可选：Q3→Q4 或继续预训练权重路径")
|
|
||||||
parser.add_argument("--force", action="store_true")
|
|
||||||
parser.add_argument("--eval-manifest", default=None)
|
|
||||||
parser.add_argument("--eval-manifest-limit", type=int, default=0)
|
|
||||||
parser.add_argument("--eval-episodes", type=int, default=5)
|
|
||||||
parser.add_argument("--local-selector", choices=("geometry", "rl"),
|
|
||||||
default="geometry")
|
|
||||||
parser.add_argument("--local-model", default=None)
|
|
||||||
parser.add_argument("--rollout-steps", type=int, default=256)
|
|
||||||
parser.add_argument("--batch-size", type=int, default=256)
|
|
||||||
parser.add_argument("--n-epochs", type=int, default=10)
|
|
||||||
parser.add_argument("--max-wall-s", type=float, default=1200.0,
|
|
||||||
help="训练墙钟硬上限；默认 20 分钟")
|
|
||||||
parser.add_argument("--reward-profile", choices=("time", "prompt"),
|
|
||||||
default="time",
|
|
||||||
help="time=P0.4 纯时间口径；prompt=原始 §5.2 塑形对照")
|
|
||||||
parser.add_argument("--progress-json", action="store_true",
|
|
||||||
help="向训练前端输出单行 JSON 进度（UI 自动启用）")
|
|
||||||
parser.add_argument("--control-file", default=None,
|
|
||||||
help="训练前端控制文件（JSON：paused/stop）；"
|
|
||||||
"跨平台暂停与优雅停止的主通道")
|
|
||||||
parser.add_argument("--progress-every", type=int, default=0,
|
|
||||||
help="JSON 进度间隔环境步数；0=每个 rollout")
|
|
||||||
parser.add_argument("--checkpoint-every", type=int, default=20000,
|
|
||||||
help="每 N 个环境步留一个历史 checkpoint；0=关闭（默认 20000）")
|
|
||||||
parser.add_argument("--recovery-every", type=int, default=2000,
|
|
||||||
help="每 N 个环境步原子覆盖写一份滚动恢复点 "
|
|
||||||
"<模型目录>/checkpoints/<stem>_latest.zip；"
|
|
||||||
"急停后最多丢 N 步。0=关闭（默认 2000）")
|
|
||||||
parser.add_argument("--checkpoint-keep", type=int, default=3,
|
|
||||||
help="checkpoint 保留最近 K 个（默认 3）")
|
|
||||||
parser.add_argument("--checkpoint-dir", default=None,
|
|
||||||
help="checkpoint 目录；默认 <模型目录>/checkpoints")
|
|
||||||
parser.add_argument("--verbose", type=int, default=1)
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
if args.timesteps <= 0 or args.envs <= 0:
|
|
||||||
parser.error("timesteps 与 envs 必须为正")
|
|
||||||
if args.envs * args.rollout_steps < 2:
|
|
||||||
parser.error("PPO rollout 至少需要 2 个样本")
|
|
||||||
if args.eval_episodes < 0:
|
|
||||||
parser.error("eval-episodes 不能为负")
|
|
||||||
if args.checkpoint_every < 0:
|
|
||||||
parser.error("checkpoint-every 不能为负")
|
|
||||||
if args.recovery_every < 0:
|
|
||||||
parser.error("recovery-every 不能为负")
|
|
||||||
if args.checkpoint_keep < 1:
|
|
||||||
parser.error("checkpoint-keep 至少为 1")
|
|
||||||
return run_training(args)
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,266 +0,0 @@
|
|||||||
"""低层 PPO 训练：8 方向到达任务（BC 直线专家热启动 + PPO 微调）。
|
|
||||||
|
|
||||||
产物：
|
|
||||||
output/hrl/hrl_low.zip / hrl_low.json PPO 模型（部署用）
|
|
||||||
output/hrl/bc_low_expert.npz / bc_low.pt 行为克隆数据与网络
|
|
||||||
output/hrl/hrl_low_curve_env*.monitor.csv 每个环境独立学习曲线
|
|
||||||
|
|
||||||
评估口径：到达成功率 + 路径效率（直线距离/实际路径长，1 为完美直线）。
|
|
||||||
这是独立导航实验：目前低层 PPO 仅供轨迹诊断，不影响正式 HTTP 任务。
|
|
||||||
|
|
||||||
示例：
|
|
||||||
uv run python scripts/train_hrl_low.py --timesteps 60000
|
|
||||||
uv run python scripts/train_hrl_low.py --skip-bc # 纯 PPO 对照
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import math
|
|
||||||
import time
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.hrl.constants import (FEATURE_VERSION, LOW_MAX_STEPS,
|
|
||||||
LOW_NUM_ACTIONS, LOW_OBS_DIM)
|
|
||||||
from mcm26b.hrl.low_level_env import LowLevelEnv
|
|
||||||
|
|
||||||
HIDDEN = (64, 64)
|
|
||||||
|
|
||||||
|
|
||||||
def _bc_module(obs_dim: int, n_actions: int, hidden: tuple[int, ...]):
|
|
||||||
import torch
|
|
||||||
|
|
||||||
class BCMlp(torch.nn.Module):
|
|
||||||
def __init__(self):
|
|
||||||
super().__init__()
|
|
||||||
layers = []
|
|
||||||
prev = obs_dim
|
|
||||||
for width in hidden:
|
|
||||||
layers += [torch.nn.Linear(prev, width), torch.nn.Tanh()]
|
|
||||||
prev = width
|
|
||||||
layers.append(torch.nn.Linear(prev, n_actions))
|
|
||||||
self.net = torch.nn.Sequential(*layers)
|
|
||||||
|
|
||||||
def forward(self, x):
|
|
||||||
return self.net(x)
|
|
||||||
|
|
||||||
return BCMlp
|
|
||||||
|
|
||||||
|
|
||||||
def collect(episodes: int, seed: int, config: ProblemConfig):
|
|
||||||
import numpy as np
|
|
||||||
env = LowLevelEnv(config=config, seed=seed)
|
|
||||||
observations, actions, episode_ids = [], [], []
|
|
||||||
for episode in range(episodes):
|
|
||||||
obs, _info = env.reset(seed=seed + episode)
|
|
||||||
for _step in range(LOW_MAX_STEPS + 2):
|
|
||||||
observations.append(obs)
|
|
||||||
actions.append(env.expert_action())
|
|
||||||
episode_ids.append(episode)
|
|
||||||
obs, _r, terminated, truncated, _info = env.step(env.expert_action())
|
|
||||||
if terminated or truncated:
|
|
||||||
break
|
|
||||||
env.close()
|
|
||||||
return (np.asarray(observations, dtype=np.float32),
|
|
||||||
np.asarray(actions, dtype=np.int64),
|
|
||||||
np.asarray(episode_ids, dtype=np.int64))
|
|
||||||
|
|
||||||
|
|
||||||
def bc_validation_mask(episode_ids, *, seed: int):
|
|
||||||
"""按整局留出，避免同一轨迹相邻状态跨训练/验证集。"""
|
|
||||||
import numpy as np
|
|
||||||
unique = np.unique(episode_ids)
|
|
||||||
if len(unique) < 2:
|
|
||||||
raise ValueError("BC 至少需要 2 局才能按整局划分训练/验证集")
|
|
||||||
rng = np.random.default_rng(seed)
|
|
||||||
validation_ids = rng.permutation(unique)[:max(1, len(unique) // 10)]
|
|
||||||
return np.isin(episode_ids, validation_ids)
|
|
||||||
|
|
||||||
|
|
||||||
def train_bc(x, y, episode_ids, *, epochs: int, batch_size: int, lr: float,
|
|
||||||
device: str, seed: int):
|
|
||||||
import torch
|
|
||||||
BCMlp = _bc_module(LOW_OBS_DIM, LOW_NUM_ACTIONS, HIDDEN)
|
|
||||||
model = BCMlp().to(device)
|
|
||||||
optimizer = torch.optim.AdamW(model.parameters(), lr=lr)
|
|
||||||
valid = torch.as_tensor(
|
|
||||||
bc_validation_mask(episode_ids, seed=seed), device=device)
|
|
||||||
x_t = torch.as_tensor(x, device=device)
|
|
||||||
y_t = torch.as_tensor(y, device=device)
|
|
||||||
x_v, y_v, x_tr, y_tr = x_t[valid], y_t[valid], x_t[~valid], y_t[~valid]
|
|
||||||
loss_fn = torch.nn.CrossEntropyLoss()
|
|
||||||
for epoch in range(epochs):
|
|
||||||
order = torch.randperm(len(x_tr), device=device)
|
|
||||||
total = correct = 0.0
|
|
||||||
for start in range(0, len(order), batch_size):
|
|
||||||
idx = order[start:start + batch_size]
|
|
||||||
logits = model(x_tr[idx])
|
|
||||||
loss = loss_fn(logits, y_tr[idx])
|
|
||||||
optimizer.zero_grad()
|
|
||||||
loss.backward()
|
|
||||||
optimizer.step()
|
|
||||||
total += float(loss.item()) * len(idx)
|
|
||||||
correct += int((logits.argmax(1) == y_tr[idx]).sum().item())
|
|
||||||
if epoch == 0 or (epoch + 1) % 5 == 0:
|
|
||||||
with torch.no_grad():
|
|
||||||
valid_acc = float((model(x_v).argmax(1) == y_v).float().mean().item())
|
|
||||||
print(json.dumps({"event": "bc_epoch", "epoch": epoch + 1,
|
|
||||||
"loss": round(total / len(x_tr), 4),
|
|
||||||
"train_acc": round(correct / len(x_tr), 4),
|
|
||||||
"valid_acc": round(valid_acc, 4)},
|
|
||||||
ensure_ascii=False))
|
|
||||||
with torch.no_grad():
|
|
||||||
valid_acc = float((model(x_v).argmax(1) == y_v).float().mean().item())
|
|
||||||
return model, valid_acc
|
|
||||||
|
|
||||||
|
|
||||||
def copy_into_ppo(bc_model, device: str, env, *, seed: int,
|
|
||||||
n_steps: int = 512) -> object:
|
|
||||||
"""把 BC 权重拷入 MaskablePPO（policy_net 两层 + action_net），前向校验。"""
|
|
||||||
import torch
|
|
||||||
from sb3_contrib import MaskablePPO
|
|
||||||
ppo = MaskablePPO("MlpPolicy", env, policy_kwargs={"net_arch": list(HIDDEN)},
|
|
||||||
n_steps=n_steps, batch_size=128, n_epochs=6,
|
|
||||||
gamma=1.0, gae_lambda=0.95, learning_rate=3e-4,
|
|
||||||
clip_range=0.2, ent_coef=0.005, vf_coef=0.5,
|
|
||||||
max_grad_norm=0.5, seed=seed, device=device)
|
|
||||||
state = {key.removeprefix("net."): value
|
|
||||||
for key, value in bc_model.state_dict().items()}
|
|
||||||
policy_state = {key: value for key, value in state.items()
|
|
||||||
if key in ("0.weight", "0.bias", "2.weight", "2.bias")}
|
|
||||||
ppo.policy.mlp_extractor.policy_net.load_state_dict(policy_state)
|
|
||||||
ppo.policy.action_net.load_state_dict(
|
|
||||||
{"weight": state["4.weight"], "bias": state["4.bias"]})
|
|
||||||
probe = torch.randn((8, LOW_OBS_DIM), device=device)
|
|
||||||
with torch.no_grad():
|
|
||||||
copied = ppo.policy.action_net(ppo.policy.mlp_extractor.policy_net(probe))
|
|
||||||
if not torch.allclose(bc_model(probe), copied, atol=1e-6):
|
|
||||||
raise RuntimeError("行为克隆与 PPO 前向映射不一致")
|
|
||||||
return ppo
|
|
||||||
|
|
||||||
|
|
||||||
def evaluate(model, *, episodes: int, seed: int,
|
|
||||||
config: ProblemConfig) -> dict:
|
|
||||||
"""确定性策略评估：成功率与路径效率（对照直线）。"""
|
|
||||||
env = LowLevelEnv(config=config, seed=seed)
|
|
||||||
successes = 0
|
|
||||||
efficiencies = []
|
|
||||||
for episode in range(episodes):
|
|
||||||
obs, info = env.reset(seed=seed + episode)
|
|
||||||
start, goal = tuple(info["position"]), tuple(info["goal"])
|
|
||||||
straight = math.dist(start, goal)
|
|
||||||
steps = 0
|
|
||||||
for _ in range(LOW_MAX_STEPS + 2):
|
|
||||||
action, _state = model.predict(obs, deterministic=True)
|
|
||||||
obs, _r, terminated, truncated, info = env.step(int(action))
|
|
||||||
steps += 1
|
|
||||||
if terminated or truncated:
|
|
||||||
break
|
|
||||||
successes += int(terminated)
|
|
||||||
efficiencies.append(min(1.0, straight /
|
|
||||||
max(info["path_length_m"], 1e-9)))
|
|
||||||
env.close()
|
|
||||||
return {"episodes": episodes,
|
|
||||||
"success_rate": successes / episodes,
|
|
||||||
"mean_path_efficiency": sum(efficiencies) / len(efficiencies)}
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("--timesteps", type=int, default=60_000)
|
|
||||||
parser.add_argument("--seed", type=int, default=20260911)
|
|
||||||
parser.add_argument("--bc-episodes", type=int, default=600)
|
|
||||||
parser.add_argument("--bc-epochs", type=int, default=12)
|
|
||||||
parser.add_argument("--envs", type=int, default=4)
|
|
||||||
parser.add_argument("--rollout-steps", type=int, default=512,
|
|
||||||
help="每环境每次 PPO rollout 长度；短 smoke 可设 8")
|
|
||||||
parser.add_argument("--device", default="cpu")
|
|
||||||
parser.add_argument("--model", default="output/hrl/hrl_low.zip")
|
|
||||||
parser.add_argument("--eval-episodes", type=int, default=40)
|
|
||||||
parser.add_argument("--skip-bc", action="store_true",
|
|
||||||
help="跳过行为克隆（纯 PPO 对照实验用）")
|
|
||||||
parser.add_argument("--verbose", type=int, default=1)
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
if min(args.envs, args.timesteps, args.eval_episodes,
|
|
||||||
args.rollout_steps) <= 0:
|
|
||||||
parser.error("envs/timesteps/eval-episodes/rollout-steps 必须为正数")
|
|
||||||
if args.envs * args.rollout_steps < 2:
|
|
||||||
parser.error("PPO rollout 至少需要 2 个样本")
|
|
||||||
if not args.skip_bc and (args.bc_episodes < 2 or args.bc_epochs < 1):
|
|
||||||
parser.error("BC 至少需要 2 局和 1 个 epoch")
|
|
||||||
|
|
||||||
import torch
|
|
||||||
torch.set_num_threads(1)
|
|
||||||
from sb3_contrib import MaskablePPO
|
|
||||||
from stable_baselines3.common.monitor import Monitor
|
|
||||||
from stable_baselines3.common.vec_env import DummyVecEnv
|
|
||||||
from stable_baselines3.common.utils import set_random_seed
|
|
||||||
set_random_seed(args.seed)
|
|
||||||
|
|
||||||
model_path = Path(args.model)
|
|
||||||
model_path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
curve_paths = [model_path.with_name(
|
|
||||||
f"{model_path.stem}_curve_env{rank}.monitor.csv")
|
|
||||||
for rank in range(args.envs)]
|
|
||||||
|
|
||||||
def _factory(rank: int):
|
|
||||||
def _make():
|
|
||||||
env = Monitor(LowLevelEnv(seed=args.seed + rank),
|
|
||||||
filename=str(curve_paths[rank]))
|
|
||||||
return env
|
|
||||||
return _make
|
|
||||||
|
|
||||||
vec_env = DummyVecEnv([_factory(rank) for rank in range(args.envs)])
|
|
||||||
started = time.monotonic()
|
|
||||||
bc_valid_acc = None
|
|
||||||
if args.skip_bc:
|
|
||||||
model = MaskablePPO("MlpPolicy", vec_env,
|
|
||||||
policy_kwargs={"net_arch": list(HIDDEN)},
|
|
||||||
n_steps=args.rollout_steps, batch_size=128, n_epochs=6,
|
|
||||||
gamma=1.0, gae_lambda=0.95, learning_rate=3e-4,
|
|
||||||
clip_range=0.2, ent_coef=0.005, vf_coef=0.5,
|
|
||||||
max_grad_norm=0.5, seed=args.seed,
|
|
||||||
device=args.device, verbose=args.verbose)
|
|
||||||
else:
|
|
||||||
x, y, episode_ids = collect(args.bc_episodes, args.seed, ProblemConfig())
|
|
||||||
bc_model, bc_valid_acc = train_bc(
|
|
||||||
x, y, episode_ids, epochs=args.bc_epochs, batch_size=256,
|
|
||||||
lr=3e-4, device=args.device, seed=args.seed)
|
|
||||||
model = copy_into_ppo(bc_model, args.device, vec_env, seed=args.seed,
|
|
||||||
n_steps=args.rollout_steps)
|
|
||||||
bc_dir = model_path.parent / "bc_low.pt"
|
|
||||||
torch.save(bc_model.state_dict(), str(bc_dir))
|
|
||||||
print(json.dumps({"event": "bc_done", "samples": int(len(x))},
|
|
||||||
ensure_ascii=False))
|
|
||||||
model.learn(total_timesteps=args.timesteps,
|
|
||||||
progress_bar=args.verbose > 0)
|
|
||||||
elapsed_s = time.monotonic() - started
|
|
||||||
|
|
||||||
model.save(str(model_path))
|
|
||||||
eval_seed = args.seed + 1_000_000
|
|
||||||
evaluation = evaluate(model, episodes=args.eval_episodes, seed=eval_seed,
|
|
||||||
config=ProblemConfig())
|
|
||||||
report = {
|
|
||||||
"level": "low", "timesteps": args.timesteps, "seed": args.seed,
|
|
||||||
"device": args.device, "wall_seconds": round(elapsed_s, 1),
|
|
||||||
"model": str(model_path), "algorithm": "maskable_ppo",
|
|
||||||
"feature_version": FEATURE_VERSION, "bc_skipped": args.skip_bc,
|
|
||||||
"bc_valid_acc": bc_valid_acc, "evaluation_seed": eval_seed,
|
|
||||||
"monitor_files": [str(path) for path in curve_paths],
|
|
||||||
"training_role": "standalone_navigation_only",
|
|
||||||
"obs_dim": LOW_OBS_DIM, "act_dim": LOW_NUM_ACTIONS,
|
|
||||||
"model_sha256": hashlib.sha256(model_path.read_bytes()).hexdigest(),
|
|
||||||
**evaluation,
|
|
||||||
}
|
|
||||||
model_path.with_suffix(".json").write_text(
|
|
||||||
json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
|
|
||||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
|
||||||
vec_env.close()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,5 +0,0 @@
|
|||||||
"""启动 HRL 模型训练前端（薄入口，便于加入支撑材料命令）。"""
|
|
||||||
from mcm26b.training_ui.server import run_training_ui
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(run_training_ui())
|
|
||||||
@@ -1,195 +0,0 @@
|
|||||||
"""问题3/4 强化学习选点策略训练（PPO + CUDA）。
|
|
||||||
|
|
||||||
训练环境 mcm26b.rl.env.LocalizationEnv 与运行闭环共享 features 观测/候选
|
|
||||||
布局。模型输出到 output/rl/q3_ppo.zip（问题3）或 q4_ppo.zip（问题4），
|
|
||||||
由 `mcm26b run --strategy rl` / `--q4-strategy rl` 加载。
|
|
||||||
|
|
||||||
示例：
|
|
||||||
uv sync --group rl --group dev
|
|
||||||
uv run python scripts/train_rl.py --problem 3 --timesteps 30000
|
|
||||||
uv run python scripts/train_rl.py --problem 4 --timesteps 30000
|
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test --strategy rl
|
|
||||||
uv run mcm26b run --robot-id 2026001 --self-test --problem 4 --q4-strategy rl
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import json
|
|
||||||
import time
|
|
||||||
from dataclasses import asdict
|
|
||||||
import hashlib
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from stable_baselines3 import PPO
|
|
||||||
from sb3_contrib import MaskablePPO
|
|
||||||
from stable_baselines3.common.vec_env import DummyVecEnv
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.rl.env import LocalizationEnv
|
|
||||||
|
|
||||||
|
|
||||||
def _make_env(problem: int, seed: int, config: ProblemConfig,
|
|
||||||
directional_probability: float):
|
|
||||||
def _factory():
|
|
||||||
env = LocalizationEnv(config, problem=problem,
|
|
||||||
directional_probability=directional_probability,
|
|
||||||
real_budget_s=3600.0)
|
|
||||||
env.reset(seed=seed)
|
|
||||||
return env
|
|
||||||
return _factory
|
|
||||||
|
|
||||||
|
|
||||||
def evaluate(model, problem: int, *, episodes: int = 24,
|
|
||||||
directional_probability: float = 0.5,
|
|
||||||
config: ProblemConfig = ProblemConfig()) -> dict:
|
|
||||||
"""独立场景评估：确定性策略的成功率、步数与平均回报。"""
|
|
||||||
env = LocalizationEnv(config, problem=problem,
|
|
||||||
directional_probability=directional_probability)
|
|
||||||
total_reward = 0.0
|
|
||||||
successes = 0
|
|
||||||
steps = []
|
|
||||||
costs = []
|
|
||||||
for episode in range(episodes):
|
|
||||||
obs, _info = env.reset(seed=1_000_000 + episode)
|
|
||||||
episode_reward = 0.0
|
|
||||||
for step in range(env.max_episode_steps + 2):
|
|
||||||
kwargs = {'action_masks': env.action_masks()} if isinstance(model, MaskablePPO) else {}
|
|
||||||
action, _state = model.predict(obs, deterministic=True, **kwargs)
|
|
||||||
obs, reward, terminated, truncated, info = env.step(int(action))
|
|
||||||
episode_reward += float(reward)
|
|
||||||
if terminated or truncated:
|
|
||||||
break
|
|
||||||
total_reward += episode_reward
|
|
||||||
cleared = bool(info.get("cleared"))
|
|
||||||
successes += int(cleared)
|
|
||||||
steps.append(step + 1)
|
|
||||||
costs.append(info['virtual_cost_s'])
|
|
||||||
env.close()
|
|
||||||
return {
|
|
||||||
"episodes": episodes,
|
|
||||||
"success_rate": successes / episodes,
|
|
||||||
"mean_steps": sum(steps) / episodes,
|
|
||||||
"mean_episode_reward": total_reward / episodes,
|
|
||||||
"successes": successes,
|
|
||||||
"mean_virtual_cost_s": sum(costs) / episodes,
|
|
||||||
"max_virtual_cost_s": max(costs),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("--problem", type=int, choices=(3, 4), default=3,
|
|
||||||
help="训练问题：3=全向源选点，4=含定向遮挡梯子")
|
|
||||||
parser.add_argument("--timesteps", type=int, default=30000,
|
|
||||||
help="PPO 总环境步数")
|
|
||||||
parser.add_argument("--seed", type=int, default=20260911)
|
|
||||||
parser.add_argument("--envs", type=int, default=2, help="并行环境份数（CPU 侧）")
|
|
||||||
parser.add_argument("--directional-probability", type=float, default=None,
|
|
||||||
help="问题4训练场景中定向源比例（默认 1.0，先学遮挡分支）")
|
|
||||||
parser.add_argument("--device", default="auto",
|
|
||||||
help="auto=优先 CUDA；或 cuda/cpu")
|
|
||||||
parser.add_argument("--model", default=None,
|
|
||||||
help="模型输出路径（默认 output/rl/q<problem>_ppo.zip）")
|
|
||||||
parser.add_argument("--init", default=None,
|
|
||||||
help="初始模型路径（如 output/rl/q3_ppo_bc_init.zip；"
|
|
||||||
"加载后继续 PPO 微调）")
|
|
||||||
parser.add_argument("--eval-episodes", type=int, default=24)
|
|
||||||
parser.add_argument("--verbose", type=int, default=1)
|
|
||||||
parser.add_argument('--algorithm', choices=('ppo', 'maskable_ppo'), default='maskable_ppo')
|
|
||||||
parser.add_argument('--torch-threads', type=int, default=1)
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
directional_probability = (args.directional_probability
|
|
||||||
if args.directional_probability is not None
|
|
||||||
else (1.0 if args.problem == 4 else 0.0))
|
|
||||||
if not 0.0 <= directional_probability <= 1.0:
|
|
||||||
parser.error("--directional-probability 须在 [0,1]")
|
|
||||||
|
|
||||||
import torch
|
|
||||||
torch.set_num_threads(args.torch_threads)
|
|
||||||
if args.device == "auto":
|
|
||||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
|
||||||
else:
|
|
||||||
device = args.device
|
|
||||||
if device == "cuda" and not torch.cuda.is_available():
|
|
||||||
parser.error("--device cuda 但当前环境检测不到 CUDA GPU")
|
|
||||||
print(json.dumps({
|
|
||||||
"event": "train_start", "problem": args.problem,
|
|
||||||
"timesteps": args.timesteps, "device": device,
|
|
||||||
"directional_probability": directional_probability,
|
|
||||||
"torch": torch.__version__,
|
|
||||||
"gpu": torch.cuda.get_device_name(0) if device == "cuda" else None,
|
|
||||||
"cuda_available": torch.cuda.is_available(),
|
|
||||||
}, ensure_ascii=False))
|
|
||||||
|
|
||||||
config = ProblemConfig()
|
|
||||||
model_cls = MaskablePPO if args.algorithm == 'maskable_ppo' else PPO
|
|
||||||
vec_env = DummyVecEnv(
|
|
||||||
[_make_env(args.problem, args.seed + rank, config,
|
|
||||||
directional_probability)
|
|
||||||
for rank in range(max(1, args.envs))])
|
|
||||||
if args.init:
|
|
||||||
metadata = json.loads(Path(args.init).with_suffix('.json').read_text(encoding='utf-8'))
|
|
||||||
if (metadata.get('algorithm', 'ppo') != args.algorithm
|
|
||||||
or metadata.get('feature_version') != 2):
|
|
||||||
parser.error('初始化模型的算法/特征版本与当前训练不符，请重训或提供v2初始化模型')
|
|
||||||
model = model_cls.load(args.init, env=vec_env, device=device)
|
|
||||||
else:
|
|
||||||
model = model_cls(
|
|
||||||
"MlpPolicy", vec_env,
|
|
||||||
policy_kwargs={"net_arch": [128, 128]},
|
|
||||||
n_steps=512,
|
|
||||||
batch_size=128,
|
|
||||||
n_epochs=6,
|
|
||||||
gamma=1.0,
|
|
||||||
gae_lambda=0.95,
|
|
||||||
learning_rate=3e-4,
|
|
||||||
clip_range=0.2,
|
|
||||||
ent_coef=0.005,
|
|
||||||
vf_coef=0.5,
|
|
||||||
max_grad_norm=0.5,
|
|
||||||
seed=args.seed,
|
|
||||||
device=device,
|
|
||||||
verbose=args.verbose,
|
|
||||||
)
|
|
||||||
|
|
||||||
started = time.monotonic()
|
|
||||||
model.learn(total_timesteps=args.timesteps,
|
|
||||||
progress_bar=args.verbose > 0)
|
|
||||||
elapsed_s = time.monotonic() - started
|
|
||||||
|
|
||||||
model_path = Path(args.model) if args.model \
|
|
||||||
else Path("output/rl") / f"q{args.problem}_ppo.zip"
|
|
||||||
model_path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
model.save(str(model_path))
|
|
||||||
|
|
||||||
evaluation = evaluate(model, args.problem, episodes=args.eval_episodes,
|
|
||||||
directional_probability=directional_probability,
|
|
||||||
config=config)
|
|
||||||
report = {
|
|
||||||
"problem": args.problem,
|
|
||||||
"timesteps": args.timesteps,
|
|
||||||
"seed": args.seed,
|
|
||||||
"device": device,
|
|
||||||
"directional_probability": directional_probability,
|
|
||||||
"wall_seconds": round(elapsed_s, 1),
|
|
||||||
"model": str(model_path),
|
|
||||||
"init": args.init,
|
|
||||||
"algorithm": args.algorithm,
|
|
||||||
"feature_version": 2,
|
|
||||||
"reward": 'negative_virtual_cost_div100_success20_failure100',
|
|
||||||
"gamma": 1.0,
|
|
||||||
"config": asdict(config),
|
|
||||||
"model_sha256": hashlib.sha256(model_path.read_bytes()).hexdigest(),
|
|
||||||
**evaluation,
|
|
||||||
}
|
|
||||||
report_path = model_path.with_suffix(".json")
|
|
||||||
report_path.write_text(json.dumps(report, ensure_ascii=False, indent=2),
|
|
||||||
encoding="utf-8")
|
|
||||||
print(json.dumps(report, ensure_ascii=False, indent=2))
|
|
||||||
vec_env.close()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,27 +0,0 @@
|
|||||||
"""HRL Optuna TPE 调参入口（P2 待实现）。
|
|
||||||
|
|
||||||
搜索空间按用户 Prompt §10，但只允许在不破坏覆盖证明/证书/预算的工程参数上搜索：
|
|
||||||
粒子数、重采样阈值、信息增益权重、移动成本权重、检测成本权重、候选点距离、
|
|
||||||
覆盖网格间距、停止连续无信号次数、清除距离阈值。目标 = 平均定位清除时间 +
|
|
||||||
未清除惩罚，每个配置至少 5 个随机种子。
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
|
|
||||||
|
|
||||||
def main(argv=None) -> int:
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("--problem", type=int, choices=(3, 4), default=3)
|
|
||||||
parser.add_argument("--trials", type=int, default=30)
|
|
||||||
parser.add_argument("--timesteps", type=int, default=2048)
|
|
||||||
parser.add_argument("--manifest", default="output/hrl/manifests/dev.json")
|
|
||||||
parser.add_argument("--output", default="output/hrl/p2-tuning")
|
|
||||||
parser.parse_args(argv)
|
|
||||||
raise NotImplementedError(
|
|
||||||
"P2 尚未实现：调参必须建立在 P1 smoke 与稳定基准之上，"
|
|
||||||
"且不能搜索真实误差上限或破坏覆盖证明的网格参数。")
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(main())
|
|
||||||
@@ -1,123 +0,0 @@
|
|||||||
"""Offline audit of official RL practice runs (Q3/Q4); never connects to simulator.
|
|
||||||
|
|
||||||
Checks the copied official .result.json/.jlog, our session.jsonl and summary.json:
|
|
||||||
package sha256, enter/exit, accepted timestamps inside official window, unique
|
|
||||||
request ids, clear-success channel count == official jammer_count, summary
|
|
||||||
consistency, and virtual time recomputation with integer microsecond accumulation.
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import argparse
|
|
||||||
import hashlib
|
|
||||||
import json
|
|
||||||
import math
|
|
||||||
from datetime import datetime
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
|
|
||||||
def save_json(path: Path, value: dict) -> None:
|
|
||||||
path.write_text(json.dumps(value, ensure_ascii=False, indent=2,
|
|
||||||
allow_nan=False) + "\n", encoding="utf-8")
|
|
||||||
|
|
||||||
|
|
||||||
def audit(folder: Path) -> dict:
|
|
||||||
records = [json.loads(line) for line in
|
|
||||||
(folder / "session.jsonl").read_text(encoding="utf-8").splitlines()]
|
|
||||||
events = [e for e in records if e.get("event") == "accepted"]
|
|
||||||
summary = json.loads((folder / "summary.json").read_text(encoding="utf-8"))
|
|
||||||
official_paths = list(folder.glob("*.result.json"))
|
|
||||||
if len(official_paths) != 1 or not events:
|
|
||||||
raise ValueError("Expected one official result and nonempty session evidence")
|
|
||||||
official = json.loads(official_paths[0].read_text(encoding="utf-8"))
|
|
||||||
package = official_paths[0].with_name(
|
|
||||||
official_paths[0].name.replace(".result.json", ".jlog"))
|
|
||||||
checks: dict = {}
|
|
||||||
|
|
||||||
checks["official_problem_matches_summary"] = (
|
|
||||||
official["problem_no"] in (3, 4)
|
|
||||||
and summary["strategy"] == ("rl" if official["problem_no"] == 3 else "rl_q4")
|
|
||||||
and 10 <= official["jammer_count"] <= 16
|
|
||||||
and official["omnidirectional_jammer_count"] + official["directional_jammer_count"]
|
|
||||||
== official["jammer_count"])
|
|
||||||
checks["official_package_sha256_matches"] = (
|
|
||||||
hashlib.sha256(package.read_bytes()).hexdigest() == official["package_sha256"])
|
|
||||||
checks["normal_enter_and_exit"] = (
|
|
||||||
events[0]["path"] == "/enter" and events[-1]["path"] == "/exit"
|
|
||||||
and events[-1]["response"].get("exit_reason") == "user_exit")
|
|
||||||
start_ms = datetime.fromisoformat(official["window_started_at_utc"]).timestamp() * 1000
|
|
||||||
end_ms = datetime.fromisoformat(official["ended_at_utc"]).timestamp() * 1000
|
|
||||||
checks["responses_within_official_run_window"] = all(
|
|
||||||
start_ms <= e["response"]["real_timestamp_ms"] <= end_ms + 1 for e in events)
|
|
||||||
ids = [e["request"]["request_id"] for e in events]
|
|
||||||
checks["unique_accepted_request_ids"] = len(ids) == len(set(ids))
|
|
||||||
clears = [e for e in events if e["path"] == "/clear"]
|
|
||||||
successes = [e for e in clears if e["response"].get("clear_result") == "success"]
|
|
||||||
cleared = sorted({e["request"]["channel"] for e in successes})
|
|
||||||
checks["all_official_sources_cleared"] = len(cleared) == official["jammer_count"]
|
|
||||||
checks["summary_completed_and_consistent"] = (
|
|
||||||
bool(summary["completed"])
|
|
||||||
and sorted(summary["cleared_channels"]) == cleared
|
|
||||||
and summary["unresolved_channels"] == []
|
|
||||||
and summary["anomalies"] == [])
|
|
||||||
|
|
||||||
position, channel, virtual_us = (0.0, 0.0), 1, 0
|
|
||||||
mismatches = 0
|
|
||||||
for event in events:
|
|
||||||
request, response, path = event["request"], event["response"], event["path"]
|
|
||||||
distance, switches, action_s = 0.0, 0, 0
|
|
||||||
if path in ("/measure", "/clear"):
|
|
||||||
target = (request["position"]["x"], request["position"]["y"])
|
|
||||||
distance = math.dist(position, target)
|
|
||||||
position = target
|
|
||||||
if path == "/measure":
|
|
||||||
switches = int(channel != request["channel"])
|
|
||||||
channel = request["channel"]
|
|
||||||
action_s = 5
|
|
||||||
else:
|
|
||||||
action_s = 5 if response["clear_result"] == "success" else 3
|
|
||||||
virtual_us += round(distance / 5 * 1_000_000) + (switches + action_s) * 1_000_000
|
|
||||||
if round(response["virtual_time_s"] * 1_000_000) != virtual_us:
|
|
||||||
mismatches += 1
|
|
||||||
checks["all_timestamps_match_microsecond_recomputation"] = mismatches == 0
|
|
||||||
real_s = (events[-1]["response"]["real_timestamp_ms"]
|
|
||||||
- events[0]["response"]["real_timestamp_ms"]) / 1000
|
|
||||||
checks["within_real_and_virtual_limits"] = (
|
|
||||||
real_s < events[0]["response"]["remaining_real_duration_s"]
|
|
||||||
and virtual_us / 1_000_000 < events[0]["response"]["max_virtual_duration_s"])
|
|
||||||
|
|
||||||
report = {
|
|
||||||
"passed": all(checks.values()),
|
|
||||||
"source": "official_practice",
|
|
||||||
"strategy": summary["strategy"],
|
|
||||||
"problem": official["problem_no"],
|
|
||||||
"case_code": official["case_code"],
|
|
||||||
"official_jammer_count": official["jammer_count"],
|
|
||||||
"directional_jammer_count": official["directional_jammer_count"],
|
|
||||||
"cleared_channels": cleared,
|
|
||||||
"accepted_requests_including_enter_exit": len(events),
|
|
||||||
"measures": len([e for e in events if e["path"] == "/measure"]),
|
|
||||||
"clear_attempts": len(clears),
|
|
||||||
"clear_successes": len(successes),
|
|
||||||
"virtual_time_s": virtual_us / 1_000_000,
|
|
||||||
"real_elapsed_server_s": real_s,
|
|
||||||
"checks": checks,
|
|
||||||
"limits": "仅验证本官方演练案例；不触碰正式测试，不解读官方加密日志内容。",
|
|
||||||
}
|
|
||||||
save_json(folder / "verification.json", report)
|
|
||||||
manifest = {p.name: {"bytes": p.stat().st_size,
|
|
||||||
"sha256": hashlib.sha256(p.read_bytes()).hexdigest()}
|
|
||||||
for p in sorted(folder.iterdir()) if p.is_file()
|
|
||||||
and p.name != "manifest.json"}
|
|
||||||
save_json(folder / "manifest.json", manifest)
|
|
||||||
return report
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
parser = argparse.ArgumentParser(description=__doc__)
|
|
||||||
parser.add_argument("folder", type=Path)
|
|
||||||
args = parser.parse_args()
|
|
||||||
result = audit(args.folder)
|
|
||||||
print(json.dumps({"passed": result["passed"], "case_code": result["case_code"],
|
|
||||||
"problem": result["problem"], "checks": result["checks"]},
|
|
||||||
ensure_ascii=False, indent=2))
|
|
||||||
raise SystemExit(0 if result["passed"] else 1)
|
|
||||||
+5
-60
@@ -30,7 +30,6 @@ from .protocol.log import SessionLog
|
|||||||
from .protocol.rules import EXIT_USER
|
from .protocol.rules import EXIT_USER
|
||||||
|
|
||||||
Q3_STRATEGIES = {"grid": OmnidirectionalLoop, "seven": SevenBatch, "bayes": BayesLoop}
|
Q3_STRATEGIES = {"grid": OmnidirectionalLoop, "seven": SevenBatch, "bayes": BayesLoop}
|
||||||
Q4_STRATEGIES = {"directional": DirectionalExtension}
|
|
||||||
|
|
||||||
|
|
||||||
def _load_referee():
|
def _load_referee():
|
||||||
@@ -252,44 +251,13 @@ def _run(args) -> int:
|
|||||||
log = SessionLog(path=Path(args.log)) if args.log else SessionLog()
|
log = SessionLog(path=Path(args.log)) if args.log else SessionLog()
|
||||||
client = SimulatorClient(args.robot_id, config=config, transport=transport, log=log)
|
client = SimulatorClient(args.robot_id, config=config, transport=transport, log=log)
|
||||||
if args.problem == 4:
|
if args.problem == 4:
|
||||||
strategy = args.q4_strategy
|
strategy = "directional"
|
||||||
loop_cls = Q4_STRATEGIES.get(strategy)
|
loop_cls = DirectionalExtension
|
||||||
else:
|
else:
|
||||||
strategy = args.strategy
|
strategy = args.strategy
|
||||||
loop_cls = Q3_STRATEGIES.get(strategy)
|
loop_cls = Q3_STRATEGIES[strategy]
|
||||||
high_policy = None
|
|
||||||
low_policy = None
|
|
||||||
try:
|
try:
|
||||||
if strategy == "rl":
|
|
||||||
# 惰性导入：默认运行闭环不加载 numpy/torch/stable-baselines3。
|
|
||||||
from .planner.rl_loop import RLBayesLoop, RLDirectionalLoop
|
|
||||||
loop_cls = RLBayesLoop if args.problem == 3 else RLDirectionalLoop
|
|
||||||
elif strategy == "hrl":
|
|
||||||
# P1：高层 MaskablePPO 调度。缺少模型时显式报错，不降级伪装。
|
|
||||||
if not args.hrl_model:
|
|
||||||
raise RuntimeError(
|
|
||||||
"--strategy hrl/--q4-strategy hrl 需要 --hrl-model；"
|
|
||||||
"无模型启发式请使用 benchmark_hrl.py 的 hrl_heuristic 变体。")
|
|
||||||
from .hrl.policy import HighLevelPolicy, LowLevelPolicy
|
|
||||||
from .planner.hrl_loop import HRLLoop
|
|
||||||
high_policy = HighLevelPolicy(
|
|
||||||
args.hrl_model, device=args.hrl_device, problem=args.problem)
|
|
||||||
if args.hrl_low_model:
|
|
||||||
low_policy = LowLevelPolicy(args.hrl_low_model, device=args.hrl_device)
|
|
||||||
loop_cls = HRLLoop
|
|
||||||
if loop_cls is None:
|
|
||||||
raise RuntimeError(f"未知策略: {strategy}")
|
|
||||||
if strategy == "rl":
|
|
||||||
summary = loop_cls(client, config, model_path=args.rl_model).run()
|
|
||||||
elif strategy == "hrl":
|
|
||||||
summary = loop_cls(
|
|
||||||
client, config, problem=args.problem, high_policy=high_policy,
|
|
||||||
low_policy=low_policy, trace_executor=args.trace_low).run()
|
|
||||||
else:
|
|
||||||
summary = loop_cls(client, config).run()
|
summary = loop_cls(client, config).run()
|
||||||
except RuntimeError as exc:
|
|
||||||
print(f"策略不可用: {exc}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
finally:
|
finally:
|
||||||
if judge is not None:
|
if judge is not None:
|
||||||
_settle_local_judge(judge)
|
_settle_local_judge(judge)
|
||||||
@@ -301,9 +269,6 @@ def _run(args) -> int:
|
|||||||
def _finalize_run(args, summary, strategy, *, truth, judge, judge_session,
|
def _finalize_run(args, summary, strategy, *, truth, judge, judge_session,
|
||||||
real_budget_s) -> int:
|
real_budget_s) -> int:
|
||||||
"""补齐 summary 元信息、核对真值并输出；返回进程退出码。"""
|
"""补齐 summary 元信息、核对真值并输出；返回进程退出码。"""
|
||||||
if args.problem == 4:
|
|
||||||
summary["strategy"] = "rl_q4" if strategy == "rl" else strategy
|
|
||||||
else:
|
|
||||||
summary["strategy"] = strategy
|
summary["strategy"] = strategy
|
||||||
if real_budget_s is not None:
|
if real_budget_s is not None:
|
||||||
summary["self_test_real_budget_s"] = real_budget_s
|
summary["self_test_real_budget_s"] = real_budget_s
|
||||||
@@ -349,30 +314,10 @@ def main(argv=None) -> int:
|
|||||||
run.add_argument("--log", default=None, help="会话日志 JSONL 输出路径")
|
run.add_argument("--log", default=None, help="会话日志 JSONL 输出路径")
|
||||||
run.add_argument("--problem", type=int, choices=(3, 4), default=3,
|
run.add_argument("--problem", type=int, choices=(3, 4), default=3,
|
||||||
help="问题3=全向源，问题4=混合定向源（默认 3）")
|
help="问题3=全向源，问题4=混合定向源（默认 3）")
|
||||||
run.add_argument("--strategy", choices=tuple(Q3_STRATEGIES) + ("rl", "hrl"),
|
run.add_argument("--strategy", choices=tuple(Q3_STRATEGIES), default="bayes",
|
||||||
default="bayes",
|
|
||||||
help="问题3策略：bayes=粒子滤波+信息增益+滚动调度（默认，"
|
help="问题3策略：bayes=粒子滤波+信息增益+滚动调度（默认，"
|
||||||
"自建消融相对 seven 再降 19.5%%），seven=七点批量，"
|
"自建消融相对 seven 再降 19.5%%），seven=七点批量，"
|
||||||
"grid=49点网格基线，rl=PPO 强化学习选点（需先训练模型），"
|
"grid=49点网格基线；仅 --problem 3 生效")
|
||||||
"hrl=高层 MaskablePPO 分层调度（需 --hrl-model）；"
|
|
||||||
"仅 --problem 3 生效")
|
|
||||||
run.add_argument("--q4-strategy", choices=tuple(Q4_STRATEGIES) + ("rl", "hrl"),
|
|
||||||
default="directional",
|
|
||||||
help="问题4策略：directional=49点覆盖+遮挡梯子+后备清除（默认），"
|
|
||||||
"rl=PPO 强化学习选点（需先训练模型），"
|
|
||||||
"hrl=高层 MaskablePPO 分层调度（需 --hrl-model）；仅 --problem 4 生效")
|
|
||||||
run.add_argument("--rl-model", default=None,
|
|
||||||
help="RL 模型路径（默认 output/rl/q3_ppo.zip 或 "
|
|
||||||
"output/rl/q4_ppo.zip；训练脚本 scripts/train_rl.py）")
|
|
||||||
run.add_argument("--hrl-model", default=None,
|
|
||||||
help="高层 HRL MaskablePPO 模型路径（--strategy hrl 时必填；"
|
|
||||||
"训练脚本 scripts/train_hrl_high.py）")
|
|
||||||
run.add_argument("--hrl-low-model", default=None,
|
|
||||||
help="可选低层 HRL 模型；当前只用于轨迹推演留痕，不改变协议终点")
|
|
||||||
run.add_argument("--hrl-device", choices=("auto", "cpu", "cuda", "mps"),
|
|
||||||
default="auto", help="HRL 模型推理设备")
|
|
||||||
run.add_argument("--trace-low", action="store_true",
|
|
||||||
help="开启低层到达推演留痕（本地虚拟轨迹，不增加协议耗时）")
|
|
||||||
run.add_argument("--bearing-margin-deg", type=float, default=0.005,
|
run.add_argument("--bearing-margin-deg", type=float, default=0.005,
|
||||||
help="示向度舍入裕量（服务器保留两位小数；默认 0.005°）")
|
help="示向度舍入裕量（服务器保留两位小数；默认 0.005°）")
|
||||||
run.add_argument("--real-budget-s", type=float, default=None,
|
run.add_argument("--real-budget-s", type=float, default=None,
|
||||||
|
|||||||
@@ -1,14 +0,0 @@
|
|||||||
"""分层强化学习实验：整场子目标调度环境与独立的八方向到达实验。
|
|
||||||
|
|
||||||
与 :mod:`mcm26b.rl`（单步 PPO 选点）的差异：高层在整场 10–16 源的统一
|
|
||||||
候选槽位上做调度决策（证书清除 / 信息增益定位 / 覆盖站扫频 / 后备清除），
|
|
||||||
当前运行闭环将子目标端点提交协议，八方向策略仅用于可选本地路径留痕，
|
|
||||||
不影响真实请求及虚拟计时。完整高层训练、CLI 与配对评估仍待接通；
|
|
||||||
优化架构及逐项验收见 docs/hrl-sol-execution.md。
|
|
||||||
|
|
||||||
信念（belief + 网格幸存集）、证书审批与后备清除复用 planner 基础设施；
|
|
||||||
每次改动仍须验证真源保留和真实全清，不能用内部状态自洽替代评测。
|
|
||||||
|
|
||||||
本包属于训练/推理侧组件，运行时对依赖保持惰性（stable-baselines3 /
|
|
||||||
sb3-contrib 仅在显式加载模型时导入）。
|
|
||||||
"""
|
|
||||||
@@ -1,49 +0,0 @@
|
|||||||
"""HRL 观测/动作空间常量：模型接口的唯一字面量来源。
|
|
||||||
|
|
||||||
槽位布局（训练与推理共享，改动任一值都会使已训练模型失配，须同步重训）：
|
|
||||||
|
|
||||||
0–19 clear 频道 c 的证书清除（槽位 = c-1，证书就绪才有效）
|
|
||||||
20–39 locate 频道 c 的定位测量（信息增益选点或定向遮挡梯子探点）
|
|
||||||
40–88 station 覆盖站（问题3：七点环 0–6 有效；问题4：49 点网格）
|
|
||||||
89–108 fallback 频道 c 的后备网格清除（仅问题4，遮挡梯子耗尽后开放）
|
|
||||||
109–119 保留（恒屏蔽）
|
|
||||||
|
|
||||||
高层观测 = 全局 8 维 + 20 频道 × 14 维 + 120 槽位 × 8 维 = 1248 维；
|
|
||||||
低层观测 10 维、动作 Discrete(8)（8 方向，步长 ≤100 m）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
FEATURE_VERSION = 4
|
|
||||||
ACTION_LAYOUT_VERSION = 1
|
|
||||||
REWARD_VERSION = 1
|
|
||||||
CHANNEL_COUNT = 20
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 高层动作槽位
|
|
||||||
CLEAR_SLOTS = CHANNEL_COUNT
|
|
||||||
LOCATE_SLOTS = CHANNEL_COUNT
|
|
||||||
STATION_SLOTS = 49
|
|
||||||
FALLBACK_SLOTS = CHANNEL_COUNT
|
|
||||||
CLEAR_OFFSET = 0
|
|
||||||
LOCATE_OFFSET = CLEAR_OFFSET + CLEAR_SLOTS
|
|
||||||
STATION_OFFSET = LOCATE_OFFSET + LOCATE_SLOTS
|
|
||||||
FALLBACK_OFFSET = STATION_OFFSET + STATION_SLOTS
|
|
||||||
N_CANDIDATES = 120
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 高层观测
|
|
||||||
PER_CHANNEL_DIM = 14
|
|
||||||
HIGH_GLOBAL_DIM = 8
|
|
||||||
PER_CANDIDATE_DIM = 8
|
|
||||||
HIGH_OBS_DIM = (HIGH_GLOBAL_DIM + CHANNEL_COUNT * PER_CHANNEL_DIM
|
|
||||||
+ N_CANDIDATES * PER_CANDIDATE_DIM) # 1248
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 低层接口
|
|
||||||
LOW_OBS_DIM = 10
|
|
||||||
LOW_NUM_ACTIONS = 8 # 8 方向，45° 等分
|
|
||||||
LOW_STEP_M = 100.0 # 每步最大步长 [题面 低层动作 50–100 m]
|
|
||||||
LOW_REACH_M = 20.0 # 到达阈值（与清除半径一致）
|
|
||||||
LOW_MAX_STEPS = 60 # 直径穿越 ~51 步上界
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 观测归一化
|
|
||||||
POS_SCALE_M = 2000.0 # 位置/位移归一化尺度（Ω 半径 1800 m 外扩）
|
|
||||||
RADIUS_SCALE_M = 2000.0
|
|
||||||
COST_SCALE_S = 600.0 # 槽位预计服务耗时归一化尺度
|
|
||||||
MEASURE_COUNT_SCALE = 12.0
|
|
||||||
@@ -1,109 +0,0 @@
|
|||||||
"""课程学习与域随机化：训练场景按阶段由易到难（prompt §7）。
|
|
||||||
|
|
||||||
四阶段课程：1 源/5 频道/无噪声 → 3 源/10 频道/±0.5° → 8 源/16 频道/±1°
|
|
||||||
→ 10–16 源/20 频道/±1°（问题4 最后阶段按比例混入定向源）。域随机化
|
|
||||||
覆盖源位置（Ω 圆域均匀）、频道（互异抽样）、有效接收半径、朝向与逐局
|
|
||||||
误差种子 [prompt §7.2]；机器狗初始位置固定 (0,0)——协议语义如此，
|
|
||||||
随机化它反而制造训练-部署差异（与 v1"训练部署特征错位"教训一致）。
|
|
||||||
|
|
||||||
真值只进入训练/评估场景生成，不进入任何观测；策略端仅见模拟器响应。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import dataclasses
|
|
||||||
import math
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from random import Random
|
|
||||||
|
|
||||||
from ..config import ProblemConfig
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class Stage:
|
|
||||||
"""课程阶段参数：源数范围、频道上限、示向度噪声与定向源比例。"""
|
|
||||||
|
|
||||||
name: str
|
|
||||||
sources_min: int
|
|
||||||
sources_max: int
|
|
||||||
channel_max: int # 频道 > channel_max 的频道保证无源
|
|
||||||
bearing_error_half_deg: float
|
|
||||||
directional_ratio: float # 问题4 最后阶段定向源比例；问题3 恒 0
|
|
||||||
recommended_timesteps: int
|
|
||||||
|
|
||||||
|
|
||||||
STAGES: tuple[Stage, ...] = (
|
|
||||||
Stage("s1-single", 1, 1, 5, 0.0, 0.0, 60_000),
|
|
||||||
Stage("s2-few", 3, 3, 10, 0.5, 0.0, 80_000),
|
|
||||||
Stage("s3-many", 8, 8, 16, 1.0, 0.0, 100_000),
|
|
||||||
Stage("s4-full", 10, 16, 20, 1.0, 0.4, 140_000),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def stage_with_noise(config: ProblemConfig, stage: Stage) -> ProblemConfig:
|
|
||||||
"""把阶段噪声写入 ProblemConfig（belief/PF 的保守裕量随之同步）。"""
|
|
||||||
field = dataclasses.replace(
|
|
||||||
config.field, bearing_error_half_deg=stage.bearing_error_half_deg)
|
|
||||||
return dataclasses.replace(config, field=field)
|
|
||||||
|
|
||||||
|
|
||||||
# 快速模式只减少选点评分的样本，不改变证明使用的幸存网格。29.3m 格距的
|
|
||||||
# Voronoi 裕量约20.72m，已经超过20m清除半径，会使粒子证书永远不可达。
|
|
||||||
# 默认使用生产配置；快速模式必须显式指定并在训练元数据记录。
|
|
||||||
TRAINING_SCORE_SUBSAMPLE = 240
|
|
||||||
TRAINING_MC_REPRESENTATIVES = 5
|
|
||||||
|
|
||||||
|
|
||||||
def training_config(config: ProblemConfig, stage: Stage, *,
|
|
||||||
fidelity: str = "production") -> ProblemConfig:
|
|
||||||
"""阶段噪声配置；fast只降低评分采样量，production与部署同参数。"""
|
|
||||||
if fidelity not in ("production", "fast"):
|
|
||||||
raise ValueError("fidelity 必须是 production 或 fast")
|
|
||||||
noisy = stage_with_noise(config, stage)
|
|
||||||
if fidelity == "production":
|
|
||||||
return noisy
|
|
||||||
bayes = dataclasses.replace(
|
|
||||||
noisy.bayes,
|
|
||||||
score_subsample=TRAINING_SCORE_SUBSAMPLE,
|
|
||||||
mc_representatives=TRAINING_MC_REPRESENTATIVES)
|
|
||||||
return dataclasses.replace(noisy, bayes=bayes)
|
|
||||||
|
|
||||||
|
|
||||||
def make_scene(rng: Random, config: ProblemConfig, problem: int,
|
|
||||||
stage: Stage, *, domain_randomization: bool = True
|
|
||||||
) -> tuple[list, list[int]]:
|
|
||||||
"""生成一局课程场景，返回 (referee.Source 列表, 已占用频道列表)。
|
|
||||||
|
|
||||||
源位置在 Ω 圆域均匀分布、频道从 1..stage.channel_max 互异抽样、
|
|
||||||
有效接收半径 [1000,1500] 均匀 [题面]；问题4 按 stage.directional_ratio
|
|
||||||
混入定向源，朝向均匀。domain_randomization=False 时只用 rng 的确定
|
|
||||||
序列（仍随机但可整场复现），供评估固定场景对照。
|
|
||||||
"""
|
|
||||||
from referee import Source
|
|
||||||
|
|
||||||
field = config.field
|
|
||||||
count = rng.randint(stage.sources_min, stage.sources_max)
|
|
||||||
channels = rng.sample(range(field.channel_min, stage.channel_max + 1), count)
|
|
||||||
sources = []
|
|
||||||
for channel in channels:
|
|
||||||
radius = field.omega_radius_m * math.sqrt(rng.random())
|
|
||||||
angle = rng.uniform(0.0, 2.0 * math.pi)
|
|
||||||
r_eff = rng.uniform(field.r_eff_min_m, field.r_eff_max_m)
|
|
||||||
directional = (problem == 4
|
|
||||||
and rng.random() < stage.directional_ratio)
|
|
||||||
heading = rng.uniform(0.0, 360.0) if directional else 0.0
|
|
||||||
if not domain_randomization:
|
|
||||||
# 固定化退耦：半径/朝向取区间代表值，保留位置随机（评估用）。
|
|
||||||
r_eff = field.r_eff_min_m
|
|
||||||
sources.append(Source(channel=channel, x=radius * math.cos(angle),
|
|
||||||
y=radius * math.sin(angle), r_eff_m=r_eff,
|
|
||||||
directional=directional, heading_deg=heading))
|
|
||||||
return sources, channels
|
|
||||||
|
|
||||||
|
|
||||||
def training_spec(config: ProblemConfig, stage: Stage, *,
|
|
||||||
max_real_duration_s: float = 3600.0):
|
|
||||||
"""带阶段噪声的 referee Spec（训练侧使用；正式对接以 /enter 为准）。"""
|
|
||||||
from ..harness import referee_spec
|
|
||||||
return referee_spec(stage_with_noise(config, stage),
|
|
||||||
max_real_duration_s=max_real_duration_s)
|
|
||||||
@@ -1,113 +0,0 @@
|
|||||||
"""低层到达执行器与观测：8 方向步进（prompt §4.2），训练/部署同源。
|
|
||||||
|
|
||||||
低层只负责"怎么去"：给定当前位置与子目标，输出离散 8 方向步进序列。
|
|
||||||
步长每步 ≤ ``LOW_STEP_M``，近目标时缩短步长；到达阈值 ``LOW_REACH_M``
|
|
||||||
与清除半径一致。推演完全本地，不与模拟器交互、不产生虚拟耗时——协议
|
|
||||||
语义下移动耗时 = 两次动作位置间的直线距离 ÷ 速度，仅由终点决定。
|
|
||||||
|
|
||||||
本模块仅供独立导航实验和轨迹诊断；当前 HTTP 执行路径仍直接服务高层
|
|
||||||
目标，不能据此声称低层 PPO 改善正式任务耗时。policy=None 使用解析
|
|
||||||
直线控制器；PPO 推演超时不会免费跳到目标。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import math
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
|
|
||||||
from ..config import Vec2
|
|
||||||
from .constants import (LOW_MAX_STEPS, LOW_NUM_ACTIONS,
|
|
||||||
LOW_REACH_M, LOW_STEP_M, POS_SCALE_M)
|
|
||||||
|
|
||||||
__all__ = ["build_low_observation", "LowLevelExecutor", "action_bearing",
|
|
||||||
"navigation_transition"]
|
|
||||||
|
|
||||||
|
|
||||||
def action_bearing(action: int) -> float:
|
|
||||||
"""离散方向 -> 弧度方位（0 = +x，逆时针 45° 等分）。"""
|
|
||||||
return float(action) * 2.0 * math.pi / LOW_NUM_ACTIONS
|
|
||||||
|
|
||||||
|
|
||||||
def navigation_transition(position: Vec2, goal: Vec2, action: int, *,
|
|
||||||
step_m: float = LOW_STEP_M) -> tuple[Vec2, float]:
|
|
||||||
"""训练与轨迹推演共享的运动学；返回位置和实际位移长度。
|
|
||||||
|
|
||||||
方向仍由策略决定，近目标时仅缩短步长，不把位置吸附到目标。
|
|
||||||
这是无障碍平面导航实验；不引入观测不可见的绝对位置裁剪。
|
|
||||||
"""
|
|
||||||
if not 0 <= int(action) < LOW_NUM_ACTIONS:
|
|
||||||
raise ValueError("低层动作必须是 0..7")
|
|
||||||
if not math.isfinite(step_m) or step_m <= 0:
|
|
||||||
raise ValueError("step_m 必须是有限正数")
|
|
||||||
distance = min(float(step_m), math.dist(position, goal))
|
|
||||||
bearing = action_bearing(int(action))
|
|
||||||
next_position = (position[0] + distance * math.cos(bearing),
|
|
||||||
position[1] + distance * math.sin(bearing))
|
|
||||||
return next_position, math.dist(position, next_position)
|
|
||||||
|
|
||||||
|
|
||||||
def build_low_observation(position: Vec2, goal: Vec2, prev_action: int | None,
|
|
||||||
steps: int) -> np.ndarray:
|
|
||||||
"""低层观测（shape=(LOW_OBS_DIM,)，训练/推理逐位一致）。"""
|
|
||||||
dx, dy = goal[0] - position[0], goal[1] - position[1]
|
|
||||||
distance = math.hypot(dx, dy)
|
|
||||||
bearing = math.atan2(dy, dx)
|
|
||||||
prev_sin = prev_cos = 0.0
|
|
||||||
if prev_action is not None:
|
|
||||||
prev_bearing = action_bearing(prev_action)
|
|
||||||
prev_sin, prev_cos = math.sin(prev_bearing), math.cos(prev_bearing)
|
|
||||||
return np.asarray([
|
|
||||||
dx / POS_SCALE_M,
|
|
||||||
dy / POS_SCALE_M,
|
|
||||||
distance / POS_SCALE_M,
|
|
||||||
math.sin(bearing),
|
|
||||||
math.cos(bearing),
|
|
||||||
steps / LOW_MAX_STEPS,
|
|
||||||
prev_sin,
|
|
||||||
prev_cos,
|
|
||||||
min(distance / LOW_STEP_M, 3.0) / 3.0,
|
|
||||||
1.0,
|
|
||||||
], dtype=np.float32)
|
|
||||||
|
|
||||||
|
|
||||||
class LowLevelExecutor:
|
|
||||||
"""子目标到达推演：低层 PPO 或直线控制器（policy=None）。"""
|
|
||||||
|
|
||||||
def __init__(self, policy=None, *, step_m: float = LOW_STEP_M,
|
|
||||||
reach_m: float = LOW_REACH_M, max_steps: int = LOW_MAX_STEPS):
|
|
||||||
self.policy = policy
|
|
||||||
self.step_m = float(step_m)
|
|
||||||
self.reach_m = float(reach_m)
|
|
||||||
self.max_steps = int(max_steps)
|
|
||||||
self.last_roll_info: dict = {}
|
|
||||||
|
|
||||||
def roll(self, start: Vec2, goal: Vec2) -> list[Vec2]:
|
|
||||||
"""返回真实推演途经点；last_roll_info 如实记录到达/截断。"""
|
|
||||||
goal = (float(goal[0]), float(goal[1]))
|
|
||||||
if self.policy is None:
|
|
||||||
self.last_roll_info = {"reached": True, "truncated": False,
|
|
||||||
"final_distance_m": 0.0,
|
|
||||||
"path_length_m": math.dist(start, goal)}
|
|
||||||
return [goal]
|
|
||||||
position = (float(start[0]), float(start[1]))
|
|
||||||
waypoints: list[Vec2] = []
|
|
||||||
prev_action: int | None = None
|
|
||||||
path_length = 0.0
|
|
||||||
for steps in range(self.max_steps):
|
|
||||||
if math.dist(position, goal) <= self.reach_m:
|
|
||||||
break
|
|
||||||
observation = build_low_observation(position, goal, prev_action,
|
|
||||||
steps)
|
|
||||||
action = int(self.policy.select(observation))
|
|
||||||
position, moved = navigation_transition(position, goal, action,
|
|
||||||
step_m=self.step_m)
|
|
||||||
path_length += moved
|
|
||||||
waypoints.append(position)
|
|
||||||
prev_action = action
|
|
||||||
distance = math.dist(position, goal)
|
|
||||||
self.last_roll_info = {"reached": distance <= self.reach_m,
|
|
||||||
"truncated": distance > self.reach_m,
|
|
||||||
"final_distance_m": distance,
|
|
||||||
"path_length_m": path_length}
|
|
||||||
return waypoints
|
|
||||||
@@ -1,260 +0,0 @@
|
|||||||
"""高层 PPO 训练环境：整场 10–16 源的发现-定位-清除调度（prompt §2/§5.2）。
|
|
||||||
|
|
||||||
与 v2 :class:`~mcm26b.rl.env.LocalizationEnv`（单源 ≤16 步子任务）的本质
|
|
||||||
差异：本环境的每一步是一次完整的高层调度决策（候选槽位选择 + 任务执行），
|
|
||||||
环境语义与运行闭环 :class:`~mcm26b.planner.hrl_loop.HRLLoop` 共用同一
|
|
||||||
混入层（hrl.scheduler），训练-部署观测逐位同源。
|
|
||||||
|
|
||||||
奖励（prompt §5.2 的可折算实现，权重见 :class:`HRLReward`）：
|
|
||||||
r = -Δ虚拟耗时/20 + 100·清除 + 5·新发现 + 0.5·Σ熵减 - 2·冗余检测
|
|
||||||
终止：全部频道清除/证明缺席（+50）；截断仍有未决频道（-100）；异常（-100）。
|
|
||||||
主口径是虚拟耗时——与题目目标函数（最小化平均定位清除时间/虚拟总时间）
|
|
||||||
一致，清除/发现/熵减是塑形项（消融实验逐项验证其必要性）。
|
|
||||||
|
|
||||||
真值只在 reset 布景时使用，绝不进入观测；感知读数全部来自
|
|
||||||
referee.LocalSimulator（与官方 HTTP 模拟器同协议语义）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from random import Random
|
|
||||||
|
|
||||||
import gymnasium as gym
|
|
||||||
import numpy as np
|
|
||||||
from gymnasium import spaces
|
|
||||||
|
|
||||||
from ..belief import ChannelStatus
|
|
||||||
from ..config import ProblemConfig
|
|
||||||
from ..hrl.curriculum import STAGES, Stage, make_scene, training_config
|
|
||||||
from ..hrl.constants import HIGH_OBS_DIM, N_CANDIDATES
|
|
||||||
from ..planner.hrl_loop import HRLLoop
|
|
||||||
from ..protocol.client import SimulatorClient
|
|
||||||
from ..protocol.log import SessionLog
|
|
||||||
|
|
||||||
__all__ = ["HRLReward", "HierarchicalEnv"]
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class HRLReward:
|
|
||||||
"""高层奖励权重。
|
|
||||||
|
|
||||||
默认保留 prompt §5.2 的可解释塑形，便于对照实验；P0.4 建议的
|
|
||||||
纯时间口径可通过 :meth:`time_only` 显式启用，训练脚本默认使用它。
|
|
||||||
"""
|
|
||||||
|
|
||||||
time_scale: float = 20.0 # r -= Δ虚拟耗时 / time_scale
|
|
||||||
clear: float = 100.0 # 每清除一个源
|
|
||||||
detect: float = 5.0 # 每新发现一个频道有信号
|
|
||||||
info_weight: float = 0.5 # 粒子云熵减（log 幸存格数）内在奖励
|
|
||||||
redundant: float = 2.0 # 同频道同点重复检测惩罚
|
|
||||||
invalid: float = 2.0 # 掩码外动作（无掩码消融路径）
|
|
||||||
terminal_accounted: float = 50.0 # 全部频道清除/证毕的收尾奖励
|
|
||||||
terminal_unresolved: float = 100.0 # 截断/停机时仍有未决频道的惩罚
|
|
||||||
failure: float = 100.0 # 协议/守恒性异常惩罚
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def time_only(cls) -> "HRLReward":
|
|
||||||
"""P0.4 纯时间目标：只惩罚已发生虚拟时间，去掉塑形项。
|
|
||||||
|
|
||||||
终止奖/罚是固定常数，不逐源叠加；失败罚按虚拟时限/时间尺度
|
|
||||||
设定为足以覆盖一整局时间成本。
|
|
||||||
"""
|
|
||||||
return cls(
|
|
||||||
time_scale=100.0,
|
|
||||||
clear=0.0,
|
|
||||||
detect=0.0,
|
|
||||||
info_weight=0.0,
|
|
||||||
redundant=0.0,
|
|
||||||
invalid=2.0,
|
|
||||||
terminal_accounted=30.0,
|
|
||||||
terminal_unresolved=300.0,
|
|
||||||
failure=3600.0,
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
class HierarchicalEnv(gym.Env):
|
|
||||||
"""整场分层调度 Gymnasium 环境（问题3/4，课程参数化）。
|
|
||||||
|
|
||||||
stage=None 时使用 STAGES 末阶段（完整任务口径）；课程训练按阶段
|
|
||||||
传入。domain_randomization=False 保留随机位置但固定半径代表值，
|
|
||||||
供配对评估。
|
|
||||||
"""
|
|
||||||
|
|
||||||
metadata = {"render_modes": []}
|
|
||||||
|
|
||||||
def __init__(self, config: ProblemConfig | None = None, *,
|
|
||||||
problem: int = 3, stage: Stage | None = None,
|
|
||||||
max_episode_steps: int = 300,
|
|
||||||
domain_randomization: bool = True,
|
|
||||||
reward: HRLReward | None = None,
|
|
||||||
real_budget_s: float = 3600.0,
|
|
||||||
error_seed: int | None = None,
|
|
||||||
fidelity: str = "production"):
|
|
||||||
super().__init__()
|
|
||||||
if problem not in (3, 4):
|
|
||||||
raise ValueError("problem 必须是 3 或 4")
|
|
||||||
self.base_config = config or ProblemConfig()
|
|
||||||
self.problem = int(problem)
|
|
||||||
self.stage = stage or STAGES[-1]
|
|
||||||
self.max_episode_steps = int(max_episode_steps)
|
|
||||||
self.domain_randomization = bool(domain_randomization)
|
|
||||||
self.reward = reward or HRLReward()
|
|
||||||
self.real_budget_s = float(real_budget_s)
|
|
||||||
self.error_seed = error_seed
|
|
||||||
self.fidelity = fidelity
|
|
||||||
# 默认与部署同精度；fast仅减少评分采样，保留证书网格精度。
|
|
||||||
self.train_config = training_config(self.base_config, self.stage,
|
|
||||||
fidelity=fidelity)
|
|
||||||
self.observation_space = spaces.Box(low=-12.0, high=12.0,
|
|
||||||
shape=(HIGH_OBS_DIM,),
|
|
||||||
dtype=np.float32)
|
|
||||||
self.action_space = spaces.Discrete(N_CANDIDATES)
|
|
||||||
|
|
||||||
self._scene_rng = Random()
|
|
||||||
self._loop: HRLLoop | None = None
|
|
||||||
self._client: SimulatorClient | None = None
|
|
||||||
self._steps = 0
|
|
||||||
self._terminated = False
|
|
||||||
self._truncated = False
|
|
||||||
self._start_virtual_s = 0.0
|
|
||||||
self._current_candidates: list = []
|
|
||||||
self._failure_reason: str | None = None
|
|
||||||
self._last_outcome = None
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 场景
|
|
||||||
def _make_episode(self) -> None:
|
|
||||||
from ..harness import LocalSimTransport
|
|
||||||
from referee import LocalSimulator
|
|
||||||
|
|
||||||
sources, _channels = make_scene(
|
|
||||||
self._scene_rng, self.train_config, self.problem, self.stage,
|
|
||||||
domain_randomization=self.domain_randomization)
|
|
||||||
self._simulator = LocalSimulator(
|
|
||||||
sources, robot_id="hrl-train",
|
|
||||||
spec=self._training_spec(),
|
|
||||||
error_seed=(self.error_seed if self.error_seed is not None
|
|
||||||
else self._scene_rng.randint(0, 2 ** 31 - 1)),
|
|
||||||
stress_mode=True)
|
|
||||||
self._client = SimulatorClient(
|
|
||||||
"hrl-train", config=self.train_config,
|
|
||||||
transport=LocalSimTransport(self._simulator), log=SessionLog())
|
|
||||||
self._client.enter()
|
|
||||||
self._loop = HRLLoop(self._client, self.train_config,
|
|
||||||
problem=self.problem)
|
|
||||||
|
|
||||||
def _training_spec(self):
|
|
||||||
from ..hrl.curriculum import training_spec
|
|
||||||
return training_spec(self.train_config, self.stage,
|
|
||||||
max_real_duration_s=self.real_budget_s)
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ Gym API
|
|
||||||
def reset(self, *, seed: int | None = None, options: dict | None = None):
|
|
||||||
super().reset(seed=seed)
|
|
||||||
if seed is not None:
|
|
||||||
self._scene_rng = Random(seed)
|
|
||||||
self._steps = 0
|
|
||||||
self._terminated = False
|
|
||||||
self._truncated = False
|
|
||||||
self._failure_reason = None
|
|
||||||
self._make_episode()
|
|
||||||
self._start_virtual_s = self._client.state.virtual_time_s
|
|
||||||
self._current_candidates = self._loop.build_candidates()
|
|
||||||
observation = self._loop.hrl_observation(self._current_candidates)
|
|
||||||
return observation, self._info()
|
|
||||||
|
|
||||||
def action_masks(self) -> np.ndarray:
|
|
||||||
"""MaskablePPO 动作掩码接口（与观测同源的候选槽位有效性）。"""
|
|
||||||
return self._loop.hrl_action_mask(self._current_candidates)
|
|
||||||
|
|
||||||
def _info(self) -> dict:
|
|
||||||
loop = self._loop
|
|
||||||
belief = loop.belief
|
|
||||||
cleared = len(loop.cleared)
|
|
||||||
absent = sum(1 for ch in belief.channels.values()
|
|
||||||
if ch.status is ChannelStatus.ABSENT)
|
|
||||||
return {
|
|
||||||
"steps": self._steps,
|
|
||||||
"cleared": cleared,
|
|
||||||
"absent_proven": absent,
|
|
||||||
"accounted": cleared + absent,
|
|
||||||
"virtual_time_s": self._client.state.virtual_time_s,
|
|
||||||
"episode_virtual_cost_s": (self._client.state.virtual_time_s
|
|
||||||
- self._start_virtual_s),
|
|
||||||
"stop_reason": loop.stop_reason,
|
|
||||||
"failure_reason": self._failure_reason,
|
|
||||||
"anomalies": list(loop.anomalies),
|
|
||||||
}
|
|
||||||
|
|
||||||
def step(self, action):
|
|
||||||
if self._terminated or self._truncated:
|
|
||||||
raise RuntimeError("step() 需要在 terminated/truncated 后先 reset()")
|
|
||||||
loop = self._loop
|
|
||||||
self._steps += 1
|
|
||||||
try:
|
|
||||||
outcome = loop.hrl_step(action, candidates=self._current_candidates)
|
|
||||||
except Exception as exc: # 协议/守恒性异常：与 v2 环境同口径收尾
|
|
||||||
self._failure_reason = type(exc).__name__ + ': ' + str(exc)
|
|
||||||
observation = np.zeros((HIGH_OBS_DIM,), dtype=np.float32)
|
|
||||||
self._terminated = True
|
|
||||||
return (observation, -self.reward.failure, True, False,
|
|
||||||
self._info())
|
|
||||||
self._last_outcome = outcome
|
|
||||||
|
|
||||||
if outcome.invalid:
|
|
||||||
reward = -self.reward.invalid
|
|
||||||
else:
|
|
||||||
reward = self._reward_from(outcome)
|
|
||||||
|
|
||||||
# 终止判定：停机原因/全部证毕；或候选耗尽时走运行闭环 run() 的
|
|
||||||
# 同款安全网（覆盖缺席标记 + 阻塞定位清除），安全网的虚拟耗时
|
|
||||||
# 计入本步奖励（归因给促成该局面的决策序列）。
|
|
||||||
terminated = loop.stop_reason is not None or loop._all_accounted()
|
|
||||||
if not terminated:
|
|
||||||
self._current_candidates = loop.build_candidates()
|
|
||||||
if not self._current_candidates:
|
|
||||||
virtual_before = self._client.state.virtual_time_s
|
|
||||||
loop._mark_absent_after_coverage()
|
|
||||||
loop._resolve_detected()
|
|
||||||
reward -= (self._client.state.virtual_time_s - virtual_before) \
|
|
||||||
/ self.reward.time_scale
|
|
||||||
self._current_candidates = []
|
|
||||||
terminated = True
|
|
||||||
else:
|
|
||||||
self._current_candidates = []
|
|
||||||
|
|
||||||
if terminated:
|
|
||||||
reward += (self.reward.terminal_accounted
|
|
||||||
if loop._all_accounted()
|
|
||||||
else -self.reward.terminal_unresolved)
|
|
||||||
truncated = False
|
|
||||||
if not terminated and self._steps >= self.max_episode_steps:
|
|
||||||
truncated = True
|
|
||||||
if not loop._all_accounted():
|
|
||||||
reward -= self.reward.terminal_unresolved
|
|
||||||
self._terminated = terminated
|
|
||||||
self._truncated = truncated
|
|
||||||
if terminated or truncated:
|
|
||||||
observation = np.zeros((HIGH_OBS_DIM,), dtype=np.float32)
|
|
||||||
else:
|
|
||||||
self._current_candidates = loop.build_candidates()
|
|
||||||
observation = loop.hrl_observation(self._current_candidates)
|
|
||||||
return observation, float(reward), terminated, truncated, self._info()
|
|
||||||
|
|
||||||
def _reward_from(self, outcome) -> float:
|
|
||||||
reward = -outcome.delta_virtual_s / self.reward.time_scale
|
|
||||||
reward += self.reward.clear * len(outcome.clear_success)
|
|
||||||
reward += self.reward.detect * len(outcome.new_detections)
|
|
||||||
reward += self.reward.info_weight * sum(outcome.info_gains.values())
|
|
||||||
if outcome.redundant:
|
|
||||||
reward -= self.reward.redundant
|
|
||||||
return reward
|
|
||||||
|
|
||||||
def close(self):
|
|
||||||
client = getattr(self, "_client", None)
|
|
||||||
if client is not None and client.state.entered \
|
|
||||||
and not client.state.exited:
|
|
||||||
try:
|
|
||||||
client.exit()
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
@@ -1,158 +0,0 @@
|
|||||||
"""低层 PPO 训练环境：随机起点-子目标的 8 方向到达任务（prompt §4.2/§5.1）。
|
|
||||||
|
|
||||||
奖励为实际位移耗时惩罚、转向惩罚与到达奖励。步长上限 100m，近目标
|
|
||||||
缩短步长，避免固定步长振荡。此环境仅用于独立导航实验；正式 HTTP
|
|
||||||
路径尚不执行低层 PPO 的途经点，不能将导航成绩当成正式任务收益。
|
|
||||||
|
|
||||||
策略端只见相对位姿观测，真值（子目标坐标）经相对向量进入观测，
|
|
||||||
与部署执行器完全同源（build_low_observation）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import math
|
|
||||||
from random import Random
|
|
||||||
|
|
||||||
import gymnasium as gym
|
|
||||||
import numpy as np
|
|
||||||
from gymnasium import spaces
|
|
||||||
|
|
||||||
from ..config import ProblemConfig
|
|
||||||
from .constants import (LOW_MAX_STEPS, LOW_NUM_ACTIONS, LOW_OBS_DIM,
|
|
||||||
LOW_REACH_M, LOW_STEP_M)
|
|
||||||
from .executor import (action_bearing, build_low_observation,
|
|
||||||
navigation_transition)
|
|
||||||
|
|
||||||
__all__ = ["LowLevelEnv"]
|
|
||||||
|
|
||||||
|
|
||||||
class LowLevelEnv(gym.Env):
|
|
||||||
"""低层到达任务：观测 10 维，动作 Discrete(8)，逐步计费。"""
|
|
||||||
|
|
||||||
metadata = {"render_modes": []}
|
|
||||||
|
|
||||||
def __init__(self, config: ProblemConfig | None = None, *,
|
|
||||||
seed: int | None = None,
|
|
||||||
step_m: float = LOW_STEP_M, reach_m: float = LOW_REACH_M,
|
|
||||||
max_steps: int = LOW_MAX_STEPS,
|
|
||||||
min_distance_m: float = 300.0,
|
|
||||||
turn_penalty: float = 0.05, reach_bonus: float = 2.0,
|
|
||||||
step_cost_scale: float = 100.0,
|
|
||||||
truncation_penalty: float = 2.0):
|
|
||||||
super().__init__()
|
|
||||||
self.config = config or ProblemConfig()
|
|
||||||
self.step_m = float(step_m)
|
|
||||||
self.reach_m = float(reach_m)
|
|
||||||
self.max_steps = int(max_steps)
|
|
||||||
self.min_distance_m = float(min_distance_m)
|
|
||||||
self.turn_penalty = float(turn_penalty)
|
|
||||||
self.reach_bonus = float(reach_bonus)
|
|
||||||
self.step_cost_scale = float(step_cost_scale)
|
|
||||||
self.truncation_penalty = float(truncation_penalty)
|
|
||||||
self.observation_space = spaces.Box(low=-np.inf, high=np.inf,
|
|
||||||
shape=(LOW_OBS_DIM,),
|
|
||||||
dtype=np.float32)
|
|
||||||
self.action_space = spaces.Discrete(LOW_NUM_ACTIONS)
|
|
||||||
self._rng = Random(seed)
|
|
||||||
self._position = (0.0, 0.0)
|
|
||||||
self._goal = (0.0, 0.0)
|
|
||||||
self._steps = 0
|
|
||||||
self._prev_action: int | None = None
|
|
||||||
self._terminated = False
|
|
||||||
self._truncated = False
|
|
||||||
self._path_length_m = 0.0
|
|
||||||
self._virtual_time_s = 0.0
|
|
||||||
self._straight_m = 0.0
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ Gym API
|
|
||||||
def reset(self, *, seed: int | None = None, options: dict | None = None):
|
|
||||||
super().reset(seed=seed)
|
|
||||||
if seed is not None:
|
|
||||||
self._rng = Random(seed)
|
|
||||||
radius = self.config.field.omega_radius_m
|
|
||||||
self._position = self._sample_point(radius)
|
|
||||||
for _ in range(100):
|
|
||||||
self._goal = self._sample_point(radius)
|
|
||||||
if math.dist(self._position, self._goal) >= self.min_distance_m:
|
|
||||||
break
|
|
||||||
self._steps = 0
|
|
||||||
self._prev_action = None
|
|
||||||
self._terminated = False
|
|
||||||
self._truncated = False
|
|
||||||
self._path_length_m = 0.0
|
|
||||||
self._virtual_time_s = 0.0
|
|
||||||
self._straight_m = math.dist(self._position, self._goal)
|
|
||||||
return self._observe(), self._info()
|
|
||||||
|
|
||||||
def action_masks(self) -> np.ndarray:
|
|
||||||
"""无障碍低层八个方向均可用，兼容 MaskablePPO。"""
|
|
||||||
return np.ones(LOW_NUM_ACTIONS, dtype=bool)
|
|
||||||
|
|
||||||
def _sample_point(self, radius: float) -> tuple[float, float]:
|
|
||||||
angle = self._rng.uniform(0.0, 2.0 * math.pi)
|
|
||||||
distance = radius * math.sqrt(self._rng.random())
|
|
||||||
return (distance * math.cos(angle), distance * math.sin(angle))
|
|
||||||
|
|
||||||
def step(self, action):
|
|
||||||
if self._terminated or self._truncated:
|
|
||||||
raise RuntimeError("step() 需要在 terminated/truncated 后先 reset()")
|
|
||||||
action = int(action)
|
|
||||||
if self._prev_action is not None:
|
|
||||||
turn = abs((action_bearing(action)
|
|
||||||
- action_bearing(self._prev_action) + math.pi)
|
|
||||||
% (2.0 * math.pi) - math.pi) / math.pi
|
|
||||||
else:
|
|
||||||
turn = 0.0
|
|
||||||
self._position, moved = navigation_transition(
|
|
||||||
self._position, self._goal, action, step_m=self.step_m)
|
|
||||||
self._prev_action = action
|
|
||||||
self._steps += 1
|
|
||||||
|
|
||||||
step_cost_s = moved / self.config.cost.speed_mps
|
|
||||||
self._path_length_m += moved
|
|
||||||
self._virtual_time_s += step_cost_s
|
|
||||||
reward = -step_cost_s / self.step_cost_scale
|
|
||||||
reward -= self.turn_penalty * turn
|
|
||||||
distance = math.dist(self._position, self._goal)
|
|
||||||
terminated = distance <= self.reach_m
|
|
||||||
truncated = False
|
|
||||||
if terminated:
|
|
||||||
reward += self.reach_bonus
|
|
||||||
elif self._steps >= self.max_steps:
|
|
||||||
truncated = True
|
|
||||||
reward -= self.truncation_penalty
|
|
||||||
self._terminated = terminated
|
|
||||||
self._truncated = truncated
|
|
||||||
info = self._info()
|
|
||||||
info["final_distance_m"] = distance
|
|
||||||
info["step_distance_m"] = moved
|
|
||||||
info["step_virtual_time_s"] = step_cost_s
|
|
||||||
return self._observe(), float(reward), terminated, truncated, info
|
|
||||||
|
|
||||||
def _observe(self) -> np.ndarray:
|
|
||||||
return build_low_observation(self._position, self._goal,
|
|
||||||
self._prev_action, self._steps)
|
|
||||||
|
|
||||||
def _info(self) -> dict:
|
|
||||||
return {
|
|
||||||
"position": self._position,
|
|
||||||
"goal": self._goal,
|
|
||||||
"steps": self._steps,
|
|
||||||
"distance_m": math.dist(self._position, self._goal),
|
|
||||||
"straight_m": self._straight_m,
|
|
||||||
"path_length_m": self._path_length_m,
|
|
||||||
"virtual_time_s": self._virtual_time_s,
|
|
||||||
}
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 专家（BC）
|
|
||||||
def expert_action(self) -> int:
|
|
||||||
"""直线专家：选择方位最接近子目标方向的离散方向（BC 标签）。"""
|
|
||||||
bearing = math.atan2(self._goal[1] - self._position[1],
|
|
||||||
self._goal[0] - self._position[0])
|
|
||||||
best, best_gap = 0, float("inf")
|
|
||||||
for action in range(LOW_NUM_ACTIONS):
|
|
||||||
gap = abs((action_bearing(action) - bearing + math.pi)
|
|
||||||
% (2.0 * math.pi) - math.pi)
|
|
||||||
if gap < best_gap:
|
|
||||||
best, best_gap = action, gap
|
|
||||||
return best
|
|
||||||
@@ -1,135 +0,0 @@
|
|||||||
"""HRL 策略加载与推理（运行时接入 planner 的唯一入口，模式同 rl.policy）。
|
|
||||||
|
|
||||||
模型文件由 scripts/train_hrl_low.py / train_hrl_high.py 生成，并伴随同名
|
|
||||||
JSON 元数据（feature_version/algorithm/problem/维度）；加载时校验，失配
|
|
||||||
即拒绝运行，杜绝静默错位。stable-baselines3 / torch 均为惰性导入。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
|
|
||||||
from .constants import (ACTION_LAYOUT_VERSION, FEATURE_VERSION,
|
|
||||||
HIGH_OBS_DIM, LOW_NUM_ACTIONS, LOW_OBS_DIM,
|
|
||||||
N_CANDIDATES, REWARD_VERSION)
|
|
||||||
|
|
||||||
__all__ = ["HighLevelPolicy", "LowLevelPolicy",
|
|
||||||
"default_high_model_path", "default_low_model_path"]
|
|
||||||
|
|
||||||
|
|
||||||
def _resolve_device(device: str) -> str:
|
|
||||||
if device != "auto":
|
|
||||||
return device
|
|
||||||
try:
|
|
||||||
import torch
|
|
||||||
return "cuda" if torch.cuda.is_available() else "cpu"
|
|
||||||
except ImportError:
|
|
||||||
return "cpu"
|
|
||||||
|
|
||||||
|
|
||||||
def _load_maskable(model_path: Path, device: str, *,
|
|
||||||
obs_dim: int, act_dim: int, problem: int):
|
|
||||||
if not model_path.is_file():
|
|
||||||
raise RuntimeError(
|
|
||||||
f"未找到 HRL 模型 {model_path}。请先训练：\n"
|
|
||||||
f" uv sync --group rl --group dev\n"
|
|
||||||
f" uv run python scripts/train_hrl_high.py --problem {problem}")
|
|
||||||
try:
|
|
||||||
from sb3_contrib import MaskablePPO
|
|
||||||
except ImportError as exc:
|
|
||||||
raise RuntimeError(
|
|
||||||
"缺少 RL 运行依赖（sb3-contrib）。请执行 `uv sync --group rl` "
|
|
||||||
"后重试。") from exc
|
|
||||||
meta_path = model_path.with_suffix('.json')
|
|
||||||
metadata = json.loads(meta_path.read_text(encoding='utf-8')) \
|
|
||||||
if meta_path.exists() else {}
|
|
||||||
# 高层模型必须携带完整语义元数据；缺字段不能默认兼容，防止旧槽位/
|
|
||||||
# 旧奖励语义模型被静默加载。低层导航模型只校验基础特征版本。
|
|
||||||
strict = act_dim == N_CANDIDATES
|
|
||||||
if strict:
|
|
||||||
required = ("feature_version", "action_layout_version",
|
|
||||||
"reward_version", "algorithm", "problem",
|
|
||||||
"obs_dim", "act_dim")
|
|
||||||
missing = [key for key in required if key not in metadata]
|
|
||||||
if missing:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"HRL 高层模型元数据缺少字段 {missing}；请用当前 "
|
|
||||||
f"scripts/train_hrl_high.py 重训。")
|
|
||||||
if metadata.get("feature_version") != FEATURE_VERSION:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"HRL 模型特征版本失配: 期望 {FEATURE_VERSION}，"
|
|
||||||
f"实际 {metadata.get('feature_version')}。请用当前版本重训。")
|
|
||||||
if metadata.get("action_layout_version") != ACTION_LAYOUT_VERSION:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"HRL 模型动作布局版本失配: 期望 {ACTION_LAYOUT_VERSION}，"
|
|
||||||
f"实际 {metadata.get('action_layout_version')}。请重训。")
|
|
||||||
if metadata.get("reward_version") != REWARD_VERSION:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"HRL 模型奖励版本失配: 期望 {REWARD_VERSION}，"
|
|
||||||
f"实际 {metadata.get('reward_version')}。请重训。")
|
|
||||||
else:
|
|
||||||
if metadata.get("feature_version") != FEATURE_VERSION:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"低层模型特征版本失配: 期望 {FEATURE_VERSION}，"
|
|
||||||
f"实际 {metadata.get('feature_version')}。请重训。")
|
|
||||||
if metadata.get("algorithm") != "maskable_ppo":
|
|
||||||
raise RuntimeError("HRL 模型必须是 maskable_ppo 训练产物")
|
|
||||||
if problem is not None and metadata.get("problem") not in (None, problem):
|
|
||||||
raise RuntimeError("模型训练问题编号与当前第三/四问不匹配")
|
|
||||||
model = MaskablePPO.load(str(model_path), device=device)
|
|
||||||
try:
|
|
||||||
loaded_obs = model.observation_space.shape[0]
|
|
||||||
loaded_act = model.action_space.n
|
|
||||||
except Exception as exc:
|
|
||||||
raise RuntimeError(f"模型空间与 HRL 接口不兼容: {model_path}") from exc
|
|
||||||
if loaded_obs != obs_dim or loaded_act != act_dim:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"模型接口失配: 期望 obs={obs_dim}/act={act_dim}，"
|
|
||||||
f"实际 obs={loaded_obs}/act={loaded_act}。请重训。")
|
|
||||||
return model
|
|
||||||
|
|
||||||
|
|
||||||
class HighLevelPolicy:
|
|
||||||
"""高层子目标调度策略：观测+掩码 -> 槽位编号（确定性 argmax）。"""
|
|
||||||
|
|
||||||
def __init__(self, model_path: str | Path, *, device: str = "auto",
|
|
||||||
problem: int | None = None):
|
|
||||||
self.device = _resolve_device(device)
|
|
||||||
self.path = Path(model_path)
|
|
||||||
self.model = _load_maskable(self.path, self.device, obs_dim=HIGH_OBS_DIM,
|
|
||||||
act_dim=N_CANDIDATES, problem=problem)
|
|
||||||
meta_path = self.path.with_suffix('.json')
|
|
||||||
self.metadata = json.loads(meta_path.read_text(encoding='utf-8')) \
|
|
||||||
if meta_path.exists() else {}
|
|
||||||
|
|
||||||
def select(self, observation: np.ndarray, action_masks: np.ndarray) -> int:
|
|
||||||
action, _state = self.model.predict(
|
|
||||||
observation, action_masks=action_masks, deterministic=True)
|
|
||||||
return int(action)
|
|
||||||
|
|
||||||
|
|
||||||
class LowLevelPolicy:
|
|
||||||
"""低层到达策略：相对位姿观测 -> 8 方向（确定性 argmax）。"""
|
|
||||||
|
|
||||||
def __init__(self, model_path: str | Path, *, device: str = "auto"):
|
|
||||||
self.device = _resolve_device(device)
|
|
||||||
self.path = Path(model_path)
|
|
||||||
self.model = _load_maskable(self.path, self.device, obs_dim=LOW_OBS_DIM,
|
|
||||||
act_dim=LOW_NUM_ACTIONS, problem=None)
|
|
||||||
|
|
||||||
def select(self, observation: np.ndarray) -> int:
|
|
||||||
action, _state = self.model.predict(observation, deterministic=True)
|
|
||||||
return int(action)
|
|
||||||
|
|
||||||
|
|
||||||
def default_high_model_path(problem: int) -> Path:
|
|
||||||
if problem not in (3, 4):
|
|
||||||
raise ValueError("problem 必须是 3 或 4")
|
|
||||||
return Path("output/hrl") / f"q{problem}_hrl_high.zip"
|
|
||||||
|
|
||||||
|
|
||||||
def default_low_model_path() -> Path:
|
|
||||||
return Path("output/hrl") / "hrl_low.zip"
|
|
||||||
@@ -1,495 +0,0 @@
|
|||||||
"""高层调度核心：候选槽位构建、观测、动作掩码与任务执行。
|
|
||||||
|
|
||||||
本模块是 :class:`~mcm26b.planner.hrl_loop.HRLLoop`（运行闭环）与
|
|
||||||
:class:`~mcm26b.hrl.hierarchical_env.HierarchicalEnv`（训练环境）共用的
|
|
||||||
混入层，保证训练-部署特征逐位同源（v1"训练部署特征错位"审核教训）。
|
|
||||||
|
|
||||||
宿主须为 :class:`~mcm26b.planner.bayes_loop.BayesLoop` 派生闭环，提供
|
|
||||||
client/belief/particles/ledger/_record_measure 等感知与执行原语；本层只做
|
|
||||||
三件事——
|
|
||||||
|
|
||||||
1. 把整场调度决策离散化为固定槽位（constants.N_CANDIDATES）：
|
|
||||||
clear(20) + locate(20) + station(49) + fallback(20) + 保留(11)；
|
|
||||||
2. 由可见观测（belief + 网格幸存集 + 覆盖账本）构造观测与动作掩码，
|
|
||||||
绝不读取源真值；
|
|
||||||
3. 执行选中候选并报告事实（虚拟耗时增量、新发现、熵减、清除结果），
|
|
||||||
奖励折算由训练环境负责。
|
|
||||||
|
|
||||||
Q4 定向源语义：no_signal 不能删除空间候选（可能是朝向遮挡），故问题4 的
|
|
||||||
幸存集只按 direction/near 更新；遮挡证据走 blocked_probes 梯子与后备清除
|
|
||||||
槽位，与 DirectionalExtension 的保守口径一致。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import math
|
|
||||||
from dataclasses import dataclass, field
|
|
||||||
from typing import NamedTuple
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
|
|
||||||
from ..belief import ChannelStatus
|
|
||||||
from ..config import Vec2
|
|
||||||
from ..protocol.rules import CLEAR_SUCCESS, DIRECTION, NEAR, NO_SIGNAL
|
|
||||||
from ..protocol.timing import move_time
|
|
||||||
from .constants import (CHANNEL_COUNT, CLEAR_OFFSET, COST_SCALE_S,
|
|
||||||
FALLBACK_OFFSET, HIGH_GLOBAL_DIM, HIGH_OBS_DIM,
|
|
||||||
LOCATE_OFFSET, MEASURE_COUNT_SCALE, N_CANDIDATES,
|
|
||||||
PER_CANDIDATE_DIM, PER_CHANNEL_DIM, POS_SCALE_M,
|
|
||||||
RADIUS_SCALE_M, STATION_OFFSET)
|
|
||||||
from ..planner import fallback_clear_cells, safe_disk_radius
|
|
||||||
|
|
||||||
__all__ = ["Candidate", "StepOutcome", "HRLControllerMixin", "point_key"]
|
|
||||||
|
|
||||||
|
|
||||||
def point_key(point: Vec2) -> tuple[int, int]:
|
|
||||||
"""候选点去重键：0.5m 网格取整（与 bayes_loop 同口径）。"""
|
|
||||||
return (int(round(point[0] * 2)), int(round(point[1] * 2)))
|
|
||||||
|
|
||||||
|
|
||||||
class Candidate(NamedTuple):
|
|
||||||
"""一个高层候选子目标。slot 为固定槽位编号，供动作掩码与特征定位。"""
|
|
||||||
|
|
||||||
slot: int
|
|
||||||
kind: str # "clear" | "locate" | "probe" | "station" | "fallback"
|
|
||||||
position: Vec2
|
|
||||||
channel: int | None = None
|
|
||||||
station_index: int | None = None
|
|
||||||
service_s: float = 0.0 # 预计服务耗时（含换频，不含移动）
|
|
||||||
targets: int = 1 # station: 待测频道数；fallback: 格数；其余 1
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
|
||||||
class StepOutcome:
|
|
||||||
"""一次高层决策-执行的客观结果（奖励折算由训练环境负责）。"""
|
|
||||||
|
|
||||||
action: int
|
|
||||||
kind: str | None = None
|
|
||||||
channel: int | None = None
|
|
||||||
invalid: bool = False # 掩码外动作（无掩码消融路径）：未与模拟器交互
|
|
||||||
delta_virtual_s: float = 0.0
|
|
||||||
new_detections: tuple[int, ...] = ()
|
|
||||||
info_gains: dict[int, float] = field(default_factory=dict) # 频道 -> 熵减(log)
|
|
||||||
clear_success: tuple[int, ...] = ()
|
|
||||||
redundant: bool = False
|
|
||||||
position_reached: Vec2 | None = None
|
|
||||||
|
|
||||||
|
|
||||||
class HRLControllerMixin:
|
|
||||||
"""高层调度混入：依赖宿主（HRLLoop）提供 BayesLoop 感知与执行原语。
|
|
||||||
|
|
||||||
宿主须在 ``__init__`` 初始化：``_obs_mec_cache={}``、
|
|
||||||
``_station_pending_map={}``、``_initial_alive={channel: len(pf)}``
|
|
||||||
（观测侧缓存与归一化基准；证书相关计算仍用 pf.mec() 精确值）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
problem: int
|
|
||||||
# 由宿主 BayesLoop.__init__ 初始化；此处声明让 pylint 可见该诊断计数。
|
|
||||||
no_shrink_measures: int = 0
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 候选构建
|
|
||||||
def build_candidates(self) -> list[Candidate]:
|
|
||||||
"""构建当前全部有效候选（固定槽位；训练/推理/heuristic 专家同源）。
|
|
||||||
|
|
||||||
副作用：缓存每频道证书（``self._cert_cache``）与站点待测集
|
|
||||||
（``self._station_pending_map``）供观测构建复用；调用方须在本
|
|
||||||
函数之后、任何状态变更之前构建观测。
|
|
||||||
"""
|
|
||||||
config = self.config
|
|
||||||
detected = self.belief.channels_by_status(ChannelStatus.DETECTED)
|
|
||||||
by_slot: dict[int, Candidate] = {}
|
|
||||||
cert_cache: dict[int, object] = {}
|
|
||||||
station_map: dict[int, list[int]] = {}
|
|
||||||
|
|
||||||
# --- clear 槽位：证书就绪且未因失败等待补测
|
|
||||||
for channel in detected:
|
|
||||||
cert = self._best_certificate(channel)
|
|
||||||
cert_cache[channel] = cert
|
|
||||||
if cert is not None and cert.certified \
|
|
||||||
and channel not in self._clear_failed:
|
|
||||||
by_slot[CLEAR_OFFSET + channel - 1] = Candidate(
|
|
||||||
CLEAR_OFFSET + channel - 1, "clear", tuple(cert.center),
|
|
||||||
channel=channel,
|
|
||||||
service_s=config.cost.clear_success_s, targets=1)
|
|
||||||
|
|
||||||
# --- locate / probe 槽位：未就绪清除的已发现频道
|
|
||||||
for channel in detected:
|
|
||||||
if CLEAR_OFFSET + channel - 1 in by_slot:
|
|
||||||
continue
|
|
||||||
region = self.belief.candidate_set(channel)
|
|
||||||
if region is None:
|
|
||||||
continue
|
|
||||||
blocked = self.blocked_probes.get(channel, []) \
|
|
||||||
if self.problem == 4 else []
|
|
||||||
if blocked:
|
|
||||||
point = self._hrl_probe_point(region, channel)
|
|
||||||
kind = "probe"
|
|
||||||
else:
|
|
||||||
point = self._select_point(channel)
|
|
||||||
kind = "locate"
|
|
||||||
if point is None:
|
|
||||||
continue
|
|
||||||
by_slot[LOCATE_OFFSET + channel - 1] = Candidate(
|
|
||||||
LOCATE_OFFSET + channel - 1, kind, tuple(point), channel=channel,
|
|
||||||
service_s=config.cost.measure_s + config.cost.switch_s, targets=1)
|
|
||||||
|
|
||||||
# --- station 槽位：尚有 UNKNOWN 待测频道的覆盖站
|
|
||||||
for index, point in enumerate(self.ledger.points):
|
|
||||||
pending = self._station_pending(index)
|
|
||||||
station_map[index] = pending
|
|
||||||
if not pending:
|
|
||||||
continue
|
|
||||||
switches = len(pending) - (1 if self.client.state.channel in pending else 0)
|
|
||||||
by_slot[STATION_OFFSET + index] = Candidate(
|
|
||||||
STATION_OFFSET + index, "station", tuple(point),
|
|
||||||
station_index=index,
|
|
||||||
service_s=(config.cost.measure_s * len(pending)
|
|
||||||
+ config.cost.switch_s * max(switches, 0)),
|
|
||||||
targets=len(pending))
|
|
||||||
|
|
||||||
# --- fallback 槽位（仅问题4）：遮挡梯子耗尽且格数在预算内
|
|
||||||
if self.problem == 4:
|
|
||||||
for channel in detected:
|
|
||||||
blocked = self.blocked_probes.get(channel, [])
|
|
||||||
if len(blocked) < config.search.max_blocked_probes:
|
|
||||||
continue
|
|
||||||
region = self.belief.candidate_set(channel)
|
|
||||||
if region is None:
|
|
||||||
continue
|
|
||||||
cells = fallback_clear_cells(region.polygon, config)
|
|
||||||
if not cells or len(cells) > config.search.fallback_max_cells:
|
|
||||||
continue
|
|
||||||
by_slot[FALLBACK_OFFSET + channel - 1] = Candidate(
|
|
||||||
FALLBACK_OFFSET + channel - 1, "fallback",
|
|
||||||
tuple(region.mec()[0]), channel=channel,
|
|
||||||
service_s=config.cost.clear_fail_s * len(cells),
|
|
||||||
targets=len(cells))
|
|
||||||
|
|
||||||
self._cert_cache = cert_cache
|
|
||||||
self._station_pending_map = station_map
|
|
||||||
self._last_candidates = list(by_slot.values())
|
|
||||||
return self._last_candidates
|
|
||||||
|
|
||||||
def candidate_by_slot(self, candidates: list[Candidate],
|
|
||||||
action: int) -> Candidate | None:
|
|
||||||
for cand in candidates:
|
|
||||||
if cand.slot == int(action):
|
|
||||||
return cand
|
|
||||||
return None
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 动作掩码
|
|
||||||
def hrl_action_mask(self, candidates: list[Candidate]) -> np.ndarray:
|
|
||||||
"""槽位有效性掩码：无候选槽位恒屏蔽（训练/推理共用）。"""
|
|
||||||
mask = np.zeros(N_CANDIDATES, dtype=bool)
|
|
||||||
for cand in candidates:
|
|
||||||
mask[cand.slot] = True
|
|
||||||
return mask
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 观测
|
|
||||||
def _hrl_obs_mec(self, channel: int) -> tuple[Vec2, float]:
|
|
||||||
"""观测用 MEC：幸存子采样上的最小包围圆，按 pf.version 缓存。
|
|
||||||
|
|
||||||
仅作策略观测特征（近似 MEU/σ），证书相关计算一律用 pf.mec()
|
|
||||||
精确值——两者分开缓存，互不影响。
|
|
||||||
"""
|
|
||||||
pf = self.particles[channel]
|
|
||||||
cached = self._obs_mec_cache.get(channel)
|
|
||||||
if cached is not None and cached[0] == pf.version:
|
|
||||||
return cached[1], cached[2]
|
|
||||||
points = pf.subsample(600)
|
|
||||||
if not points:
|
|
||||||
return (0.0, 0.0), 0.0
|
|
||||||
from ..geometry.localization import smallest_enclosing_circle
|
|
||||||
center, radius = smallest_enclosing_circle(points)
|
|
||||||
self._obs_mec_cache[channel] = (pf.version, center, radius)
|
|
||||||
return center, radius
|
|
||||||
|
|
||||||
def _hrl_channel_status_index(self, channel: int) -> int:
|
|
||||||
"""六状态编码：UNKNOWN/DETECTED/TRACKING/LOCATED/CLEARED/ABSENT。"""
|
|
||||||
status = self.belief.channels[channel].status
|
|
||||||
if status is ChannelStatus.UNKNOWN:
|
|
||||||
return 0
|
|
||||||
if status is ChannelStatus.CLEARED:
|
|
||||||
return 4
|
|
||||||
if status is ChannelStatus.ABSENT:
|
|
||||||
return 5
|
|
||||||
cert = getattr(self, "_cert_cache", {}).get(channel)
|
|
||||||
if cert is not None and getattr(cert, "certified", False):
|
|
||||||
return 3
|
|
||||||
_, radius = self._hrl_obs_mec(channel)
|
|
||||||
return 1 if radius > 500.0 else 2
|
|
||||||
|
|
||||||
def hrl_observation(self, candidates: list[Candidate]) -> np.ndarray:
|
|
||||||
"""构造高层观测（shape=(HIGH_OBS_DIM,)，训练/推理逐位一致）。"""
|
|
||||||
config = self.config
|
|
||||||
field_cfg = config.field
|
|
||||||
position = self.client.state.position
|
|
||||||
belief = self.belief
|
|
||||||
|
|
||||||
detected_count = len(belief.channels_by_status(ChannelStatus.DETECTED))
|
|
||||||
accounted = sum(1 for ch in belief.channels.values()
|
|
||||||
if ch.status in (ChannelStatus.CLEARED,
|
|
||||||
ChannelStatus.ABSENT))
|
|
||||||
cleared = len(self.cleared)
|
|
||||||
glob = [
|
|
||||||
position[0] / POS_SCALE_M,
|
|
||||||
position[1] / POS_SCALE_M,
|
|
||||||
min(self.client.state.virtual_time_s, field_cfg.max_virtual_duration_s)
|
|
||||||
/ field_cfg.max_virtual_duration_s,
|
|
||||||
cleared / 16.0,
|
|
||||||
accounted / CHANNEL_COUNT,
|
|
||||||
(self.client.state.channel - field_cfg.channel_min)
|
|
||||||
/ max(field_cfg.channel_max - field_cfg.channel_min, 1),
|
|
||||||
float(self.problem == 4),
|
|
||||||
detected_count / 16.0,
|
|
||||||
]
|
|
||||||
assert len(glob) == HIGH_GLOBAL_DIM
|
|
||||||
|
|
||||||
pending_station = np.zeros(CHANNEL_COUNT + 1, dtype=bool)
|
|
||||||
for channels in getattr(self, "_station_pending_map", {}).values():
|
|
||||||
for channel in channels:
|
|
||||||
pending_station[channel] = True
|
|
||||||
|
|
||||||
channel_features: list[float] = []
|
|
||||||
for channel in range(field_cfg.channel_min, field_cfg.channel_max + 1):
|
|
||||||
ch = belief.channels[channel]
|
|
||||||
status6 = [0.0] * 6
|
|
||||||
status6[self._hrl_channel_status_index(channel)] = 1.0
|
|
||||||
pf = self.particles[channel]
|
|
||||||
mec_visible = ch.status is ChannelStatus.DETECTED
|
|
||||||
if mec_visible:
|
|
||||||
center, radius = self._hrl_obs_mec(channel)
|
|
||||||
else:
|
|
||||||
center, radius = (0.0, 0.0), 0.0
|
|
||||||
initial = max(self._initial_alive.get(channel, 1), 1)
|
|
||||||
channel_features.extend([
|
|
||||||
*status6,
|
|
||||||
center[0] / POS_SCALE_M,
|
|
||||||
center[1] / POS_SCALE_M,
|
|
||||||
radius / RADIUS_SCALE_M,
|
|
||||||
math.log(len(pf) + 1) / math.log(initial + 1)
|
|
||||||
if initial > 1 else 0.0,
|
|
||||||
min(len(ch.measurements), MEASURE_COUNT_SCALE * 2)
|
|
||||||
/ MEASURE_COUNT_SCALE / 2,
|
|
||||||
math.hypot(center[0] - position[0], center[1] - position[1])
|
|
||||||
/ POS_SCALE_M if mec_visible else 0.0,
|
|
||||||
len(self.blocked_probes.get(channel, []))
|
|
||||||
/ max(config.search.max_blocked_probes, 1)
|
|
||||||
if self.problem == 4 else 0.0,
|
|
||||||
1.0 if (ch.status is ChannelStatus.UNKNOWN
|
|
||||||
and pending_station[channel]) else 0.0,
|
|
||||||
])
|
|
||||||
assert len(channel_features) == CHANNEL_COUNT * PER_CHANNEL_DIM
|
|
||||||
|
|
||||||
by_slot = {cand.slot: cand for cand in candidates}
|
|
||||||
candidate_features: list[float] = []
|
|
||||||
for slot in range(N_CANDIDATES):
|
|
||||||
cand = by_slot.get(slot)
|
|
||||||
if cand is None:
|
|
||||||
candidate_features.extend([0.0] * PER_CANDIDATE_DIM)
|
|
||||||
continue
|
|
||||||
dx, dy = cand.position[0] - position[0], cand.position[1] - position[1]
|
|
||||||
move_s = move_time(position, cand.position, config.cost)
|
|
||||||
candidate_features.extend([
|
|
||||||
1.0 if cand.kind == "clear" else 0.0,
|
|
||||||
1.0 if cand.kind in ("locate", "probe") else 0.0,
|
|
||||||
1.0 if cand.kind == "station" else 0.0,
|
|
||||||
1.0 if cand.kind == "fallback" else 0.0,
|
|
||||||
dx / POS_SCALE_M,
|
|
||||||
dy / POS_SCALE_M,
|
|
||||||
min((move_s + cand.service_s) / COST_SCALE_S, 2.0),
|
|
||||||
min(cand.targets / 16.0, 1.0),
|
|
||||||
])
|
|
||||||
assert len(candidate_features) == N_CANDIDATES * PER_CANDIDATE_DIM
|
|
||||||
|
|
||||||
observation = np.asarray(glob + channel_features + candidate_features,
|
|
||||||
dtype=np.float32)
|
|
||||||
assert observation.shape == (HIGH_OBS_DIM,)
|
|
||||||
return observation
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 启发式专家
|
|
||||||
def heuristic_action(self, candidates: list[Candidate]) -> int:
|
|
||||||
"""把 BayesLoop 任务排序映射为槽位（BC 专家 + 无策略降级路径）。
|
|
||||||
|
|
||||||
任务表与排序完全复用 :meth:`BayesLoop._build_tasks` /
|
|
||||||
:meth:`BayesLoop._order_tasks`；映射失败（如 Q4 梯子任务不在
|
|
||||||
任务表中）退化为最近有效候选。
|
|
||||||
"""
|
|
||||||
by_slot = {cand.slot: cand for cand in candidates}
|
|
||||||
if not by_slot:
|
|
||||||
raise RuntimeError("heuristic_action：当前无可用候选")
|
|
||||||
try:
|
|
||||||
order = self._order_tasks(self._build_tasks())
|
|
||||||
first = order[0]
|
|
||||||
if first.kind == "clear" and first.channel is not None:
|
|
||||||
slot = CLEAR_OFFSET + first.channel - 1
|
|
||||||
elif first.kind == "locate" and first.channel is not None:
|
|
||||||
slot = LOCATE_OFFSET + first.channel - 1
|
|
||||||
else:
|
|
||||||
slot = STATION_OFFSET + (first.station_index or 0)
|
|
||||||
if slot in by_slot:
|
|
||||||
return slot
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
position = self.client.state.position
|
|
||||||
return min(by_slot, key=lambda s: (
|
|
||||||
math.dist(position, by_slot[s].position), s))
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 决策执行
|
|
||||||
def hrl_step(self, action: int,
|
|
||||||
candidates: list[Candidate] | None = None) -> StepOutcome:
|
|
||||||
"""执行一次高层决策；candidates 为空则现场构建（训练环境逐位同源）。"""
|
|
||||||
action = int(action)
|
|
||||||
self._reset_step_recording()
|
|
||||||
old_virtual = self.client.state.virtual_time_s
|
|
||||||
if candidates is None:
|
|
||||||
candidates = self.build_candidates()
|
|
||||||
cand = self.candidate_by_slot(candidates, action)
|
|
||||||
if cand is None:
|
|
||||||
return StepOutcome(action=action, invalid=True)
|
|
||||||
if self.stop_reason is not None:
|
|
||||||
return StepOutcome(action=action, kind=cand.kind,
|
|
||||||
channel=cand.channel, invalid=True)
|
|
||||||
|
|
||||||
if cand.kind == "station":
|
|
||||||
self._execute_station(cand.station_index)
|
|
||||||
elif cand.kind in ("locate", "probe"):
|
|
||||||
self._hrl_execute_locate(cand)
|
|
||||||
elif cand.kind == "clear":
|
|
||||||
self._hrl_execute_clear(cand)
|
|
||||||
elif cand.kind == "fallback":
|
|
||||||
self._hrl_execute_fallback(cand)
|
|
||||||
else: # pragma: no cover - 布局守卫
|
|
||||||
raise ValueError(f"未知候选类型: {cand.kind}")
|
|
||||||
|
|
||||||
outcome = StepOutcome(action=action, kind=cand.kind,
|
|
||||||
channel=cand.channel,
|
|
||||||
position_reached=tuple(self.client.state.position))
|
|
||||||
outcome.delta_virtual_s = self.client.state.virtual_time_s - old_virtual
|
|
||||||
new_detections: list[int] = []
|
|
||||||
info_gains: dict[int, float] = {}
|
|
||||||
redundant = False
|
|
||||||
for event in self._step_events:
|
|
||||||
if event["was_unknown"] and event["result"] in (DIRECTION, NEAR):
|
|
||||||
new_detections.append(event["channel"])
|
|
||||||
gain = math.log10(event["pf_before"] + 1) - math.log10(event["pf_after"] + 1)
|
|
||||||
info_gains[event["channel"]] = (info_gains.get(event["channel"], 0.0)
|
|
||||||
+ min(max(gain, 0.0), 2.0))
|
|
||||||
redundant = redundant or event["redundant"]
|
|
||||||
outcome.new_detections = tuple(new_detections)
|
|
||||||
outcome.info_gains = info_gains
|
|
||||||
outcome.redundant = redundant
|
|
||||||
outcome.clear_success = tuple(self._step_clears)
|
|
||||||
self._step_clears = []
|
|
||||||
return outcome
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 执行原语
|
|
||||||
def _hrl_execute_locate(self, cand: Candidate) -> None:
|
|
||||||
"""定位测量（含 Q4 梯子探点）：记录证据，不自动清除（高层决策）。"""
|
|
||||||
config = self.config
|
|
||||||
channel = cand.channel
|
|
||||||
pf = self.particles[channel]
|
|
||||||
before = len(pf)
|
|
||||||
outcome = self.client.measure(cand.position, channel)
|
|
||||||
self._record_measure(channel, cand.position, outcome)
|
|
||||||
self._may_clear.add(channel)
|
|
||||||
if self.problem == 4:
|
|
||||||
if outcome.result == NO_SIGNAL:
|
|
||||||
if channel not in self.directional_observed:
|
|
||||||
region = self.belief.candidate_set(channel)
|
|
||||||
safe = safe_disk_radius(region, config) if region else -1.0
|
|
||||||
if safe >= 0 and math.dist(cand.position, region.mec()[0]) <= safe:
|
|
||||||
self.directional_observed.append(channel)
|
|
||||||
self.blocked_probes.setdefault(channel, []).append(cand.position)
|
|
||||||
else:
|
|
||||||
self.blocked_probes.pop(channel, None)
|
|
||||||
if len(pf) == before:
|
|
||||||
self._stall[channel] = self._stall.get(channel, 0) + 1
|
|
||||||
self.no_shrink_measures += 1
|
|
||||||
else:
|
|
||||||
self._stall[channel] = 0
|
|
||||||
|
|
||||||
def _hrl_execute_clear(self, cand: Candidate) -> None:
|
|
||||||
"""证书清除：证书失效（不应被掩码允许）时静默跳过。"""
|
|
||||||
cert = self._best_certificate(cand.channel)
|
|
||||||
if cert is None or not cert.certified:
|
|
||||||
self._anomaly("hrl_clear_without_certificate", cand.channel)
|
|
||||||
return
|
|
||||||
if self._attempt_clear(cand.channel, cert.center, cert):
|
|
||||||
self._clear_failed.discard(cand.channel)
|
|
||||||
self._stall.pop(cand.channel, None)
|
|
||||||
else:
|
|
||||||
self._clear_failed.add(cand.channel)
|
|
||||||
self._may_clear.discard(cand.channel)
|
|
||||||
|
|
||||||
def _hrl_execute_fallback(self, cand: Candidate) -> None:
|
|
||||||
"""Q4 后备网格清除（与 DirectionalExtension._fallback_clear 同口径）。"""
|
|
||||||
channel = cand.channel
|
|
||||||
region = self.belief.candidate_set(channel)
|
|
||||||
if region is None:
|
|
||||||
return
|
|
||||||
config, session = self.config, self.config.session
|
|
||||||
cells = fallback_clear_cells(region.polygon, config)
|
|
||||||
entry = {"channel": channel, "cells": len(cells)}
|
|
||||||
self.fallback_log.append(entry)
|
|
||||||
if len(cells) > config.search.fallback_max_cells:
|
|
||||||
entry["outcome"] = "skipped_cell_cap"
|
|
||||||
return
|
|
||||||
remaining = self.client.real_remaining_s
|
|
||||||
if remaining is not None and len(cells) * session.real_rtt_p95_s \
|
|
||||||
+ session.reserve_exit_s > remaining:
|
|
||||||
entry["outcome"] = "skipped_budget"
|
|
||||||
return
|
|
||||||
remaining_cells = list(cells)
|
|
||||||
attempts = 0
|
|
||||||
while remaining_cells:
|
|
||||||
if self._virtual_cap_reached():
|
|
||||||
entry["outcome"] = "skipped_virtual_budget"
|
|
||||||
entry["attempts"] = attempts
|
|
||||||
return
|
|
||||||
cell = min(remaining_cells, key=lambda c: (
|
|
||||||
math.dist(self.client.state.position, c), c))
|
|
||||||
remaining_cells.remove(cell)
|
|
||||||
attempts += 1
|
|
||||||
outcome = self.client.clear(cell, channel)
|
|
||||||
if outcome.result == CLEAR_SUCCESS:
|
|
||||||
self.belief.mark_cleared(channel)
|
|
||||||
self.cleared.append(channel)
|
|
||||||
self._step_clears.append(channel)
|
|
||||||
self.blocked_probes.pop(channel, None)
|
|
||||||
entry["outcome"] = "cleared"
|
|
||||||
entry["attempts"] = attempts
|
|
||||||
return
|
|
||||||
entry["outcome"] = "exhausted"
|
|
||||||
entry["attempts"] = attempts
|
|
||||||
self._anomaly("hrl_fallback_exhausted", channel, cells=len(cells))
|
|
||||||
|
|
||||||
def _hrl_probe_point(self, region, channel: int) -> Vec2:
|
|
||||||
"""定向遮挡梯子探点（与 DirectionalExtension._next_probe 同口径）。"""
|
|
||||||
from ..planner import select_measurement_point
|
|
||||||
blocked = self.blocked_probes.get(channel, [])
|
|
||||||
center, radius = region.mec()
|
|
||||||
if not blocked:
|
|
||||||
return select_measurement_point(region, self.client.state.position,
|
|
||||||
self.config).position
|
|
||||||
safe = safe_disk_radius(region, self.config)
|
|
||||||
first = blocked[0]
|
|
||||||
if len(blocked) == 1 and safe >= 0:
|
|
||||||
dx, dy = center[0] - first[0], center[1] - first[1]
|
|
||||||
norm = math.hypot(dx, dy)
|
|
||||||
if norm <= 1e-9:
|
|
||||||
return (center[0] + safe, center[1])
|
|
||||||
scale = min(1.0, safe / norm)
|
|
||||||
return (center[0] + dx * scale, center[1] + dy * scale)
|
|
||||||
base = math.atan2(first[1] - center[1], first[0] - center[0])
|
|
||||||
azimuth = base + math.radians(45.0 * (len(blocked) - 1))
|
|
||||||
return (center[0] + radius * math.cos(azimuth),
|
|
||||||
center[1] + radius * math.sin(azimuth))
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 事件捕获
|
|
||||||
def _reset_step_recording(self) -> None:
|
|
||||||
self._step_events = []
|
|
||||||
self._step_clears = []
|
|
||||||
|
|
||||||
def _record_event(self, **event) -> None:
|
|
||||||
self._step_events.append(event)
|
|
||||||
@@ -1,172 +0,0 @@
|
|||||||
"""P7 分层强化学习闭环（``--strategy hrl`` / ``--q4-strategy hrl``）。
|
|
||||||
|
|
||||||
边界与保守性声明（与 rl_loop.py 同口径）：
|
|
||||||
- 高层只在固定候选槽位上做调度决策（证书清除 / 信息增益定位 / 覆盖站 /
|
|
||||||
Q4 梯子探点与后备清除），候选几何全部由既有规则生成；
|
|
||||||
- 感知（belief + 网格幸存集）、清除证书、跳测规则、异常留痕完全继承
|
|
||||||
:class:`BayesLoop`，不触碰真源保留定理的前提；
|
|
||||||
- 问题3 的幸存集按全部读数更新（no_signal 删盘）；问题4 定向源的
|
|
||||||
no_signal 不能排除朝向遮挡，幸存集只按 direction/near 更新，遮挡
|
|
||||||
证据走 blocked_probes 梯子与后备清除槽位（DirectionalExtension 口径）；
|
|
||||||
- 高层策略缺失时经 :meth:`heuristic_action` 降级为 BayesLoop 任务排序，
|
|
||||||
闭环仍然完备可运行。
|
|
||||||
|
|
||||||
问题4 发现覆盖使用 49 点网格（七点环的覆盖证明只对全向源成立）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from ..belief import ChannelStatus
|
|
||||||
from ..config import ProblemConfig, Vec2
|
|
||||||
from ..coverage import CoverageLedger, coverage_points
|
|
||||||
from ..hrl.scheduler import HRLControllerMixin, point_key
|
|
||||||
from ..protocol.rules import NO_SIGNAL
|
|
||||||
from .bayes_loop import BayesLoop
|
|
||||||
|
|
||||||
|
|
||||||
class HRLLoop(HRLControllerMixin, BayesLoop):
|
|
||||||
"""分层闭环：BayesLoop 感知/证书 + 高层槽位调度（问题3/4 共用）。"""
|
|
||||||
|
|
||||||
def __init__(self, client, config: ProblemConfig = ProblemConfig(), *,
|
|
||||||
problem: int = 3, high_policy=None, low_policy=None,
|
|
||||||
trace_executor: bool = False):
|
|
||||||
super().__init__(client, config)
|
|
||||||
if problem not in (3, 4):
|
|
||||||
raise ValueError("problem 必须是 3 或 4")
|
|
||||||
self.problem = int(problem)
|
|
||||||
self.high_policy = high_policy
|
|
||||||
self.low_policy = low_policy
|
|
||||||
self.trace_executor = bool(trace_executor)
|
|
||||||
self.executor_trace: list[dict] = []
|
|
||||||
self.hrl_decisions: list[dict] = []
|
|
||||||
self._step_events: list[dict] = []
|
|
||||||
self._step_clears: list[int] = []
|
|
||||||
self._cert_cache: dict = {}
|
|
||||||
self._last_candidates: list = []
|
|
||||||
self._obs_mec_cache: dict = {}
|
|
||||||
self._station_pending_map: dict = {}
|
|
||||||
# 观测里幸存格数的归一化基准（初始全集），使特征对训练/部署的
|
|
||||||
# 网格密度差不变。
|
|
||||||
self._initial_alive = {channel: len(pf)
|
|
||||||
for channel, pf in self.particles.items()}
|
|
||||||
if problem == 4:
|
|
||||||
self.directional_observed: list[int] = []
|
|
||||||
self.blocked_probes: dict[int, list[Vec2]] = {}
|
|
||||||
self.fallback_log: list[dict] = []
|
|
||||||
# 49 点网格：七点环覆盖证明只对全向源成立 [口径见模块注释]
|
|
||||||
self.ledger = CoverageLedger(config, points=coverage_points(config))
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 主循环
|
|
||||||
def _coverage_phase(self) -> None:
|
|
||||||
"""统一任务循环：每步构建候选，由高层策略（或启发式降级）选槽执行。"""
|
|
||||||
while self.stop_reason is None:
|
|
||||||
if self._count_limit_reached() or self._virtual_cap_reached():
|
|
||||||
return
|
|
||||||
# 顺路观测会改变信念、证书和候选；必须发生在本轮快照之前。
|
|
||||||
if self.high_policy is None and self.problem == 3 \
|
|
||||||
and self.config.bayes.opportunistic_measure:
|
|
||||||
self._measure_at_stop()
|
|
||||||
if self.stop_reason is not None or self._count_limit_reached() \
|
|
||||||
or self._virtual_cap_reached():
|
|
||||||
return
|
|
||||||
candidates = self.build_candidates()
|
|
||||||
if not candidates:
|
|
||||||
return
|
|
||||||
if self.high_policy is not None:
|
|
||||||
observation = self.hrl_observation(candidates)
|
|
||||||
mask = self.hrl_action_mask(candidates)
|
|
||||||
action = self.high_policy.select(observation, mask)
|
|
||||||
else:
|
|
||||||
# 降级路径：BayesLoop 任务排序（无 RL 依赖时闭环仍完备）
|
|
||||||
action = self.heuristic_action(candidates)
|
|
||||||
if self.candidate_by_slot(candidates, action) is None:
|
|
||||||
# 非法策略输出不能造成零耗时无限循环；事实记入异常再安全调度。
|
|
||||||
self._anomaly("hrl_invalid_policy_action", 0, action=int(action))
|
|
||||||
action = self.heuristic_action(candidates)
|
|
||||||
cand = self.candidate_by_slot(candidates, action)
|
|
||||||
if cand is not None:
|
|
||||||
self._log_low_level(cand)
|
|
||||||
self.hrl_decisions.append({
|
|
||||||
"action": int(action), "kind": cand.kind,
|
|
||||||
"channel": cand.channel,
|
|
||||||
"virtual_time_s": self.client.state.virtual_time_s})
|
|
||||||
self.hrl_step(action, candidates=candidates)
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 感知记录
|
|
||||||
def _record_measure(self, channel: int, position: Vec2, outcome) -> None:
|
|
||||||
"""BayesLoop._record_measure 的事件捕获版；问题4 no_signal 不删格。
|
|
||||||
|
|
||||||
Q4 语义：no_signal 可能是朝向遮挡 [观测推理]，删除幸存格会破坏
|
|
||||||
真源保留，因此问题4 幸存集只按 direction/near 更新；缺席证明
|
|
||||||
交由 49 点网格覆盖账本（继承的 _mark_absent_after_coverage）。
|
|
||||||
"""
|
|
||||||
key = point_key(position)
|
|
||||||
redundant = key in self._measured_points.get(channel, set())
|
|
||||||
belief = self.belief.channels[channel]
|
|
||||||
was_unknown = belief.status is ChannelStatus.UNKNOWN
|
|
||||||
pf = self.particles[channel]
|
|
||||||
before = len(pf)
|
|
||||||
self.belief.update_measure(channel, position, outcome.result,
|
|
||||||
outcome.svd_deg)
|
|
||||||
self._measured_points.setdefault(channel, set()).add(key)
|
|
||||||
self._clear_failed.discard(channel)
|
|
||||||
belief = self.belief.channels[channel]
|
|
||||||
if belief.status not in (ChannelStatus.CLEARED, ChannelStatus.ABSENT):
|
|
||||||
if not (self.problem == 4 and outcome.result == NO_SIGNAL):
|
|
||||||
pf.apply_measure(position, outcome.result, outcome.svd_deg)
|
|
||||||
if not pf:
|
|
||||||
if belief.status is ChannelStatus.UNKNOWN:
|
|
||||||
self.belief.mark_absent(channel)
|
|
||||||
if channel not in self.pf_absent_channels:
|
|
||||||
self.pf_absent_channels.append(channel)
|
|
||||||
else:
|
|
||||||
self._anomaly("pf_empty_while_detected", channel,
|
|
||||||
position=position)
|
|
||||||
self._record_event(channel=channel, result=outcome.result,
|
|
||||||
pf_before=before, pf_after=len(pf),
|
|
||||||
redundant=redundant, was_unknown=was_unknown)
|
|
||||||
|
|
||||||
def _attempt_clear(self, channel: int, target: Vec2, cert) -> bool:
|
|
||||||
success = super()._attempt_clear(channel, target, cert)
|
|
||||||
if success and hasattr(self, "blocked_probes"):
|
|
||||||
self.blocked_probes.pop(channel, None)
|
|
||||||
if success:
|
|
||||||
self._step_clears.append(channel)
|
|
||||||
return success
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 低层执行留痕
|
|
||||||
def _log_low_level(self, cand) -> None:
|
|
||||||
"""低层到达推演留痕（本地步进，不与模拟器交互，不产生虚拟耗时）。"""
|
|
||||||
if not self.trace_executor:
|
|
||||||
return
|
|
||||||
from ..hrl.executor import LowLevelExecutor
|
|
||||||
executor = getattr(self, "_executor", None)
|
|
||||||
if executor is None:
|
|
||||||
executor = self._executor = LowLevelExecutor(self.low_policy)
|
|
||||||
waypoints = executor.roll(self.client.state.position, cand.position)
|
|
||||||
if self.low_policy is not None:
|
|
||||||
self.executor_trace.append({
|
|
||||||
"kind": cand.kind, "channel": cand.channel,
|
|
||||||
"goal": [round(v, 2) for v in cand.position],
|
|
||||||
"waypoints": [[round(v, 2) for v in p] for p in waypoints],
|
|
||||||
"navigation": dict(executor.last_roll_info),
|
|
||||||
})
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ 收尾
|
|
||||||
def _summary(self, info) -> dict:
|
|
||||||
report = super()._summary(info)
|
|
||||||
report["strategy"] = "hrl_q4" if self.problem == 4 else "hrl"
|
|
||||||
report["hrl_problem"] = self.problem
|
|
||||||
kinds: dict[str, int] = {}
|
|
||||||
for decision in self.hrl_decisions:
|
|
||||||
kinds[decision["kind"]] = kinds.get(decision["kind"], 0) + 1
|
|
||||||
report["hrl_action_kinds"] = kinds
|
|
||||||
report["hrl_decisions"] = self.hrl_decisions
|
|
||||||
report["hrl_executor_trace"] = self.executor_trace
|
|
||||||
if self.problem == 4:
|
|
||||||
report["directional_observed_channels"] = \
|
|
||||||
list(self.directional_observed)
|
|
||||||
report["fallback_clears"] = self.fallback_log
|
|
||||||
report["fallback_clear_attempts"] = sum(
|
|
||||||
entry.get("attempts", 0) for entry in self.fallback_log)
|
|
||||||
return report
|
|
||||||
@@ -1,94 +0,0 @@
|
|||||||
"""P6b 强化学习闭环接入：PPO 学习"下一检测点"选择（问题3/4）。
|
|
||||||
|
|
||||||
边界与保守性声明：
|
|
||||||
- RL 只替换测量点排序（在既有几何候选集内做 argmax），不生成新的
|
|
||||||
空间候选、不修改覆盖证明、不修改清除证书；
|
|
||||||
- 问题3：继承 :class:`BayesLoop`（七点覆盖 + 粒子滤波 + 滚动任务调度），
|
|
||||||
仅 :meth:`_select_point` 改由 PPO 输出候选编号；
|
|
||||||
- 问题4：继承 :class:`DirectionalExtension`（49 点定向覆盖 + 对跖/环探 +
|
|
||||||
20m 有限圆盘覆盖后备清除），仅 :meth:`_next_probe` 改由 PPO 输出；
|
|
||||||
遮挡计数、定向观测与后备清除兜底完全保留；
|
|
||||||
- 训练环境与运行特征同源（mcm26b.rl.features），模型加载时校验
|
|
||||||
观测/动作维度，不匹配即拒绝运行，避免静默错位。
|
|
||||||
|
|
||||||
默认模型位置：output/rl/q3_ppo.zip、output/rl/q4_ppo.zip。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
from ..config import ProblemConfig, Vec2
|
|
||||||
from ..rl.policy import PointSelectionPolicy, default_model_path
|
|
||||||
from . import DirectionalExtension
|
|
||||||
from .bayes_loop import BayesLoop
|
|
||||||
|
|
||||||
|
|
||||||
class RLBayesLoop(BayesLoop):
|
|
||||||
"""问题3：贝叶斯闭环骨架 + PPO 定位选点（--strategy rl）。"""
|
|
||||||
|
|
||||||
def __init__(self, client, config: ProblemConfig = ProblemConfig(), *,
|
|
||||||
model_path: str | Path | None = None, device: str = "auto"):
|
|
||||||
super().__init__(client, config)
|
|
||||||
self.rl_model_path = Path(model_path) if model_path else default_model_path(3)
|
|
||||||
self.rl_policy = PointSelectionPolicy(self.rl_model_path, device=device)
|
|
||||||
self.rl_probe_actions: list[dict] = []
|
|
||||||
|
|
||||||
def _select_point(self, channel: int) -> Vec2 | None:
|
|
||||||
region = self.belief.candidate_set(channel)
|
|
||||||
if region is None:
|
|
||||||
return None
|
|
||||||
point, action = self.rl_policy.select(
|
|
||||||
region, self.client.state.position, channel, self.config,
|
|
||||||
problem=3, blocked=[], belief=self.belief.channels[channel],
|
|
||||||
virtual_time_s=self.client.state.virtual_time_s,
|
|
||||||
receiver_channel=self.client.state.channel)
|
|
||||||
self.rl_probe_actions.append({
|
|
||||||
"channel": channel, "action": int(action),
|
|
||||||
"position": [round(point[0], 3), round(point[1], 3)],
|
|
||||||
"virtual_time_s": self.client.state.virtual_time_s,
|
|
||||||
})
|
|
||||||
return point
|
|
||||||
|
|
||||||
def _summary(self, info) -> dict:
|
|
||||||
report = super()._summary(info)
|
|
||||||
report["strategy"] = "rl"
|
|
||||||
report["rl_model"] = str(self.rl_model_path)
|
|
||||||
report["rl_device"] = self.rl_policy.device
|
|
||||||
report["rl_probe_actions"] = self.rl_probe_actions
|
|
||||||
return report
|
|
||||||
|
|
||||||
|
|
||||||
class RLDirectionalLoop(DirectionalExtension):
|
|
||||||
"""问题4：定向闭环骨架 + PPO 遮挡梯子选点（--q4-strategy rl）。"""
|
|
||||||
|
|
||||||
def __init__(self, client, config: ProblemConfig = ProblemConfig(), *,
|
|
||||||
model_path: str | Path | None = None, device: str = "auto"):
|
|
||||||
super().__init__(client, config)
|
|
||||||
self.rl_model_path = Path(model_path) if model_path else default_model_path(4)
|
|
||||||
self.rl_policy = PointSelectionPolicy(self.rl_model_path, device=device)
|
|
||||||
self.rl_probe_actions: list[dict] = []
|
|
||||||
|
|
||||||
def _next_probe(self, region, channel: int) -> Vec2:
|
|
||||||
blocked = self.blocked_probes.get(channel, [])
|
|
||||||
point, action = self.rl_policy.select(
|
|
||||||
region, self.client.state.position, channel, self.config,
|
|
||||||
problem=4, blocked=blocked,
|
|
||||||
belief=self.belief.channels[channel],
|
|
||||||
virtual_time_s=self.client.state.virtual_time_s,
|
|
||||||
receiver_channel=self.client.state.channel)
|
|
||||||
self.rl_probe_actions.append({
|
|
||||||
"channel": channel, "action": int(action),
|
|
||||||
"position": [round(point[0], 3), round(point[1], 3)],
|
|
||||||
"blocked_count": len(blocked),
|
|
||||||
"virtual_time_s": self.client.state.virtual_time_s,
|
|
||||||
})
|
|
||||||
return point
|
|
||||||
|
|
||||||
def _summary(self, info) -> dict:
|
|
||||||
report = super()._summary(info)
|
|
||||||
report["strategy"] = "rl_q4"
|
|
||||||
report["rl_model"] = str(self.rl_model_path)
|
|
||||||
report["rl_device"] = self.rl_policy.device
|
|
||||||
report["rl_probe_actions"] = self.rl_probe_actions
|
|
||||||
return report
|
|
||||||
@@ -1,175 +0,0 @@
|
|||||||
"""跨平台子进程控制：启动隔离、协作式暂停/停止、必要时强杀整棵进程树。
|
|
||||||
|
|
||||||
训练前端要在 POSIX 与 Windows 上表现一致，而两边的进程模型差别正好卡在
|
|
||||||
「停止训练」这条路径上：
|
|
||||||
|
|
||||||
- ``os.killpg`` / ``os.getpgid`` **只存在于 POSIX**；Windows 上访问它们会抛
|
|
||||||
``AttributeError``。旧实现只捕获 ``ProcessLookupError`` / ``PermissionError``，
|
|
||||||
于是 ``POST /api/train/stop`` 在 Windows 上直接 500，训练进程照跑——表现就是
|
|
||||||
「点停止没反应」。
|
|
||||||
- ``subprocess.Popen(start_new_session=True)`` 在 Windows 上被**静默忽略**：
|
|
||||||
CPython 的 Windows ``_execute_child`` 形参名就是 ``unused_start_new_session``，
|
|
||||||
要隔离进程组只能传 ``CREATE_NEW_PROCESS_GROUP``。
|
|
||||||
- POSIX 的 SIGTERM 在 Windows 上经 ``os.kill`` 会退化成 ``TerminateProcess``：
|
|
||||||
硬杀，训练脚本里"收到信号后保存模型"的处理器根本不会执行。
|
|
||||||
|
|
||||||
所以「优雅暂停 / 优雅停止」不依赖信号量语义，而是走一个**控制文件**：前端原子
|
|
||||||
写 JSON，训练脚本每步轮询。两端都只是普通文件 IO，Windows 与 POSIX 行为一致；
|
|
||||||
信号与强杀退居兜底，只在训练进程不响应控制文件时使用。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import signal
|
|
||||||
import subprocess
|
|
||||||
import time
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"IS_WINDOWS", "ControlState", "spawn_kwargs", "write_control",
|
|
||||||
"read_control", "write_json_atomic", "terminate_process_tree",
|
|
||||||
]
|
|
||||||
|
|
||||||
# 运行期常量而非"导入期分支"：调用方（含测试）可以 monkeypatch 它来模拟 Windows
|
|
||||||
IS_WINDOWS = os.name == "nt"
|
|
||||||
|
|
||||||
# 只有 Windows 定义 CREATE_NEW_PROCESS_GROUP；POSIX 上取 0 表示不使用 creationflags
|
|
||||||
_CREATE_NEW_PROCESS_GROUP = getattr(subprocess, "CREATE_NEW_PROCESS_GROUP", 0)
|
|
||||||
|
|
||||||
# 发出「停止」到强杀之间留给训练脚本收尾（保存模型 + 写元数据）的秒数
|
|
||||||
DEFAULT_GRACE_S = 15.0
|
|
||||||
|
|
||||||
|
|
||||||
def spawn_kwargs() -> dict[str, Any]:
|
|
||||||
"""返回 ``Popen`` 的平台相关参数：让训练进程独立于前端进程组。
|
|
||||||
|
|
||||||
POSIX 用 ``setsid``（``start_new_session``）；Windows 上用
|
|
||||||
``CREATE_NEW_PROCESS_GROUP``，否则前端所在控制台的 Ctrl+C 会连带打到训练进程。
|
|
||||||
"""
|
|
||||||
if IS_WINDOWS:
|
|
||||||
if _CREATE_NEW_PROCESS_GROUP:
|
|
||||||
return {"creationflags": _CREATE_NEW_PROCESS_GROUP}
|
|
||||||
return {}
|
|
||||||
return {"start_new_session": True}
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class ControlState:
|
|
||||||
"""控制文件的一次快照；缺失或损坏时字段全为默认（即"无指令"）。"""
|
|
||||||
|
|
||||||
paused: bool = False
|
|
||||||
stop: bool = False
|
|
||||||
seq: int = 0
|
|
||||||
ts: float = 0.0
|
|
||||||
|
|
||||||
|
|
||||||
def write_control(path: Path, *, paused: bool = False, stop: bool = False,
|
|
||||||
seq: int | None = None) -> ControlState:
|
|
||||||
"""原子写控制文件（同目录临时文件 + ``os.replace``）。
|
|
||||||
|
|
||||||
训练侧随时可能来读，必须避免读到写了一半的 JSON。``os.replace`` 在
|
|
||||||
Windows 与 POSIX 上都是原子的（同目录、同卷）。
|
|
||||||
"""
|
|
||||||
path = Path(path)
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
state = ControlState(
|
|
||||||
paused=bool(paused),
|
|
||||||
stop=bool(stop),
|
|
||||||
seq=int(seq) if seq is not None else time.time_ns(),
|
|
||||||
ts=time.time(),
|
|
||||||
)
|
|
||||||
payload = {"paused": state.paused, "stop": state.stop,
|
|
||||||
"seq": state.seq, "ts": state.ts}
|
|
||||||
tmp = path.with_name(f".{path.name}.{os.getpid()}.tmp")
|
|
||||||
tmp.write_text(json.dumps(payload, ensure_ascii=False), encoding="utf-8")
|
|
||||||
os.replace(tmp, path)
|
|
||||||
return state
|
|
||||||
|
|
||||||
|
|
||||||
def read_control(path: Path) -> ControlState:
|
|
||||||
"""读控制文件；缺失 / 半截 / 损坏一律当作"无指令"，绝不让训练崩在控制面上。"""
|
|
||||||
try:
|
|
||||||
raw = Path(path).read_text(encoding="utf-8")
|
|
||||||
payload = json.loads(raw)
|
|
||||||
except (OSError, ValueError, UnicodeDecodeError):
|
|
||||||
return ControlState()
|
|
||||||
if not isinstance(payload, dict):
|
|
||||||
return ControlState()
|
|
||||||
return ControlState(
|
|
||||||
paused=bool(payload.get("paused", False)),
|
|
||||||
stop=bool(payload.get("stop", False)),
|
|
||||||
seq=int(payload.get("seq") or 0),
|
|
||||||
ts=float(payload.get("ts") or 0.0),
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
def write_json_atomic(path: Path, payload: Any) -> None:
|
|
||||||
"""原子写 JSON 文件（同目录临时文件 + ``os.replace``）。
|
|
||||||
|
|
||||||
用于"急停恢复"相关的标记文件（``resume.json`` / ``forced_stop.json`` 等）：
|
|
||||||
读取方随时可能来读，必须避免看到写了一半的内容。
|
|
||||||
"""
|
|
||||||
path = Path(path)
|
|
||||||
path.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
tmp = path.with_name(f".{path.name}.{os.getpid()}.tmp")
|
|
||||||
tmp.write_text(json.dumps(payload, ensure_ascii=False, indent=2),
|
|
||||||
encoding="utf-8")
|
|
||||||
os.replace(tmp, path)
|
|
||||||
|
|
||||||
|
|
||||||
def terminate_process_tree(process: subprocess.Popen | None, *,
|
|
||||||
grace_s: float = DEFAULT_GRACE_S,
|
|
||||||
poll_s: float = 0.2) -> bool:
|
|
||||||
"""等待 ``grace_s`` 让协作式停止生效，超时则强杀整棵进程树。
|
|
||||||
|
|
||||||
返回**是否动用了强杀**：``False`` 表示进程在宽限期内自行退出（模型有机会
|
|
||||||
落盘），``True`` 表示进程被强杀、最后一段产物可能缺失。调用方应先通过控制
|
|
||||||
文件请求停止，再调用本函数；本函数不负责写控制文件。
|
|
||||||
"""
|
|
||||||
if process is None:
|
|
||||||
return False
|
|
||||||
if process.poll() is not None:
|
|
||||||
return False
|
|
||||||
deadline = time.monotonic() + max(0.0, grace_s)
|
|
||||||
while time.monotonic() < deadline:
|
|
||||||
if process.poll() is not None:
|
|
||||||
return False
|
|
||||||
time.sleep(min(poll_s, max(0.0, deadline - time.monotonic())))
|
|
||||||
if process.poll() is not None:
|
|
||||||
return False
|
|
||||||
_hard_kill_tree(process)
|
|
||||||
try:
|
|
||||||
process.wait(timeout=10.0)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
pass
|
|
||||||
return True
|
|
||||||
|
|
||||||
|
|
||||||
def _hard_kill_tree(process: subprocess.Popen) -> None:
|
|
||||||
"""强杀进程树：POSIX 走进程组 SIGKILL，Windows 走 ``taskkill /T``。
|
|
||||||
|
|
||||||
这里所有 POSIX 专属调用都走 ``getattr`` 探测，Windows 上拿不到分支就退回
|
|
||||||
``Popen.kill()``——不会再出现 ``module 'os' has no attribute 'killpg'``。
|
|
||||||
"""
|
|
||||||
if IS_WINDOWS:
|
|
||||||
try:
|
|
||||||
subprocess.run(["taskkill", "/F", "/T", "/PID", str(process.pid)],
|
|
||||||
capture_output=True, check=False)
|
|
||||||
return
|
|
||||||
except OSError:
|
|
||||||
process.kill()
|
|
||||||
return
|
|
||||||
killpg = getattr(os, "killpg", None)
|
|
||||||
getpgid = getattr(os, "getpgid", None)
|
|
||||||
sigkill = getattr(signal, "SIGKILL", None)
|
|
||||||
if killpg is not None and getpgid is not None and sigkill is not None:
|
|
||||||
try:
|
|
||||||
killpg(getpgid(process.pid), sigkill)
|
|
||||||
return
|
|
||||||
except (ProcessLookupError, PermissionError, OSError):
|
|
||||||
pass
|
|
||||||
process.kill()
|
|
||||||
@@ -1,14 +0,0 @@
|
|||||||
"""问题3/4 强化学习扩展（可选依赖组 ``rl``）。
|
|
||||||
|
|
||||||
- features.py：观测/候选动作的纯几何接口（numpy），训练与运行共享；
|
|
||||||
- env.py：单源定位 Gymnasium 训练环境（依赖 referee，仅训练侧加载）；
|
|
||||||
- policy.py：PPO 模型加载与确定性推理（惰性加载 torch/sb3）。
|
|
||||||
|
|
||||||
运行闭环默认不导入 torch/stable-baselines3；只有 CLI 显式选择
|
|
||||||
``--strategy rl``（问题3）或 ``--q4-strategy rl``（问题4）时，
|
|
||||||
``planner.rl_loop`` 才构造 :class:`policy.PointSelectionPolicy`。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from .features import candidate_actions, build_observation
|
|
||||||
|
|
||||||
__all__ = ["candidate_actions", "build_observation"]
|
|
||||||
@@ -1,11 +0,0 @@
|
|||||||
"""RL 观测/动作空间常量：模型接口的唯一字面量来源。
|
|
||||||
|
|
||||||
布局说明见 features.py；这些数字由训练脚本写入模型接口，
|
|
||||||
改动其中任一值都会使已训练模型失配，须同步重训。
|
|
||||||
"""
|
|
||||||
|
|
||||||
GLOBAL_DIM = 41
|
|
||||||
PER_ACTION_DIM = 6
|
|
||||||
NUM_ACTIONS = 60
|
|
||||||
LADDER_SLOTS = 8
|
|
||||||
OBS_DIM = GLOBAL_DIM + NUM_ACTIONS * PER_ACTION_DIM # 401
|
|
||||||
@@ -1,331 +0,0 @@
|
|||||||
"""问题3/4 的强化学习训练环境：单源"测向—定位—清除"子任务。
|
|
||||||
|
|
||||||
环境与自建评测机 referee 共享同一边界（内存回环，与真实 HTTP 同协议），
|
|
||||||
观测只来自 :class:`mcm26b.belief.BeliefState` 与协议响应；真值仅在 reset
|
|
||||||
时用于布设课程场景，不进入观测。策略端看到的接口与闭环运行时一致。
|
|
||||||
|
|
||||||
问题抽象：机器狗已获得某频道首次 direction 读数，需要在一组几何可行候选
|
|
||||||
检测点中选择下一个测点，目标是尽快把候选集最小包围圆压到 20m 清除证书
|
|
||||||
阈值并清除成功。问题4额外引入定向源遮挡：保证接收盘内出现 no_signal 是
|
|
||||||
"朝向遮挡"的观测证据，环境记录 blocked 序列并开放环探候选。
|
|
||||||
|
|
||||||
该环境是训练侧组件（依赖 referee + gymnasium），运行时代码不导入本模块；
|
|
||||||
导入依赖由可选组 ``[dependency-groups] rl`` 提供。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import math
|
|
||||||
from random import Random
|
|
||||||
|
|
||||||
import gymnasium as gym
|
|
||||||
import numpy as np
|
|
||||||
from gymnasium import spaces
|
|
||||||
|
|
||||||
from ..belief import BeliefState
|
|
||||||
from ..config import ProblemConfig
|
|
||||||
from ..planner import clear_certificate, fallback_clear_cells, safe_disk_radius
|
|
||||||
from ..protocol.client import SimulatorClient
|
|
||||||
from ..protocol.log import SessionLog
|
|
||||||
from ..protocol.rules import CLEAR_SUCCESS, DIRECTION, NO_SIGNAL
|
|
||||||
from .features import NUM_ACTIONS, OBS_DIM, action_mask, build_observation, candidate_actions
|
|
||||||
|
|
||||||
|
|
||||||
class LocalizationEnv(gym.Env):
|
|
||||||
"""单源定位清除 Gymnasium 环境。
|
|
||||||
|
|
||||||
problem=3：全向源；problem=4：定向/全向混合（定向概率 0.5，训练分布
|
|
||||||
比题面更稠密以覆盖遮挡分支；正式策略仍由闭环的后备清除兜底）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
metadata = {"render_modes": []}
|
|
||||||
|
|
||||||
def __init__(self, config: ProblemConfig | None = None, *,
|
|
||||||
problem: int = 3, max_episode_steps: int = 16,
|
|
||||||
directional_probability: float = 0.5,
|
|
||||||
real_budget_s: float = 3600.0,
|
|
||||||
error_seed: int | None = None,
|
|
||||||
handoff_probability: float = 0.35, boundary_probability: float = 0.2):
|
|
||||||
super().__init__()
|
|
||||||
if problem not in (3, 4):
|
|
||||||
raise ValueError("problem 必须是 3 或 4")
|
|
||||||
self.config = config or ProblemConfig()
|
|
||||||
self.problem = int(problem)
|
|
||||||
self.max_episode_steps = int(max_episode_steps)
|
|
||||||
self.directional_probability = float(directional_probability)
|
|
||||||
self.real_budget_s = float(real_budget_s)
|
|
||||||
self.error_seed = error_seed
|
|
||||||
self.handoff_probability = handoff_probability
|
|
||||||
self.boundary_probability = boundary_probability
|
|
||||||
self.observation_space = spaces.Box(low=-10.0, high=10.0,
|
|
||||||
shape=(OBS_DIM,), dtype=np.float32)
|
|
||||||
self.action_space = spaces.Discrete(NUM_ACTIONS)
|
|
||||||
|
|
||||||
# 惰性导入：本模块只属于训练/评估侧，运行闭环不加载 referee。
|
|
||||||
from referee import LocalSimulator as _RefereeLocalSimulator
|
|
||||||
from referee import Source
|
|
||||||
from ..harness import LocalSimTransport, referee_spec
|
|
||||||
self._source_cls = Source
|
|
||||||
self._local_simulator_cls = _RefereeLocalSimulator
|
|
||||||
self._local_sim_transport_cls = LocalSimTransport
|
|
||||||
self._referee_spec = referee_spec
|
|
||||||
|
|
||||||
self._scene_rng = Random()
|
|
||||||
self._simulator = None
|
|
||||||
self._client = None
|
|
||||||
self._source = None
|
|
||||||
self._belief: BeliefState | None = None
|
|
||||||
self._channel: int = 1
|
|
||||||
self._blocked: list = []
|
|
||||||
self._directional_observed = False
|
|
||||||
self._clear_failed = False
|
|
||||||
self._steps = 0
|
|
||||||
self._terminated = False
|
|
||||||
self._truncated = False
|
|
||||||
self._measured_points: set = set()
|
|
||||||
self._fallback_attempts = 0
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------ 场景
|
|
||||||
def _make_source_and_probe(self):
|
|
||||||
"""用真值布设一个"首次测向必为 direction"的课程场景（真值不进观测）。"""
|
|
||||||
rng = self._scene_rng
|
|
||||||
field = self.config.field
|
|
||||||
channel = rng.randint(field.channel_min, field.channel_max)
|
|
||||||
radius = field.omega_radius_m * math.sqrt(rng.random())
|
|
||||||
edge = rng.random() < self.boundary_probability
|
|
||||||
if edge:
|
|
||||||
radius = field.omega_radius_m * rng.uniform(0.98, 1.0)
|
|
||||||
angle = rng.uniform(0.0, 2 * math.pi)
|
|
||||||
x, y = radius * math.cos(angle), radius * math.sin(angle)
|
|
||||||
r_eff = rng.uniform(field.r_eff_min_m, field.r_eff_max_m)
|
|
||||||
if edge:
|
|
||||||
r_eff = field.r_eff_min_m
|
|
||||||
directional = self.problem == 4 and rng.random() < self.directional_probability
|
|
||||||
heading = rng.uniform(0.0, 360.0) if directional else 0.0
|
|
||||||
|
|
||||||
# 首次检测点：距离 [300, min(1450, r_eff-20)] 内随机，定向源取朝向
|
|
||||||
# ±89.9°（接近但不跨越90°边界，保证direction）。机器狗可到Ω外 [题面]。
|
|
||||||
max_distance = max(310.0, min(1450.0, r_eff - 20.0))
|
|
||||||
max_distance = max(max_distance, 310.0)
|
|
||||||
probe_distance = rng.uniform(300.0, max_distance)
|
|
||||||
probe_bearing = (heading + rng.uniform(-89.9, 89.9)
|
|
||||||
if directional else rng.uniform(0.0, 360.0))
|
|
||||||
probe_radians = math.radians(probe_bearing)
|
|
||||||
probe = (x + probe_distance * math.cos(probe_radians),
|
|
||||||
y + probe_distance * math.sin(probe_radians))
|
|
||||||
source = self._source_cls(channel=channel, x=x, y=y, r_eff_m=r_eff,
|
|
||||||
directional=directional, heading_deg=heading)
|
|
||||||
return source, probe
|
|
||||||
|
|
||||||
def _make_episode(self) -> bool:
|
|
||||||
"""重建模拟器并完成首次测向；返回 False 表示连续未得到 direction。"""
|
|
||||||
for _ in range(20):
|
|
||||||
self._source, probe = self._make_source_and_probe()
|
|
||||||
self._simulator = self._local_simulator_cls(
|
|
||||||
[self._source], robot_id="rl-train",
|
|
||||||
spec=self._referee_spec(
|
|
||||||
self.config, max_real_duration_s=self.real_budget_s),
|
|
||||||
error_seed=(self.error_seed if self.error_seed is not None
|
|
||||||
else self._scene_rng.randint(0, 2 ** 31 - 1)),
|
|
||||||
stress_mode=True)
|
|
||||||
self._client = SimulatorClient(
|
|
||||||
"rl-train", config=self.config,
|
|
||||||
transport=self._local_sim_transport_cls(self._simulator),
|
|
||||||
log=SessionLog())
|
|
||||||
self._client.enter()
|
|
||||||
outcome = self._client.measure(probe, self._source.channel)
|
|
||||||
if outcome.result == DIRECTION:
|
|
||||||
self._channel = self._source.channel
|
|
||||||
self._belief = BeliefState(self.config)
|
|
||||||
self._belief.update_measure(self._channel, probe, outcome.result,
|
|
||||||
outcome.svd_deg)
|
|
||||||
self._blocked = []
|
|
||||||
self._directional_observed = False
|
|
||||||
self._clear_failed = False
|
|
||||||
self._measured_points = {self._point_key(probe)}
|
|
||||||
self._fallback_attempts = 0
|
|
||||||
# 运行中可能先服务其他频道后再定位本源；模拟这种位置/频道交接。
|
|
||||||
if self._scene_rng.random() < self.handoff_probability:
|
|
||||||
angle = self._scene_rng.uniform(0, 2*math.pi)
|
|
||||||
reach = self.config.field.omega_radius_m * math.sqrt(self._scene_rng.random())
|
|
||||||
other = (self.config.field.channel_min
|
|
||||||
if self._channel != self.config.field.channel_min
|
|
||||||
else self.config.field.channel_max)
|
|
||||||
self._client.measure((reach*math.cos(angle), reach*math.sin(angle)), other)
|
|
||||||
self._start_virtual_s = self._client.state.virtual_time_s
|
|
||||||
return True
|
|
||||||
# near / no_signal 首测属于概率为零的边界或课程构造失败，换场景。
|
|
||||||
continue
|
|
||||||
return False
|
|
||||||
|
|
||||||
# ------------------------------------------------------------------ Gym API
|
|
||||||
@staticmethod
|
|
||||||
def _point_key(point) -> tuple[int, int]:
|
|
||||||
return (int(round(point[0] * 2)), int(round(point[1] * 2)))
|
|
||||||
|
|
||||||
def reset(self, *, seed: int | None = None, options: dict | None = None):
|
|
||||||
super().reset(seed=seed)
|
|
||||||
self.close()
|
|
||||||
if seed is not None:
|
|
||||||
self._scene_rng = Random(seed)
|
|
||||||
self._steps = 0
|
|
||||||
self._localize_steps = 0
|
|
||||||
self._failure_reason = None
|
|
||||||
self._terminated = False
|
|
||||||
self._truncated = False
|
|
||||||
if not self._make_episode():
|
|
||||||
raise RuntimeError("连续 20 次无法构造 direction 首测场景")
|
|
||||||
observation = self._observe()
|
|
||||||
return observation, self._info()
|
|
||||||
|
|
||||||
def _observe(self) -> np.ndarray:
|
|
||||||
region = self._belief.candidate_set(self._channel)
|
|
||||||
if region is None:
|
|
||||||
raise RuntimeError("首次测向后信念状态缺失候选集")
|
|
||||||
return build_observation(
|
|
||||||
region, self._client.state.position, self._channel, self.config,
|
|
||||||
self.problem, self._blocked, self._belief.channels[self._channel],
|
|
||||||
virtual_time_s=self._client.state.virtual_time_s,
|
|
||||||
receiver_channel=self._client.state.channel)
|
|
||||||
|
|
||||||
def action_masks(self):
|
|
||||||
return action_mask(self._belief.candidate_set(self._channel), self._client.state.position,
|
|
||||||
self.config, self.problem, self._blocked,
|
|
||||||
self._belief.channels[self._channel])
|
|
||||||
|
|
||||||
def _info(self) -> dict:
|
|
||||||
source = self._source
|
|
||||||
return {
|
|
||||||
"channel": self._channel,
|
|
||||||
"source": {"x": source.x, "y": source.y, "r_eff_m": source.r_eff_m,
|
|
||||||
"directional": source.directional,
|
|
||||||
"heading_deg": source.heading_deg},
|
|
||||||
"steps": self._steps,
|
|
||||||
"blocked_count": len(self._blocked),
|
|
||||||
"directional_observed": self._directional_observed,
|
|
||||||
"fallback_attempts": self._fallback_attempts,
|
|
||||||
"virtual_cost_s": self._client.state.virtual_time_s - self._start_virtual_s,
|
|
||||||
"failure_reason": self._failure_reason,
|
|
||||||
}
|
|
||||||
|
|
||||||
def step(self, action):
|
|
||||||
if self._terminated or self._truncated:
|
|
||||||
raise RuntimeError("step() 需要在 terminated/truncated 后先 reset()")
|
|
||||||
if not 0 <= int(action) < NUM_ACTIONS:
|
|
||||||
raise ValueError(f"动作 {action} 超出 Discrete({NUM_ACTIONS})")
|
|
||||||
action = int(action)
|
|
||||||
self._steps += 1
|
|
||||||
region = self._belief.candidate_set(self._channel)
|
|
||||||
if region is None:
|
|
||||||
return self._finish(observation=self._observe(), reward=-20.0,
|
|
||||||
terminated=True, truncated=False)
|
|
||||||
|
|
||||||
# 达到清除证书且上次清除失败已被新测量重置：先自动清除（闭环语义）。
|
|
||||||
cert = clear_certificate(region, self._channel, self.config)
|
|
||||||
if cert.certified and not self._clear_failed:
|
|
||||||
return self._try_clear(cert, observation=self._observe())
|
|
||||||
|
|
||||||
old_virtual = self._client.state.virtual_time_s
|
|
||||||
candidates = candidate_actions(region, self._client.state.position,
|
|
||||||
self.config, self._blocked)
|
|
||||||
point = candidates[action]
|
|
||||||
try:
|
|
||||||
outcome = self._client.measure(point, self._channel)
|
|
||||||
self._belief.update_measure(self._channel, point, outcome.result,
|
|
||||||
outcome.svd_deg)
|
|
||||||
except Exception as exc:
|
|
||||||
self._failure_reason = type(exc).__name__ + ': ' + str(exc)
|
|
||||||
return self._finish(observation=self._observe(), reward=-100.0,
|
|
||||||
terminated=True, truncated=False)
|
|
||||||
|
|
||||||
cost_s = self._client.state.virtual_time_s - old_virtual
|
|
||||||
reward = -cost_s / 100.0
|
|
||||||
self._measured_points.add(self._point_key(point))
|
|
||||||
new_region = self._belief.candidate_set(self._channel)
|
|
||||||
if outcome.result == NO_SIGNAL and self.problem == 4:
|
|
||||||
center = region.mec()[0]
|
|
||||||
safe = safe_disk_radius(region, self.config)
|
|
||||||
if safe >= 0 and math.dist(point, center) <= safe + 1e-6:
|
|
||||||
if not self._directional_observed:
|
|
||||||
self._directional_observed = True
|
|
||||||
self._blocked.append(point)
|
|
||||||
else:
|
|
||||||
self._blocked = []
|
|
||||||
self._localize_steps += 1
|
|
||||||
|
|
||||||
self._clear_failed = False
|
|
||||||
cert = clear_certificate(new_region, self._channel, self.config) \
|
|
||||||
if new_region is not None else None
|
|
||||||
observation = self._observe()
|
|
||||||
if cert is not None and cert.certified:
|
|
||||||
return self._try_clear(cert, observation=observation,
|
|
||||||
reward=reward)
|
|
||||||
if self.problem == 4 and (len(self._blocked) >= self.config.search.max_blocked_probes
|
|
||||||
or self._localize_steps >= self.config.search.max_localize_steps):
|
|
||||||
return self._fallback_clear(observation=observation, reward=reward)
|
|
||||||
if self._steps >= self.max_episode_steps:
|
|
||||||
return self._finish(observation=observation, reward=reward - 100.0,
|
|
||||||
terminated=False, truncated=True)
|
|
||||||
return observation, reward, False, False, self._info()
|
|
||||||
|
|
||||||
def _fallback_clear(self, *, observation, reward: float):
|
|
||||||
"""问题4后备清除语义：遮挡梯子用尽后按 20m 网格覆盖逐格 /clear。
|
|
||||||
|
|
||||||
与 DirectionalExtension._fallback_clear 同口径；训练环境借此让策略
|
|
||||||
学到"梯子耗尽的代价"，运行闭环仍由原后备清除函数执行。
|
|
||||||
"""
|
|
||||||
region = self._belief.candidate_set(self._channel)
|
|
||||||
cells = fallback_clear_cells(region.polygon, self.config)
|
|
||||||
self._fallback_attempts = 0
|
|
||||||
if len(cells) > self.config.search.fallback_max_cells:
|
|
||||||
return self._finish(observation=observation, reward=reward - 100.0,
|
|
||||||
terminated=False, truncated=True)
|
|
||||||
attempts = 0
|
|
||||||
while cells:
|
|
||||||
cell = min(cells, key=lambda c: (
|
|
||||||
math.dist(self._client.state.position, c), c))
|
|
||||||
cells.remove(cell)
|
|
||||||
attempts += 1
|
|
||||||
old_virtual = self._client.state.virtual_time_s
|
|
||||||
outcome = self._client.clear(cell, self._channel)
|
|
||||||
reward += -(self._client.state.virtual_time_s - old_virtual) / 100.0
|
|
||||||
if outcome.result == CLEAR_SUCCESS:
|
|
||||||
self._belief.mark_cleared(self._channel)
|
|
||||||
reward += 20.0
|
|
||||||
self._fallback_attempts = attempts
|
|
||||||
return self._finish(observation=observation, reward=reward,
|
|
||||||
terminated=True, truncated=False)
|
|
||||||
self._fallback_attempts = attempts
|
|
||||||
return self._finish(observation=observation, reward=reward - 100.0,
|
|
||||||
terminated=True, truncated=False)
|
|
||||||
|
|
||||||
def _try_clear(self, cert, *, observation, reward: float = 0.0):
|
|
||||||
old_virtual = self._client.state.virtual_time_s
|
|
||||||
outcome = self._client.clear(cert.center, self._channel)
|
|
||||||
cost_s = self._client.state.virtual_time_s - old_virtual
|
|
||||||
reward += -cost_s / 100.0
|
|
||||||
if outcome.result == CLEAR_SUCCESS:
|
|
||||||
self._belief.mark_cleared(self._channel)
|
|
||||||
reward += 20.0
|
|
||||||
return self._finish(observation=observation, reward=reward,
|
|
||||||
terminated=True, truncated=False)
|
|
||||||
self._clear_failed = True
|
|
||||||
reward -= 1.0
|
|
||||||
if self._steps >= self.max_episode_steps:
|
|
||||||
return self._finish(observation=observation, reward=reward,
|
|
||||||
terminated=False, truncated=True)
|
|
||||||
return observation, reward, False, False, self._info()
|
|
||||||
|
|
||||||
def _finish(self, *, observation, reward, terminated, truncated):
|
|
||||||
self._terminated = bool(terminated)
|
|
||||||
self._truncated = bool(truncated)
|
|
||||||
info = self._info()
|
|
||||||
info["cleared"] = (self._belief.channels[self._channel].status.value
|
|
||||||
== "cleared")
|
|
||||||
self.close()
|
|
||||||
return observation, float(reward), self._terminated, self._truncated, info
|
|
||||||
|
|
||||||
def close(self):
|
|
||||||
if (self._client is not None and self._client.state.entered
|
|
||||||
and not self._client.state.exited):
|
|
||||||
self._client.exit()
|
|
||||||
@@ -1,209 +0,0 @@
|
|||||||
"""RL 状态特征与候选动作生成（问题3/4 共用，运行时与训练环境同源）。
|
|
||||||
|
|
||||||
设计目标：把"下一检测点去哪"这一连续决策转化为固定顺序的离散动作，
|
|
||||||
让 PPO 策略在 GPU 上学习，同时不触碰闭环的保守性证明。候选集合始终由
|
|
||||||
既有几何规则生成（保证接收盘、对称交会、定向遮挡梯子），RL 只负责在
|
|
||||||
这些物理可行候选中排序；证书、覆盖证明与后备清除仍由 planner 层负责。
|
|
||||||
|
|
||||||
观测维度与候选编号一旦训练即成为模型接口，不得随意重排：
|
|
||||||
- 全局特征 41 维（含频道 one-hot 20 维）；
|
|
||||||
- 每个候选 6 维（相对位移、移动耗时、到 MEC 圆心距离、方位、保证接收标记），
|
|
||||||
共 60 × 6 = 360 维；
|
|
||||||
- 总观测 401 维。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import math
|
|
||||||
|
|
||||||
import numpy as np
|
|
||||||
|
|
||||||
from ..config import ProblemConfig, Vec2
|
|
||||||
from ..planner import safe_disk_radius
|
|
||||||
from ._constants import GLOBAL_DIM, LADDER_SLOTS, NUM_ACTIONS, OBS_DIM, PER_ACTION_DIM
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"GLOBAL_DIM",
|
|
||||||
"LADDER_SLOTS",
|
|
||||||
"NUM_ACTIONS",
|
|
||||||
"OBS_DIM",
|
|
||||||
"PER_ACTION_DIM",
|
|
||||||
"candidate_actions",
|
|
||||||
"build_observation",
|
|
||||||
"empty_observation",
|
|
||||||
"action_mask",
|
|
||||||
]
|
|
||||||
|
|
||||||
|
|
||||||
# 候选编号布局（训练/推理共享）：
|
|
||||||
# 0 保证接收盘圆心（或宽候选集 MEC 圆心）
|
|
||||||
# 1..36 保证接收盘 3 环 × 12 方向
|
|
||||||
# 37..51 相对"机器人->MEC 圆心"来向的对称交会候选 3 环 × 5 偏角
|
|
||||||
# 52..59 定向遮挡环探梯子 8 个方向（问题3也保留，作为宽环探索）
|
|
||||||
_RING_RATIOS = (0.4, 0.75, 1.0)
|
|
||||||
_RING_ANGLES = 12
|
|
||||||
_GEO_OFFSETS = (-math.pi / 3, -math.pi / 6, 0.0, math.pi / 6, math.pi / 3)
|
|
||||||
|
|
||||||
|
|
||||||
def _probe_radius(region_radius_m: float, safe_m: float) -> float:
|
|
||||||
"""环候选的基准半径：有保证接收盘用盘半径，否则退化为 MEC 半径。"""
|
|
||||||
if safe_m >= 0.0:
|
|
||||||
return safe_m
|
|
||||||
return max(region_radius_m, 10.0)
|
|
||||||
|
|
||||||
|
|
||||||
def candidate_actions(region, position: Vec2, config: ProblemConfig,
|
|
||||||
blocked: list[Vec2] | None = None) -> list[Vec2]:
|
|
||||||
"""生成固定顺序的 60 个候选检测点。
|
|
||||||
|
|
||||||
参数 region 为 belief.CandidateRegion（任意凸候选集）；blocked 是
|
|
||||||
定向遮挡记录（问题4），问题3 传空列表即可。返回顺序与模型训练时一致。
|
|
||||||
"""
|
|
||||||
center, radius = region.mec()
|
|
||||||
safe = safe_disk_radius(region, config)
|
|
||||||
base_radius = _probe_radius(radius, safe)
|
|
||||||
# 半径恰为0时也保留固定编号，重复位置由action_mask屏蔽。
|
|
||||||
points = [tuple(center)]
|
|
||||||
for fraction in _RING_RATIOS:
|
|
||||||
for k in range(_RING_ANGLES):
|
|
||||||
angle = (k + 0.5) * 2 * math.pi / _RING_ANGLES
|
|
||||||
points.append((center[0] + base_radius*fraction*math.cos(angle),
|
|
||||||
center[1] + base_radius*fraction*math.sin(angle)))
|
|
||||||
assert len(points) == 37
|
|
||||||
|
|
||||||
base_angle = math.atan2(position[1] - center[1],
|
|
||||||
position[0] - center[0])
|
|
||||||
for fraction in _RING_RATIOS:
|
|
||||||
for offset in _GEO_OFFSETS:
|
|
||||||
angle = base_angle + offset
|
|
||||||
points.append((center[0] + base_radius * fraction * math.cos(angle),
|
|
||||||
center[1] + base_radius * fraction * math.sin(angle)))
|
|
||||||
|
|
||||||
blocked = list(blocked or [])
|
|
||||||
first = blocked[0] if blocked else None
|
|
||||||
anchor = 0.0
|
|
||||||
if first is not None:
|
|
||||||
anchor = math.atan2(first[1] - center[1], first[0] - center[0])
|
|
||||||
ladder_radius = max(radius, 20.0)
|
|
||||||
for slot in range(LADDER_SLOTS):
|
|
||||||
angle = anchor + slot * math.pi / 4
|
|
||||||
points.append((center[0] + ladder_radius * math.cos(angle),
|
|
||||||
center[1] + ladder_radius * math.sin(angle)))
|
|
||||||
assert len(points) == NUM_ACTIONS
|
|
||||||
return points
|
|
||||||
|
|
||||||
|
|
||||||
def action_mask(region, position: Vec2, config: ProblemConfig, problem: int,
|
|
||||||
blocked=None, belief=None) -> np.ndarray:
|
|
||||||
"""只依据可见观测屏蔽同地重测、重复候选、越界点。
|
|
||||||
|
|
||||||
Q3有距离保证盘时屏蔽盘外探点；Q4允许盘外探点，但绝不把距离保证
|
|
||||||
当作朝向保证。训练与推理共用此函数，不读取源位置或朝向。
|
|
||||||
"""
|
|
||||||
def key(p):
|
|
||||||
return (round(p[0]*2), round(p[1]*2))
|
|
||||||
visited = {key(r['position']) for r in getattr(belief, 'measurements', [])}
|
|
||||||
seen = set()
|
|
||||||
center, _ = region.mec()
|
|
||||||
safe = safe_disk_radius(region, config)
|
|
||||||
result = []
|
|
||||||
for p in candidate_actions(region, position, config, blocked):
|
|
||||||
tag = key(p)
|
|
||||||
valid = (tag not in visited and tag not in seen
|
|
||||||
and all(math.isfinite(v) and abs(v) <= config.field.coord_abs_limit_m for v in p)
|
|
||||||
and (problem != 3 or safe < 0 or math.dist(p, center) <= safe + 1e-6))
|
|
||||||
result.append(valid)
|
|
||||||
if valid:
|
|
||||||
seen.add(tag)
|
|
||||||
if not any(result):
|
|
||||||
# 退化几何无新候选时保留一个合法动作，由步数/预算边界收尾。
|
|
||||||
result[0] = True
|
|
||||||
return np.asarray(result, dtype=bool)
|
|
||||||
|
|
||||||
|
|
||||||
def _last_measurement_features(belief) -> tuple[tuple[float, float, float],
|
|
||||||
tuple[float, float]]:
|
|
||||||
"""末次读数的 one-hot + 方向角 sin/cos（无读数取零）。"""
|
|
||||||
readings = getattr(belief, "measurements", None) or []
|
|
||||||
if not readings:
|
|
||||||
return (1.0, 0.0, 0.0), (0.0, 0.0)
|
|
||||||
last = readings[-1]
|
|
||||||
result = last.get("result")
|
|
||||||
no_signal = 1.0 if result == "no_signal" else 0.0
|
|
||||||
direction = 1.0 if result == "direction" else 0.0
|
|
||||||
near = 1.0 if result == "near" else 0.0
|
|
||||||
svd = last.get("svd_deg")
|
|
||||||
if direction and svd is not None:
|
|
||||||
radians = math.radians(float(svd))
|
|
||||||
return (no_signal, direction, near), (math.sin(radians), math.cos(radians))
|
|
||||||
return (no_signal, direction, near), (0.0, 0.0)
|
|
||||||
|
|
||||||
|
|
||||||
def build_observation(region, position: Vec2, channel: int, config: ProblemConfig,
|
|
||||||
problem: int, blocked: list[Vec2] | None = None,
|
|
||||||
belief=None, virtual_time_s: float = 0.0,
|
|
||||||
receiver_channel: int | None = None) -> np.ndarray:
|
|
||||||
"""构造 PPO 观测向量（shape=(401,)，与训练环境逐位一致）。"""
|
|
||||||
field, search = config.field, config.search
|
|
||||||
center, radius = region.mec()
|
|
||||||
safe = safe_disk_radius(region, config)
|
|
||||||
readings, svd_pair = _last_measurement_features(belief)
|
|
||||||
xs = [p[0] for p in region.polygon]
|
|
||||||
ys = [p[1] for p in region.polygon]
|
|
||||||
|
|
||||||
global_features = [
|
|
||||||
position[0] / 2000.0,
|
|
||||||
position[1] / 2000.0,
|
|
||||||
((channel if receiver_channel is None else receiver_channel) - field.channel_min)
|
|
||||||
/ max(field.channel_max - field.channel_min, 1),
|
|
||||||
center[0] / 2000.0,
|
|
||||||
center[1] / 2000.0,
|
|
||||||
radius / 2000.0,
|
|
||||||
max(-1.0, min(1.0, safe / 1000.0)),
|
|
||||||
min(len(getattr(belief, "measurements", []) if belief is not None else []),
|
|
||||||
12) / 12.0,
|
|
||||||
min(float(virtual_time_s or 0.0), field.max_virtual_duration_s)
|
|
||||||
/ field.max_virtual_duration_s,
|
|
||||||
# 证书信息由运行时的 clear_certificate 判定传入；训练环境用 region.mec
|
|
||||||
# 的保守近似：radius + slack <= 20 - margin 即可清除。
|
|
||||||
1.0 if radius + region.slack_m <= field.clear_radius_m - search.clear_margin_m else 0.0,
|
|
||||||
float(problem == 4),
|
|
||||||
min(len(blocked or []) / max(search.max_blocked_probes, 1), 1.0),
|
|
||||||
(blocked[0][0] / 2000.0) if blocked else 0.0,
|
|
||||||
(blocked[0][1] / 2000.0) if blocked else 0.0,
|
|
||||||
readings[0],
|
|
||||||
readings[1],
|
|
||||||
readings[2],
|
|
||||||
svd_pair[0],
|
|
||||||
svd_pair[1],
|
|
||||||
(max(xs) - min(xs)) / 4000.0,
|
|
||||||
(max(ys) - min(ys)) / 4000.0,
|
|
||||||
]
|
|
||||||
one_hot = np.zeros(field.channel_max - field.channel_min + 1, dtype=np.float32)
|
|
||||||
one_hot[int(channel) - field.channel_min] = 1.0
|
|
||||||
global_features.extend(one_hot.tolist())
|
|
||||||
assert len(global_features) == GLOBAL_DIM
|
|
||||||
|
|
||||||
action_features: list[float] = []
|
|
||||||
for point in candidate_actions(region, position, config, blocked):
|
|
||||||
dx, dy = point[0] - position[0], point[1] - position[1]
|
|
||||||
distance = math.hypot(dx, dy)
|
|
||||||
action_features.extend([
|
|
||||||
dx / 2000.0,
|
|
||||||
dy / 2000.0,
|
|
||||||
min(distance / (config.cost.speed_mps * 1500.0), 2.0), # 移动耗时 / 1500s
|
|
||||||
math.hypot(point[0] - center[0], point[1] - center[1]) / 2000.0,
|
|
||||||
(math.degrees(math.atan2(dy, dx)) % 360.0) / 360.0,
|
|
||||||
1.0 if safe >= 0 and math.hypot(point[0] - center[0],
|
|
||||||
point[1] - center[1]) <= safe + 1e-6 else 0.0,
|
|
||||||
])
|
|
||||||
assert len(action_features) == NUM_ACTIONS * PER_ACTION_DIM
|
|
||||||
|
|
||||||
observation = np.asarray(global_features + action_features, dtype=np.float32)
|
|
||||||
assert observation.shape == (OBS_DIM,)
|
|
||||||
return observation
|
|
||||||
|
|
||||||
|
|
||||||
def empty_observation() -> np.ndarray:
|
|
||||||
"""依赖缺失时的零观测（仅用于错误提示路径，不进入训练数据）。"""
|
|
||||||
return np.zeros(OBS_DIM, dtype=np.float32)
|
|
||||||
@@ -1,95 +0,0 @@
|
|||||||
"""RL 选点策略加载与推理（运行时接入 planner 的唯一入口）。
|
|
||||||
|
|
||||||
运行闭环对依赖保持惰性：只有显式选择 ``--strategy rl`` / ``--q4-strategy rl``
|
|
||||||
时才导入 stable_baselines3/torch。模型文件由 scripts/train_rl.py 生成，
|
|
||||||
观测/动作布局与训练环境共享 features 模块。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from pathlib import Path
|
|
||||||
import json
|
|
||||||
|
|
||||||
from ..config import ProblemConfig, Vec2
|
|
||||||
from ._constants import NUM_ACTIONS, OBS_DIM
|
|
||||||
from .features import action_mask, build_observation, candidate_actions
|
|
||||||
|
|
||||||
|
|
||||||
class PointSelectionPolicy:
|
|
||||||
"""加载 PPO 模型，把当前候选集映射到 60 个候选检测点之一。"""
|
|
||||||
|
|
||||||
def __init__(self, model_path: str | Path, *, device: str = "auto"):
|
|
||||||
path = Path(model_path)
|
|
||||||
if not path.is_file():
|
|
||||||
raise RuntimeError(
|
|
||||||
f"未找到 RL 模型 {path}。请先安装 rl 依赖组并训练：\n"
|
|
||||||
f" uv sync --group rl --group dev\n"
|
|
||||||
f" uv run python scripts/train_rl.py --problem 3 --timesteps 120000\n"
|
|
||||||
f" uv run python scripts/train_rl.py --problem 4 --timesteps 40000")
|
|
||||||
try:
|
|
||||||
from stable_baselines3 import PPO
|
|
||||||
except ImportError as exc:
|
|
||||||
raise RuntimeError(
|
|
||||||
"缺少 RL 运行依赖（stable-baselines3）。请执行 "
|
|
||||||
"`uv sync --group rl` 后重试。") from exc
|
|
||||||
if device == "auto":
|
|
||||||
try:
|
|
||||||
import torch
|
|
||||||
device = "cuda" if torch.cuda.is_available() else "cpu"
|
|
||||||
except ImportError:
|
|
||||||
device = "cpu"
|
|
||||||
self.device = device
|
|
||||||
meta_path = path.with_suffix('.json')
|
|
||||||
self.metadata = (json.loads(meta_path.read_text(encoding='utf-8'))
|
|
||||||
if meta_path.exists() else {})
|
|
||||||
self.feature_version = self.metadata.get('feature_version', 1)
|
|
||||||
if self.feature_version not in (1, 2):
|
|
||||||
raise RuntimeError('不支持的RL特征版本')
|
|
||||||
self.masked = self.metadata.get('algorithm') == 'maskable_ppo'
|
|
||||||
if self.masked:
|
|
||||||
from sb3_contrib import MaskablePPO
|
|
||||||
if self.metadata.get('feature_version') != 2:
|
|
||||||
raise RuntimeError('动作屏蔽模型的特征版本不匹配')
|
|
||||||
loader = MaskablePPO
|
|
||||||
else:
|
|
||||||
loader = PPO
|
|
||||||
self.model = loader.load(str(path), device=device)
|
|
||||||
if not self.masked and type(self.model.policy).__module__.startswith('sb3_contrib'):
|
|
||||||
raise RuntimeError('动作屏蔽模型缺少匹配的JSON元数据，不能按普通PPO推理')
|
|
||||||
try:
|
|
||||||
obs_dim = self.model.observation_space.shape[0]
|
|
||||||
action_dim = self.model.action_space.n
|
|
||||||
except Exception as exc:
|
|
||||||
raise RuntimeError(f"模型空间与 RL 接口不兼容: {path}") from exc
|
|
||||||
if obs_dim != OBS_DIM or action_dim != NUM_ACTIONS:
|
|
||||||
raise RuntimeError(
|
|
||||||
f"模型接口失配: 期望 obs={OBS_DIM}/act={NUM_ACTIONS}，"
|
|
||||||
f"实际 obs={obs_dim}/act={action_dim}。请用当前版本重新训练。")
|
|
||||||
self.path = path
|
|
||||||
|
|
||||||
def select(self, region, position: Vec2, channel: int,
|
|
||||||
config: ProblemConfig, problem: int,
|
|
||||||
blocked: list[Vec2] | None = None, belief=None,
|
|
||||||
virtual_time_s: float = 0.0,
|
|
||||||
receiver_channel: int | None = None) -> tuple[Vec2, int]:
|
|
||||||
"""返回 (检测点, 候选编号)；确定性 argmax（评估/比赛口径）。"""
|
|
||||||
if self.metadata.get('problem', problem) != problem:
|
|
||||||
raise RuntimeError('模型训练问题编号与当前第三/四问不匹配')
|
|
||||||
observation = build_observation(
|
|
||||||
region, position, channel, config, problem, blocked, belief,
|
|
||||||
virtual_time_s=virtual_time_s,
|
|
||||||
receiver_channel=receiver_channel if self.feature_version >= 2 else None)
|
|
||||||
kwargs = ({'action_masks': action_mask(region, position, config, problem,
|
|
||||||
blocked, belief)}
|
|
||||||
if self.masked else {})
|
|
||||||
action, _state = self.model.predict(observation, deterministic=True, **kwargs)
|
|
||||||
action = int(action)
|
|
||||||
candidates = candidate_actions(region, position, config, blocked)
|
|
||||||
return candidates[action], action
|
|
||||||
|
|
||||||
|
|
||||||
def default_model_path(problem: int) -> Path:
|
|
||||||
"""各问题的默认模型位置（训练脚本输出口径）。"""
|
|
||||||
if problem not in (3, 4):
|
|
||||||
raise ValueError("problem 必须是 3 或 4")
|
|
||||||
return Path("output/rl") / f"q{problem}_ppo.zip"
|
|
||||||
@@ -1,15 +0,0 @@
|
|||||||
"""模型训练前端：本地回环 Web UI + 训练子进程监控。
|
|
||||||
|
|
||||||
启动：
|
|
||||||
python -m mcm26b.training_ui --ui-port 8036
|
|
||||||
或：
|
|
||||||
python scripts/train_hrl_ui.py --ui-port 8036
|
|
||||||
|
|
||||||
子命令：
|
|
||||||
python -m mcm26b.training_ui --print-schema # 配置契约
|
|
||||||
python -m mcm26b.training_ui --print-preview # 离线示例快照
|
|
||||||
"""
|
|
||||||
from .server import TrainingUI, TrainingJob, run_training_ui
|
|
||||||
from .schema import config_schema
|
|
||||||
|
|
||||||
__all__ = ["TrainingUI", "TrainingJob", "run_training_ui", "config_schema"]
|
|
||||||
@@ -1,4 +0,0 @@
|
|||||||
from .server import run_training_ui
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
raise SystemExit(run_training_ui())
|
|
||||||
@@ -1,397 +0,0 @@
|
|||||||
"""训练快照的纯函数派生：曲线统计、收敛判读、进程视图。
|
|
||||||
|
|
||||||
从 ``server.py`` 拆出来的两个理由：
|
|
||||||
|
|
||||||
1. 断开 ``demo`` ↔ ``server`` 的循环导入——离线预览（``demo.py``）只想复用
|
|
||||||
这两个纯函数，没必要反向 import 整个 HTTP 服务器模块；
|
|
||||||
2. 让"数值判读"可以脱离 HTTP 与子进程单独测试。
|
|
||||||
|
|
||||||
本模块只依赖标准库，不做任何 IO、不持有任何状态。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import math
|
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import time
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
__all__ = ["MAX_LOG_LINES", "MAX_METRICS", "finite_float",
|
|
||||||
"summarize_convergence", "describe_process", "process_pid"]
|
|
||||||
|
|
||||||
MAX_LOG_LINES = 4000
|
|
||||||
MAX_METRICS = 5000
|
|
||||||
|
|
||||||
_NUM_RE = re.compile(r"^[+-]?(?:\d+\.?\d*|\.\d+)(?:[eE][+-]?\d+)?$")
|
|
||||||
|
|
||||||
|
|
||||||
def finite_float(value: Any) -> float | None:
|
|
||||||
"""把任意输入转成有限浮点数；NaN / ±inf / 非数字一律返回 None。
|
|
||||||
|
|
||||||
旧实现用 ``value != value`` 判 NaN，pylint 报 ``comparison-with-itself``；
|
|
||||||
这里用 ``math.isfinite`` 表达同一语义，意图更直白。
|
|
||||||
"""
|
|
||||||
if value is None or isinstance(value, bool):
|
|
||||||
return None
|
|
||||||
if isinstance(value, (int, float)):
|
|
||||||
value = float(value)
|
|
||||||
if not math.isfinite(value):
|
|
||||||
return None
|
|
||||||
return value
|
|
||||||
text = str(value).strip()
|
|
||||||
if not _NUM_RE.match(text):
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return float(text)
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _series(metrics: list[dict[str, Any]], key: str) -> list[tuple[float, float]]:
|
|
||||||
points: list[tuple[float, float]] = []
|
|
||||||
for point in metrics:
|
|
||||||
value = finite_float(point.get(key))
|
|
||||||
steps = finite_float(point.get("timesteps")) or finite_float(
|
|
||||||
point.get("total_timesteps"))
|
|
||||||
if value is None:
|
|
||||||
continue
|
|
||||||
points.append((steps if steps is not None else float(len(points) + 1), value))
|
|
||||||
return points
|
|
||||||
|
|
||||||
|
|
||||||
def _last_value(points: list[tuple[float, float]]) -> float | None:
|
|
||||||
return points[-1][1] if points else None
|
|
||||||
|
|
||||||
|
|
||||||
def _spread(points: list[tuple[float, float]], window: int = 10
|
|
||||||
) -> tuple[float, float] | None:
|
|
||||||
"""最近 window 个点的 (极差, 起点绝对值)，用于判断"是否还在动"。"""
|
|
||||||
sample = points[-window:]
|
|
||||||
if len(sample) < 3:
|
|
||||||
return None
|
|
||||||
ys = [item[1] for item in sample]
|
|
||||||
return max(ys) - min(ys), abs(ys[0])
|
|
||||||
|
|
||||||
|
|
||||||
def _cv(points: list[tuple[float, float]], window: int = 10) -> float | None:
|
|
||||||
"""最近 window 个点的变异系数（标准差/|均值|），越小越平稳。"""
|
|
||||||
sample = [item[1] for item in points[-window:]]
|
|
||||||
if len(sample) < 3:
|
|
||||||
return None
|
|
||||||
mean = sum(sample) / len(sample)
|
|
||||||
if abs(mean) < 1e-12:
|
|
||||||
return None
|
|
||||||
var = sum((value - mean) ** 2 for value in sample) / len(sample)
|
|
||||||
return (var ** 0.5) / abs(mean)
|
|
||||||
|
|
||||||
|
|
||||||
def _slope(points: list[tuple[float, float]], window: int = 8) -> float | None:
|
|
||||||
sample = points[-window:]
|
|
||||||
if len(sample) < 3:
|
|
||||||
return None
|
|
||||||
xs = [item[0] for item in sample]
|
|
||||||
ys = [item[1] for item in sample]
|
|
||||||
n = len(xs)
|
|
||||||
mean_x = sum(xs) / n
|
|
||||||
mean_y = sum(ys) / n
|
|
||||||
denom = sum((x - mean_x) ** 2 for x in xs)
|
|
||||||
if abs(denom) < 1e-12:
|
|
||||||
return None
|
|
||||||
return sum((x - mean_x) * (y - mean_y) for x, y in zip(xs, ys)) / denom
|
|
||||||
|
|
||||||
|
|
||||||
def _rate_per_1k(reward_points: list[tuple[float, float]],
|
|
||||||
window: int = 8) -> float | None:
|
|
||||||
"""最近 window 个点的奖励增益换算成"每 1000 环境步"，便于读趋势。"""
|
|
||||||
slope = _slope(reward_points, window)
|
|
||||||
if slope is None:
|
|
||||||
return None
|
|
||||||
return slope * 1000.0
|
|
||||||
|
|
||||||
|
|
||||||
def _fmt_num(value: float | None, digits: int = 4) -> str:
|
|
||||||
if value is None:
|
|
||||||
return "—"
|
|
||||||
if value != 0 and abs(value) < 1e-3:
|
|
||||||
return f"{value:.3e}"
|
|
||||||
return f"{value:.{digits}g}"
|
|
||||||
|
|
||||||
|
|
||||||
def _flatness(points: list[tuple[float, float]], window: int = 10
|
|
||||||
) -> tuple[float, float] | None:
|
|
||||||
"""窗口内"趋势位移 / 逐点噪声"比值：<1 表示看不出系统性变化。
|
|
||||||
|
|
||||||
判定收敛时只看斜率绝对值会把"很慢但持续的提升"误判为平台；这里把窗口
|
|
||||||
对半切，用两半均值之差当趋势位移，用相邻点平均绝对差当噪声尺度。返回
|
|
||||||
(比值, 噪声尺度)，两者都小才是真的"又平又稳"。
|
|
||||||
"""
|
|
||||||
sample = points[-window:]
|
|
||||||
if len(sample) < 4:
|
|
||||||
return None
|
|
||||||
ys = [item[1] for item in sample]
|
|
||||||
half = len(ys) // 2
|
|
||||||
shift = abs(sum(ys[half:]) / len(ys[half:]) - sum(ys[:half]) / len(ys[:half]))
|
|
||||||
noise = sum(abs(b - a) for a, b in zip(ys, ys[1:])) / (len(ys) - 1)
|
|
||||||
if noise <= 0:
|
|
||||||
return (0.0, 0.0) if shift <= 0 else (float("inf"), 0.0)
|
|
||||||
return shift / noise, noise
|
|
||||||
|
|
||||||
|
|
||||||
def summarize_convergence(metrics: list[dict[str, Any]],
|
|
||||||
monitor: list[dict[str, Any]]) -> dict[str, Any]:
|
|
||||||
"""给前端返回可解释的收敛状态与各参数最近值。
|
|
||||||
|
|
||||||
这是**启发式判读**，不替代正式学习曲线分析：只依据最近窗口的斜率、
|
|
||||||
极差与变异系数给出"平坦/上升/未上升"的分类，并把同一组数字交给前端
|
|
||||||
展示，便于人工复核。
|
|
||||||
"""
|
|
||||||
reward_points = _series(metrics, "ep_rew_mean")
|
|
||||||
if not reward_points and monitor:
|
|
||||||
reward_points = [(float(index), float(row.get("r", 0.0)))
|
|
||||||
for index, row in enumerate(monitor)
|
|
||||||
if finite_float(row.get("r")) is not None]
|
|
||||||
kl_points = _series(metrics, "approx_kl")
|
|
||||||
ev_points = _series(metrics, "explained_variance")
|
|
||||||
vf_points = _series(metrics, "value_loss")
|
|
||||||
pg_points = _series(metrics, "policy_gradient_loss")
|
|
||||||
entropy_points = _series(metrics, "entropy_loss")
|
|
||||||
fps_points = _series(metrics, "fps")
|
|
||||||
|
|
||||||
reward_slope = _slope(reward_points)
|
|
||||||
reward_rate = _rate_per_1k(reward_points)
|
|
||||||
spread = _spread(reward_points)
|
|
||||||
cv = _cv(reward_points)
|
|
||||||
latest_kl = _last_value(kl_points)
|
|
||||||
latest_ev = _last_value(ev_points)
|
|
||||||
latest_vf = _last_value(vf_points)
|
|
||||||
latest_pg = _last_value(pg_points)
|
|
||||||
latest_entropy = _last_value(entropy_points)
|
|
||||||
latest_fps = _last_value(fps_points)
|
|
||||||
latest_timesteps = metrics[-1].get("timesteps") if metrics else None
|
|
||||||
if latest_timesteps is None and metrics:
|
|
||||||
latest_timesteps = metrics[-1].get("total_timesteps")
|
|
||||||
latest_reward = _last_value(reward_points)
|
|
||||||
latest_len = _last_value(_series(metrics, "ep_len_mean"))
|
|
||||||
latest_updates = _last_value(_series(metrics, "n_updates"))
|
|
||||||
window = min(10, max(3, len(reward_points)))
|
|
||||||
reward_window = [round(value, 6) for _, value in reward_points[-window:]]
|
|
||||||
best_reward = max((value for _, value in reward_points), default=None)
|
|
||||||
|
|
||||||
notes: list[str] = []
|
|
||||||
if not metrics and not monitor:
|
|
||||||
status, tone = "等待首个 rollout", "gray"
|
|
||||||
notes.append("训练进程启动后先输出环境初始化与首个 rollout 的进度行。")
|
|
||||||
elif not metrics:
|
|
||||||
status, tone = "已收到回合奖励，等待进度行", "warn"
|
|
||||||
notes.append("Monitor CSV 已有回合记录，但尚未收到 --progress-json 进度行；"
|
|
||||||
"首个 rollout 结束后会出现完整指标。")
|
|
||||||
elif len(metrics) < 3:
|
|
||||||
status, tone = "预热阶段", "warn"
|
|
||||||
notes.append("已收到初始指标，需要更多 rollout 才能判断趋势。")
|
|
||||||
else:
|
|
||||||
tone, status = "ok", "收敛趋势良好"
|
|
||||||
# 平台判据（三条同时成立）：对半窗口看不出位移、窗口内波动小、
|
|
||||||
# 逐点噪声本身也小。慢速但持续的提升会因第一条被排除。
|
|
||||||
flat = _flatness(reward_points)
|
|
||||||
plateau = False
|
|
||||||
if flat is not None and cv is not None and latest_reward is not None:
|
|
||||||
flat_ratio, flat_noise = flat
|
|
||||||
scale = max(abs(latest_reward), 1e-9)
|
|
||||||
plateau = (flat_ratio <= 1.0 and cv <= 0.15
|
|
||||||
and flat_noise <= 0.02 * scale
|
|
||||||
and (spread is None or spread[1] < 1e-12
|
|
||||||
or spread[0] <= 0.15 * spread[1]))
|
|
||||||
if latest_kl is not None and latest_kl > 0.05:
|
|
||||||
status, tone = "KL 偏大，策略更新可能过猛", "bad"
|
|
||||||
notes.append(f"approx_kl={latest_kl:.4f} > 0.05")
|
|
||||||
elif latest_ev is not None and latest_ev < 0.0:
|
|
||||||
status, tone = "价值函数尚未拟合", "warn"
|
|
||||||
notes.append(f"explained_variance={latest_ev:.3f} < 0")
|
|
||||||
elif reward_slope is not None and reward_slope < 0:
|
|
||||||
status, tone = "奖励未形成上升趋势", "warn"
|
|
||||||
notes.append(f"最近 {window} 点奖励斜率为负（{_fmt_num(reward_slope)}/step）")
|
|
||||||
elif plateau:
|
|
||||||
status, tone = "已趋于平稳（疑似收敛）", "ok"
|
|
||||||
notes.append("窗口内对半位移小于逐点噪声，且波动很小"
|
|
||||||
f"（变异系数 {_fmt_num(cv, 3)}）")
|
|
||||||
else:
|
|
||||||
notes.append(f"最近 {window} 点奖励斜率为正（{_fmt_num(reward_slope)}/step）")
|
|
||||||
if latest_kl is not None and latest_kl <= 0.02:
|
|
||||||
notes.append(f"approx_kl={latest_kl:.4f} 较稳定")
|
|
||||||
if latest_ev is not None:
|
|
||||||
notes.append(f"explained_variance={latest_ev:.3f}")
|
|
||||||
if cv is not None:
|
|
||||||
notes.append(f"奖励变异系数={_fmt_num(cv, 3)}")
|
|
||||||
if latest_entropy is not None and abs(latest_entropy) < 1e-3:
|
|
||||||
notes.append("策略熵接近 0，探索可能过早停止")
|
|
||||||
return {
|
|
||||||
"status": status,
|
|
||||||
"tone": tone,
|
|
||||||
"notes": notes,
|
|
||||||
"window": window,
|
|
||||||
"latest_timesteps": latest_timesteps,
|
|
||||||
"reward_slope": reward_slope,
|
|
||||||
"reward_rate_per_1k": reward_rate,
|
|
||||||
"reward_cv": cv,
|
|
||||||
"reward_spread": spread[0] if spread else None,
|
|
||||||
"reward_window": reward_window,
|
|
||||||
"best_reward": best_reward,
|
|
||||||
"latest_reward": latest_reward,
|
|
||||||
"latest_ep_len": latest_len,
|
|
||||||
"latest_updates": latest_updates,
|
|
||||||
"latest_kl": latest_kl,
|
|
||||||
"latest_explained_variance": latest_ev,
|
|
||||||
"latest_value_loss": latest_vf,
|
|
||||||
"latest_policy_loss": latest_pg,
|
|
||||||
"latest_entropy_loss": latest_entropy,
|
|
||||||
"latest_fps": latest_fps,
|
|
||||||
"reward_points": len(reward_points),
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
# 快照状态 -> 界面展示用的进程语义（阶段名称、色调、是否终态）
|
|
||||||
_STATUS_VIEW = {
|
|
||||||
"created": ("已创建", "gray", False, "命令已构造，等待 spawn"),
|
|
||||||
"running": ("运行中", "ok", False, "子进程持有控制台输出与独立进程组"),
|
|
||||||
"paused": ("已暂停", "warn", False, "进程存活但停在 step 边界；点「继续」从暂停处恢复"),
|
|
||||||
"stopping": ("停止中", "warn", False, "已请求优雅停止，等待子进程保存并退出"),
|
|
||||||
"stopped": ("已手动停止", "warn", True, "收到停止请求后退出；脚本会在 step 边界保存模型与元数据"),
|
|
||||||
"exited": ("正常结束", "ok", True, "退出码 0：训练与评估流程走完"),
|
|
||||||
"failed": ("异常结束", "bad", True, "非零退出码或启动失败，见日志尾部"),
|
|
||||||
}
|
|
||||||
|
|
||||||
_PHASES = [
|
|
||||||
("created", "构造命令"),
|
|
||||||
("spawn", "启动子进程"),
|
|
||||||
("rollout", "采集 rollout"),
|
|
||||||
("update", "PPO 更新"),
|
|
||||||
("evaluate", "评估 / 保存"),
|
|
||||||
("done", "收敛归档"),
|
|
||||||
]
|
|
||||||
|
|
||||||
# 状态 -> 当前所处的阶段节点；节点之前为已完成，之后为未开始。
|
|
||||||
_STATUS_PHASE = {
|
|
||||||
"created": "created",
|
|
||||||
"running": "update",
|
|
||||||
"paused": "update",
|
|
||||||
"stopping": "update",
|
|
||||||
"stopped": "done",
|
|
||||||
"exited": "done",
|
|
||||||
"failed": "done",
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def process_pid(process: subprocess.Popen | None) -> int | None:
|
|
||||||
"""已退出进程的 pid 访问会抛 OSError；这里统一吞掉，只用于展示。"""
|
|
||||||
if process is None:
|
|
||||||
return None
|
|
||||||
try:
|
|
||||||
return int(process.pid)
|
|
||||||
except (OSError, ValueError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def describe_process(snapshot: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
"""把 snapshot 派生为界面直接可渲染的进程对象。
|
|
||||||
|
|
||||||
包含：状态语义词、进度（步/百分比）、墙钟与剩余时间估计、最近吞吐、
|
|
||||||
进程时间线节点。数字都取自 snapshot，前端不再重复累加，避免两个刷新
|
|
||||||
频率不同的计数器互相打架。
|
|
||||||
"""
|
|
||||||
status = str(snapshot.get("status") or "created")
|
|
||||||
label, tone, terminal, hint = _STATUS_VIEW.get(
|
|
||||||
status, (status, "gray", False, ""))
|
|
||||||
config = snapshot.get("config") or {}
|
|
||||||
order = [key for key, _ in _PHASES]
|
|
||||||
current = _STATUS_PHASE.get(status, "created")
|
|
||||||
timeline = []
|
|
||||||
for key, title in _PHASES:
|
|
||||||
index = order.index(key)
|
|
||||||
if index < order.index(current):
|
|
||||||
state = "done"
|
|
||||||
elif index == order.index(current):
|
|
||||||
state = "active"
|
|
||||||
else:
|
|
||||||
state = "pending"
|
|
||||||
timeline.append({"key": key, "label": title, "state": state})
|
|
||||||
|
|
||||||
metrics = snapshot.get("metrics") or []
|
|
||||||
latest: dict[str, Any] = metrics[-1] if metrics else {}
|
|
||||||
convergence = snapshot.get("convergence") or {}
|
|
||||||
total_steps = finite_float(config.get("timesteps")) or 0.0
|
|
||||||
# SB3 表格行可能没有 steps 字段，因此进度取"所有进度点的最大步数"，
|
|
||||||
# 而不是只看最后一个点——否则末行恰好缺字段时进度会回退到 0。
|
|
||||||
done_steps = 0.0
|
|
||||||
for point in metrics:
|
|
||||||
value = (finite_float(point.get("timesteps"))
|
|
||||||
or finite_float(point.get("total_timesteps")))
|
|
||||||
if value is not None and value > done_steps:
|
|
||||||
done_steps = value
|
|
||||||
if done_steps == 0.0:
|
|
||||||
done_steps = finite_float(convergence.get("latest_timesteps")) or 0.0
|
|
||||||
# init/resume（课程迁移）时 num_timesteps 从上一阶段继续，进度必须按本次
|
|
||||||
# 增量算；start_timesteps 由训练脚本的进度 JSON 提供，缺省按 0 处理。
|
|
||||||
start_steps = 0.0
|
|
||||||
for point in metrics:
|
|
||||||
value = finite_float(point.get("start_timesteps"))
|
|
||||||
if value is not None:
|
|
||||||
start_steps = value
|
|
||||||
break
|
|
||||||
trained_steps = max(0.0, done_steps - start_steps)
|
|
||||||
percent = (100.0 * trained_steps / total_steps) if total_steps > 0 else None
|
|
||||||
if percent is not None:
|
|
||||||
percent = max(0.0, min(100.0, percent))
|
|
||||||
elapsed = float(snapshot.get("elapsed_s") or 0.0)
|
|
||||||
rate = _latest_rate(metrics)
|
|
||||||
remaining: float | None = None
|
|
||||||
eta_ts: float | None = None
|
|
||||||
if rate and rate > 0 and total_steps > 0 and trained_steps < total_steps:
|
|
||||||
remaining = (total_steps - trained_steps) / rate
|
|
||||||
eta_ts = float(snapshot.get("finished_at") or time.time()) + remaining
|
|
||||||
if not terminal and remaining is None:
|
|
||||||
cap = finite_float(config.get("max_wall_s")) or 0.0
|
|
||||||
if cap > 0:
|
|
||||||
remaining = max(0.0, cap - elapsed)
|
|
||||||
return {
|
|
||||||
"label": label,
|
|
||||||
"tone": tone,
|
|
||||||
"terminal": terminal,
|
|
||||||
"hint": hint,
|
|
||||||
"phase": current,
|
|
||||||
"timeline": timeline,
|
|
||||||
"percent": percent,
|
|
||||||
"done_steps": done_steps,
|
|
||||||
"start_steps": start_steps,
|
|
||||||
"trained_steps": trained_steps,
|
|
||||||
"total_steps": total_steps,
|
|
||||||
"elapsed_s": round(elapsed, 3),
|
|
||||||
"rate_per_s": rate,
|
|
||||||
"remaining_s": remaining,
|
|
||||||
"eta_epoch_s": eta_ts,
|
|
||||||
"log_total": snapshot.get("log_total"),
|
|
||||||
"log_capacity": MAX_LOG_LINES,
|
|
||||||
"metric_total": snapshot.get("metric_total"),
|
|
||||||
"latest": latest,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def _latest_rate(metrics: list[dict[str, Any]], window: int = 6) -> float | None:
|
|
||||||
"""最近若干个**带 fps 的**进度点的平均吞吐（步/秒），用于剩余时间估计。
|
|
||||||
|
|
||||||
progress JSON 行不含 fps，只有 SB3 表格行有；若只看最后 window 个点，
|
|
||||||
恰好落在若干 JSON 行之间时会得到 None，剩余时间就会退化成墙钟上限的
|
|
||||||
粗糙估计。这里向前扫描直到拿到至少 1 个 fps（最多看 3*window 个点）。
|
|
||||||
"""
|
|
||||||
if not metrics:
|
|
||||||
return None
|
|
||||||
usable: list[float] = []
|
|
||||||
for point in reversed(metrics[-window * 3:]):
|
|
||||||
value = finite_float(point.get("fps"))
|
|
||||||
if value and value > 0:
|
|
||||||
usable.append(value)
|
|
||||||
if len(usable) >= window:
|
|
||||||
break
|
|
||||||
if not usable:
|
|
||||||
return None
|
|
||||||
return sum(usable) / len(usable)
|
|
||||||
@@ -1,223 +0,0 @@
|
|||||||
"""训练前端示例快照：把一段"看起来像真的" PPO 训练曲线合成为合法快照。
|
|
||||||
|
|
||||||
用途有两个，都不涉及真实训练：
|
|
||||||
1. ``GET /api/train/state?preview=1`` 让界面在没有训练进程时也能验证渲染
|
|
||||||
（曲线、指标卡、进程时间线、日志着色、逐回合表）；
|
|
||||||
2. 单元测试可以断言前端的收敛判读分支（上升/平坦/KL 过猛）各有稳定输入。
|
|
||||||
|
|
||||||
曲线是解析函数 + 固定种子的确定性噪声，不含任何真实评测数据。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import math
|
|
||||||
import random
|
|
||||||
import time
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
from .analysis import describe_process, summarize_convergence
|
|
||||||
|
|
||||||
__all__ = ["DEMO_CONFIG", "DEMO_SCENARIOS", "demo_metrics", "demo_monitor",
|
|
||||||
"demo_log_lines", "demo_snapshot"]
|
|
||||||
|
|
||||||
DEMO_SCENARIOS = ("rising", "plateau", "kl-spike")
|
|
||||||
|
|
||||||
DEMO_CONFIG: dict[str, Any] = {
|
|
||||||
"problem": 3, "stage": 2, "timesteps": 12000, "seed": 320001,
|
|
||||||
"envs": 4, "vec_env": "subproc", "device": "mps",
|
|
||||||
"config_profile": "production", "reward_profile": "time",
|
|
||||||
"model": "output/hrl/ui/demo/q3.zip", "init": None, "resume": False,
|
|
||||||
"eval_episodes": 2, "rollout_steps": 256, "batch_size": 256,
|
|
||||||
"n_epochs": 10, "max_wall_s": 1200.0, "local_selector": "geometry",
|
|
||||||
"local_model": None, "verbose": 1,
|
|
||||||
}
|
|
||||||
|
|
||||||
DEMO_COMMAND = [
|
|
||||||
"python", "scripts/train_hrl_high.py", "--problem", "3", "--stage", "2",
|
|
||||||
"--timesteps", "12000", "--seed", "320001", "--envs", "4",
|
|
||||||
"--vec-env", "subproc", "--device", "mps", "--config-profile", "production",
|
|
||||||
"--reward-profile", "time", "--model", "output/hrl/ui/demo/q3.zip",
|
|
||||||
"--eval-episodes", "2", "--rollout-steps", "256", "--batch-size", "256",
|
|
||||||
"--n-epochs", "10", "--max-wall-s", "1200.0", "--local-selector",
|
|
||||||
"geometry", "--progress-json", "--verbose", "1",
|
|
||||||
]
|
|
||||||
|
|
||||||
DEMO_STEPS = 48
|
|
||||||
|
|
||||||
|
|
||||||
def _finite(value: float) -> float | None:
|
|
||||||
if not math.isfinite(value):
|
|
||||||
return None
|
|
||||||
return round(value, 8)
|
|
||||||
|
|
||||||
|
|
||||||
def demo_metrics(count: int = DEMO_STEPS, steps: int = 12000, *,
|
|
||||||
seed: int = 7, scenario: str = "rising",
|
|
||||||
total_steps: int | None = None, fps: float = 420.0
|
|
||||||
) -> list[dict[str, Any]]:
|
|
||||||
"""合成 count 个进度点，横轴覆盖到 ``total_steps`` 的 ``count/DEMO_STEPS``。
|
|
||||||
|
|
||||||
场景（用于验证收敛判读的各个分支，不含真实数据）：
|
|
||||||
rising 奖励对数饱和上升、KL 正常 → "收敛趋势良好"
|
|
||||||
plateau 奖励已进入平台期 → "已趋于平稳（疑似收敛）"
|
|
||||||
kl-spike 末段近似 KL 冲高 → "KL 偏大，策略更新可能过猛"
|
|
||||||
"""
|
|
||||||
if scenario not in DEMO_SCENARIOS:
|
|
||||||
raise ValueError(f"未知示例场景: {scenario}")
|
|
||||||
total = int(total_steps if total_steps is not None else steps)
|
|
||||||
rng = random.Random(seed)
|
|
||||||
points: list[dict[str, Any]] = []
|
|
||||||
per = total / DEMO_STEPS
|
|
||||||
ceiling = 42.0
|
|
||||||
span = min(1.0, max(1e-6, count / float(DEMO_STEPS)))
|
|
||||||
for index in range(1, count + 1):
|
|
||||||
t = (index / float(DEMO_STEPS)) / span # 折到整段训练的进度比例
|
|
||||||
steps_now = int(round(index * per))
|
|
||||||
if scenario == "plateau":
|
|
||||||
reward = 41.5 + rng.uniform(-0.35, 0.35)
|
|
||||||
value_loss_curve = 3.6 + rng.uniform(-0.2, 0.2)
|
|
||||||
approx_kl = max(0.001, 0.004 + rng.uniform(-0.001, 0.001))
|
|
||||||
explained = min(0.95, 0.88 + rng.uniform(-0.02, 0.02))
|
|
||||||
elif scenario == "kl-spike":
|
|
||||||
reward = ceiling * (1.0 - math.exp(-3.1 * t)) + rng.uniform(-1.2, 1.2)
|
|
||||||
value_loss_curve = 62.0 * math.exp(-2.4 * t) + 3.4
|
|
||||||
base_kl = 0.031 * math.exp(-1.7 * t)
|
|
||||||
approx_kl = base_kl + (0.075 if t > 0.8 else 0.0) + rng.uniform(-0.003, 0.003)
|
|
||||||
explained = -0.09 + 1.05 * (1.0 - math.exp(-2.2 * t))
|
|
||||||
else:
|
|
||||||
noise = rng.uniform(-1.4, 1.4) * (1.0 - 0.6 * t)
|
|
||||||
reward = ceiling * (1.0 - math.exp(-3.1 * t)) + noise
|
|
||||||
value_loss_curve = 62.0 * math.exp(-2.4 * t) + 3.4
|
|
||||||
approx_kl = 0.031 * math.exp(-1.7 * t)
|
|
||||||
explained = -0.09 + 1.05 * (1.0 - math.exp(-2.2 * t))
|
|
||||||
value_loss = value_loss_curve + rng.uniform(-0.5, 0.5)
|
|
||||||
policy_loss = -0.021 + 0.006 * math.sin(index / 4.0) + rng.uniform(-0.002, 0.002)
|
|
||||||
entropy = 1.35 - 0.55 * (t if scenario != "plateau" else 0.9) \
|
|
||||||
+ rng.uniform(-0.02, 0.02)
|
|
||||||
points.append({
|
|
||||||
"timesteps": steps_now,
|
|
||||||
"n_updates": index * 12,
|
|
||||||
"ep_rew_mean": _finite(reward),
|
|
||||||
"ep_len_mean": _finite(24.0 + 21.0 * t + rng.uniform(-1.5, 1.5)),
|
|
||||||
"value_loss": _finite(value_loss),
|
|
||||||
"policy_gradient_loss": _finite(policy_loss),
|
|
||||||
"approx_kl": _finite(max(0.0008, approx_kl + rng.uniform(-0.002, 0.002))),
|
|
||||||
"entropy_loss": _finite(entropy),
|
|
||||||
"explained_variance": _finite(min(0.97, explained
|
|
||||||
+ rng.uniform(-0.03, 0.03))),
|
|
||||||
# 吞吐尽量接近真实量级（envs=4 时约 400+ 步/秒），否则剩余时间
|
|
||||||
# 估计会给出荒谬的 ETA
|
|
||||||
"fps": _finite(max(30.0, fps * (0.92 + 0.16 * t)
|
|
||||||
+ rng.uniform(-12.0, 12.0))),
|
|
||||||
})
|
|
||||||
return points
|
|
||||||
|
|
||||||
|
|
||||||
def demo_monitor(count: int = 60, seed: int = 11) -> list[dict[str, Any]]:
|
|
||||||
rng = random.Random(seed)
|
|
||||||
rows = []
|
|
||||||
clock = 0.0
|
|
||||||
for index in range(count):
|
|
||||||
t = index / max(1, count - 1)
|
|
||||||
clock += rng.uniform(4.0, 14.0)
|
|
||||||
rows.append({
|
|
||||||
"r": _finite(38.0 * (1.0 - math.exp(-3.0 * t)) + rng.uniform(-3.0, 3.0)),
|
|
||||||
"l": float(rng.randint(28, 74)),
|
|
||||||
"t": round(clock, 4),
|
|
||||||
})
|
|
||||||
return rows
|
|
||||||
|
|
||||||
|
|
||||||
def demo_log_lines() -> list[str]:
|
|
||||||
"""一段包含各类着色分支的日志：SB3 表格、进度 JSON、警告与收尾。"""
|
|
||||||
lines = [
|
|
||||||
"=== 训练前端离线预览：以下为示例快照，非真实训练输出 ===",
|
|
||||||
"使用设备: mps",
|
|
||||||
"创建 4 个 DummyVecEnv/SubprocVecEnv 环境 (stage=2, problem=3)",
|
|
||||||
"Monitor CSV: output/hrl/ui/demo/high_curve_q3_env0.monitor.csv",
|
|
||||||
"-----------------------------",
|
|
||||||
"| rollout/ | |",
|
|
||||||
"| ep_len_mean | 31.4 |",
|
|
||||||
"| ep_rew_mean | 12.83 |",
|
|
||||||
"| time/ | |",
|
|
||||||
"| fps | 412 |",
|
|
||||||
"| total_timesteps | 512 |",
|
|
||||||
"-----------------------------",
|
|
||||||
]
|
|
||||||
for index in range(1, 9):
|
|
||||||
t = index / 8.0
|
|
||||||
lines.append(
|
|
||||||
'{"event": "progress", "timesteps": %d, "n_updates": %d, '
|
|
||||||
'"ep_rew_mean": %.3f, "ep_len_mean": %.2f, "value_loss": %.4f, '
|
|
||||||
'"policy_gradient_loss": %.5f, "approx_kl": %.5f, '
|
|
||||||
'"entropy_loss": %.4f, "explained_variance": %.4f, "fps": %.1f}'
|
|
||||||
% (index * 1500, index * 12, 42.0 * (1 - math.exp(-3.1 * t)),
|
|
||||||
24 + 21 * t, 62.0 * math.exp(-2.4 * t) + 3.4, -0.021 + 0.006 * t,
|
|
||||||
0.031 * math.exp(-1.7 * t), 1.35 - 0.55 * t,
|
|
||||||
-0.09 + 1.05 * (1 - math.exp(-2.2 * t)), 390 + 70 * t)
|
|
||||||
)
|
|
||||||
lines += [
|
|
||||||
"UserWarning: 示例警告行，用于验证警告着色",
|
|
||||||
"确定性评估：2/2 成功，accounted=20，无异常",
|
|
||||||
"模型已保存: output/hrl/ui/demo/q3.zip",
|
|
||||||
"元数据已写入: output/hrl/ui/demo/q3.json",
|
|
||||||
]
|
|
||||||
return lines
|
|
||||||
|
|
||||||
|
|
||||||
def demo_snapshot(*, log_tail: int = 400, scenario: str = "rising",
|
|
||||||
done_ratio: float = 0.62, fps: float = 420.0) -> dict[str, Any]:
|
|
||||||
"""合成一次"跑到中途"的快照（键与 TrainingJob.snapshot 一致）。
|
|
||||||
|
|
||||||
``done_ratio`` 控制已完成的步数比例，用来让预览更接近真实运行中状态
|
|
||||||
（而不是刚好 100%）；``scenario`` 选择收敛判读分支；``fps`` 决定剩余
|
|
||||||
时间估计的吞吐量级。
|
|
||||||
"""
|
|
||||||
now = time.time()
|
|
||||||
ratio = min(1.0, max(0.05, float(done_ratio)))
|
|
||||||
metrics = demo_metrics(count=max(6, int(round(DEMO_STEPS * ratio))),
|
|
||||||
total_steps=int(DEMO_CONFIG["timesteps"]),
|
|
||||||
scenario=scenario, fps=fps)
|
|
||||||
monitor = demo_monitor(count=max(6, int(round(60 * ratio))))
|
|
||||||
config = dict(DEMO_CONFIG)
|
|
||||||
lines = demo_log_lines()
|
|
||||||
logs = (lines * 12)[-max(1, int(log_tail)):]
|
|
||||||
status = "running"
|
|
||||||
payload: dict[str, Any] = {
|
|
||||||
"job_id": "job-preview",
|
|
||||||
"status": status,
|
|
||||||
"pid": 42424,
|
|
||||||
"created_at": now - 186.0,
|
|
||||||
"started_at": now - 184.0,
|
|
||||||
"finished_at": None,
|
|
||||||
"elapsed_s": 184.0,
|
|
||||||
"returncode": None,
|
|
||||||
"error": None,
|
|
||||||
"forced_kill": False,
|
|
||||||
"command": list(DEMO_COMMAND),
|
|
||||||
"model": str(config["model"]),
|
|
||||||
"config": config,
|
|
||||||
"log_lines": logs,
|
|
||||||
"log_total": len(logs) * 3,
|
|
||||||
"metrics": metrics,
|
|
||||||
"metric_offset": 0,
|
|
||||||
"metric_end": len(metrics),
|
|
||||||
"metric_total": len(metrics),
|
|
||||||
"monitor": monitor,
|
|
||||||
"recovery": {
|
|
||||||
"can_resume": True,
|
|
||||||
"model": str(config["model"]),
|
|
||||||
"timesteps": int(DEMO_STEPS * ratio * 12),
|
|
||||||
"reason": "recovery",
|
|
||||||
"updated_unix_s": now - 24.0,
|
|
||||||
"source": "resume.json",
|
|
||||||
"hint_file": str(config["model"]).rsplit("/", 1)[0] + "/resume.json",
|
|
||||||
"candidates": 3,
|
|
||||||
"preview": True,
|
|
||||||
},
|
|
||||||
"preview": True,
|
|
||||||
"preview_scenario": scenario,
|
|
||||||
}
|
|
||||||
payload["convergence"] = summarize_convergence(metrics, monitor)
|
|
||||||
payload["process"] = describe_process(payload)
|
|
||||||
return payload
|
|
||||||
@@ -1,328 +0,0 @@
|
|||||||
"""训练前端配置契约：字段元数据、分组、预设与指标字典。
|
|
||||||
|
|
||||||
为什么单独成模块：前端表单的取值域、单位、分组与后端 ``TrainConfig`` 的
|
|
||||||
校验规则必须来自同一处，否则界面会允许后端拒绝的输入（或反之）。这里只
|
|
||||||
声明"可展示"的元数据，真正的合法性判定仍由 :class:`TrainConfig.validate`
|
|
||||||
单点负责——前端只做提前提示，不复制规则。
|
|
||||||
|
|
||||||
指标字典 ``METRIC_CATALOG`` 同样是唯一出处：``server.summarize_convergence``
|
|
||||||
用它取数，``/api/train/schema`` 把它交给前端画图，避免两边各写一份键名。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
from typing import Any
|
|
||||||
|
|
||||||
__all__ = [
|
|
||||||
"CONFIG_FIELDS", "CONFIG_GROUPS", "PRESETS", "METRIC_CATALOG",
|
|
||||||
"config_schema", "preset_payloads", "metric_keys",
|
|
||||||
]
|
|
||||||
|
|
||||||
# 字段元数据：
|
|
||||||
# kind number|integer|text|select|bool
|
|
||||||
# 其余键 前端直接使用（label/unit/group/choices/min/max/step/help/advanced）
|
|
||||||
CONFIG_FIELDS: list[dict[str, Any]] = [
|
|
||||||
# ---------------------------------------------------------- 环境与课程
|
|
||||||
{
|
|
||||||
"key": "problem", "kind": "select", "group": "env",
|
|
||||||
"label": "问题", "default": 3, "choices": [
|
|
||||||
{"value": 3, "label": "问题 3（全向源）"},
|
|
||||||
{"value": 4, "label": "问题 4（含定向源）"},
|
|
||||||
],
|
|
||||||
"help": "决定场景源类型与末段课程配置。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "stage", "kind": "select", "group": "env",
|
|
||||||
"label": "课程阶段", "default": 1, "choices": [
|
|
||||||
{"value": 1, "label": "1 · 单源 / 5 频道 / 无噪"},
|
|
||||||
{"value": 2, "label": "2 · 3 源 / 10 频道 / ±0.5°"},
|
|
||||||
{"value": 3, "label": "3 · 8 源 / 16 频道 / ±1°"},
|
|
||||||
{"value": 4, "label": "4 · 10–16 源 / 20 频道 / ±1°"},
|
|
||||||
],
|
|
||||||
"help": "课程递进；跨阶段续训用下方 --init 迁移权重。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "config_profile", "kind": "select", "group": "env",
|
|
||||||
"label": "配置档", "default": "production", "choices": [
|
|
||||||
{"value": "production", "label": "production（生产保真）"},
|
|
||||||
{"value": "coarse-pretrain", "label": "coarse-pretrain（快速粗训）"},
|
|
||||||
],
|
|
||||||
"help": "coarse-pretrain 显式映射为 fast 评分配置并写入模型元数据。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "reward_profile", "kind": "select", "group": "env",
|
|
||||||
"label": "奖励口径", "default": "time", "choices": [
|
|
||||||
{"value": "time", "label": "time（P0.4 纯时间目标）"},
|
|
||||||
{"value": "prompt", "label": "prompt（§5.2 原始塑形）"},
|
|
||||||
],
|
|
||||||
"help": "对照实验用；口径固化进模型元数据，不静默替换。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "seed", "kind": "integer", "group": "env",
|
|
||||||
"label": "随机种子", "default": 320001, "min": 0, "max": 2_147_483_647,
|
|
||||||
"help": "不同种子用于多次重复训练与方差估计。",
|
|
||||||
},
|
|
||||||
# ---------------------------------------------------------- PPO 超参
|
|
||||||
{
|
|
||||||
"key": "rollout_steps", "kind": "integer", "group": "ppo",
|
|
||||||
"label": "采样步数 n_steps", "default": 128, "min": 1, "max": 100_000,
|
|
||||||
"help": "每次更新前每个环境采集的步数；近似一个 rollout 的宽度。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "batch_size", "kind": "integer", "group": "ppo",
|
|
||||||
"label": "批大小 batch_size", "default": 128, "min": 1, "max": 100_000,
|
|
||||||
"help": "建议 batch_size ≤ n_steps × envs。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "n_epochs", "kind": "integer", "group": "ppo",
|
|
||||||
"label": "更新轮数 n_epochs", "default": 3, "min": 1, "max": 100,
|
|
||||||
"help": "每批数据的 SGD 轮数；过大易使 KL 飙升。",
|
|
||||||
},
|
|
||||||
# ---------------------------------------------------------- 运行环境
|
|
||||||
{
|
|
||||||
"key": "envs", "kind": "integer", "group": "runtime",
|
|
||||||
"label": "并行环境数", "default": 1, "min": 1, "max": 32,
|
|
||||||
"help": "SubprocVecEnv 下每个 worker 一个进程与 Monitor 文件。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "vec_env", "kind": "select", "group": "runtime",
|
|
||||||
"label": "向量化方式", "default": "dummy", "choices": [
|
|
||||||
{"value": "dummy", "label": "dummy（单进程，调试友好）"},
|
|
||||||
{"value": "subproc", "label": "subproc（多进程，吞吐优先）"},
|
|
||||||
],
|
|
||||||
"help": "envs>1 且追求吞吐时选 subproc。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "device", "kind": "select", "group": "runtime",
|
|
||||||
"label": "计算设备", "default": "auto", "choices": [
|
|
||||||
{"value": "auto", "label": "auto（自动选择）"},
|
|
||||||
{"value": "cpu", "label": "cpu"},
|
|
||||||
{"value": "cuda", "label": "cuda（NVIDIA）"},
|
|
||||||
{"value": "mps", "label": "mps（Apple 芯片）"},
|
|
||||||
],
|
|
||||||
"help": "CPU 路径最稳；GPU 需与 torch 构建匹配。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "local_selector", "kind": "select", "group": "runtime",
|
|
||||||
"label": "低层执行器", "default": "geometry", "choices": [
|
|
||||||
{"value": "geometry", "label": "geometry（几何选点，默认）"},
|
|
||||||
{"value": "rl", "label": "rl（低层 PPO，接口预留）"},
|
|
||||||
],
|
|
||||||
"help": "rl 仅记录元数据，当前 HTTP 执行路径仍由高层宏动作终点决定。",
|
|
||||||
},
|
|
||||||
# ---------------------------------------------------------- 训练规模
|
|
||||||
{
|
|
||||||
"key": "timesteps", "kind": "integer", "group": "scale",
|
|
||||||
"label": "总步数 timesteps", "default": 2048, "min": 1,
|
|
||||||
"max": 10_000_000, "help": "本次训练的环境步总数（续训时为增量步数）。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "max_wall_s", "kind": "number", "group": "scale",
|
|
||||||
"label": "墙钟上限 (s)", "default": 1200.0, "min": 0.1, "max": 7200.0,
|
|
||||||
"step": 30, "help": "硬上限：到点由回调主动停止并保存模型。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "eval_episodes", "kind": "integer", "group": "scale",
|
|
||||||
"label": "评估回合数", "default": 2, "min": 0, "max": 1000,
|
|
||||||
"help": "训练结束后确定性评估；0 表示跳过。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "verbose", "kind": "select", "group": "scale",
|
|
||||||
"label": "SB3 日志级别", "default": 1, "choices": [
|
|
||||||
{"value": 0, "label": "0 · 静默（只留 JSON 进度）"},
|
|
||||||
{"value": 1, "label": "1 · 表格（默认）"},
|
|
||||||
{"value": 2, "label": "2 · 调试"},
|
|
||||||
],
|
|
||||||
"help": "前端已解析 SB3 表格，默认 1 便于对照原始输出。",
|
|
||||||
},
|
|
||||||
# ---------------------------------------------------------- 模型路径
|
|
||||||
{
|
|
||||||
"key": "model", "kind": "text", "group": "paths",
|
|
||||||
"label": "模型输出路径", "default": "output/hrl/ui/q3.zip",
|
|
||||||
"help": "相对仓库根；同目录会写入 .json 元数据与 Monitor CSV。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "init", "kind": "text", "group": "paths", "optional": True,
|
|
||||||
"label": "初始权重 --init", "default": None,
|
|
||||||
"help": "留空=从零；填上一阶段 .zip 即课程迁移。PPO 超参与步数计数沿用"
|
|
||||||
"该模型（timesteps 按增量理解，进度条按增量显示）。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "local_model", "kind": "text", "group": "paths", "optional": True,
|
|
||||||
"label": "低层模型 --local-model", "default": None,
|
|
||||||
"advanced": True,
|
|
||||||
"help": "仅 local_selector=rl 时有意义。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "resume", "kind": "bool", "group": "paths",
|
|
||||||
"label": "续训 --resume", "default": False,
|
|
||||||
"help": "从模型输出路径已有 .zip 继续，timesteps 按增量理解。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "force", "kind": "bool", "group": "paths",
|
|
||||||
"label": "覆盖已有模型 --force", "default": False,
|
|
||||||
"help": "输出路径已存在时必须勾选，否则训练脚本会拒绝覆盖。与续训互斥。",
|
|
||||||
},
|
|
||||||
# ---------------------------------------------------------- 恢复与落盘
|
|
||||||
{
|
|
||||||
"key": "recovery_every", "kind": "integer", "group": "recovery",
|
|
||||||
"label": "滚动恢复点间隔 --recovery-every", "default": 2000,
|
|
||||||
"help": "每 N 个环境步原子覆盖写 <checkpoints>/<stem>_latest.zip，"
|
|
||||||
"训练开始前先写起步基线；急停（强杀）后最多丢 N 步。0=关闭。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "checkpoint_every", "kind": "integer", "group": "recovery",
|
|
||||||
"label": "历史 checkpoint 间隔 --checkpoint-every", "default": 20000,
|
|
||||||
"help": "每 N 个环境步另留一份 <stem>_<步数>.zip 供返祖对比；"
|
|
||||||
"0=只保留滚动恢复点。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "checkpoint_keep", "kind": "integer", "group": "recovery",
|
|
||||||
"label": "历史 checkpoint 保留数 --checkpoint-keep", "default": 3,
|
|
||||||
"help": "只保留最近 K 个历史 checkpoint；滚动恢复点与起步基线不受影响。",
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
CONFIG_GROUPS: list[dict[str, str]] = [
|
|
||||||
{"key": "env", "label": "环境与课程", "hint": "题目、课程阶段与奖励口径"},
|
|
||||||
{"key": "ppo", "label": "PPO 超参数", "hint": "采样宽度与更新强度"},
|
|
||||||
{"key": "runtime", "label": "运行环境", "hint": "并行、设备与低层执行器"},
|
|
||||||
{"key": "scale", "label": "训练规模", "hint": "步数、墙钟上限与评估"},
|
|
||||||
{"key": "recovery", "label": "恢复与落盘", "hint": "急停恢复点与历史 checkpoint"},
|
|
||||||
{"key": "paths", "label": "模型与迁移", "hint": "产物路径、续训与课程迁移"},
|
|
||||||
]
|
|
||||||
|
|
||||||
# 预设 = 只覆盖差异字段；后端 from_payload 会补齐其余默认值。
|
|
||||||
PRESETS: list[dict[str, Any]] = [
|
|
||||||
{
|
|
||||||
"key": "smoke",
|
|
||||||
"label": "冒烟（约 1 分钟）",
|
|
||||||
"note": "64 步、CPU、无评估——只验证链路与界面。",
|
|
||||||
"payload": {
|
|
||||||
"problem": 3, "stage": 1, "timesteps": 64, "seed": 1,
|
|
||||||
"envs": 1, "vec_env": "dummy", "device": "cpu",
|
|
||||||
"config_profile": "coarse-pretrain", "reward_profile": "time",
|
|
||||||
"model": "output/hrl/ui/smoke/q3.zip", "eval_episodes": 0,
|
|
||||||
"rollout_steps": 32, "batch_size": 32, "n_epochs": 2,
|
|
||||||
"max_wall_s": 180.0, "verbose": 1, "force": True,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "local-check",
|
|
||||||
"label": "本地验证（约 10 分钟）",
|
|
||||||
"note": "stage 1 单源课程 2048 步 + 2 回合评估。",
|
|
||||||
"payload": {
|
|
||||||
"problem": 3, "stage": 1, "timesteps": 2048, "seed": 320001,
|
|
||||||
"envs": 1, "vec_env": "dummy", "device": "auto",
|
|
||||||
"config_profile": "production", "reward_profile": "time",
|
|
||||||
"model": "output/hrl/ui/local/q3.zip", "eval_episodes": 2,
|
|
||||||
"rollout_steps": 128, "batch_size": 128, "n_epochs": 3,
|
|
||||||
"max_wall_s": 900.0, "verbose": 1, "force": True,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "q3-main",
|
|
||||||
"label": "Q3 主训练",
|
|
||||||
"note": "stage 3 八源课程，4 进程并行；建议在训练机上跑。",
|
|
||||||
"payload": {
|
|
||||||
"problem": 3, "stage": 3, "timesteps": 200000, "seed": 320001,
|
|
||||||
"envs": 4, "vec_env": "subproc", "device": "auto",
|
|
||||||
"config_profile": "production", "reward_profile": "time",
|
|
||||||
"model": "output/hrl/ui/q3-main/q3.zip", "eval_episodes": 5,
|
|
||||||
"rollout_steps": 256, "batch_size": 256, "n_epochs": 10,
|
|
||||||
"max_wall_s": 7200.0, "verbose": 1,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "q4-transfer",
|
|
||||||
"label": "Q4 课程迁移",
|
|
||||||
"note": "从 Q3 权重迁移到问题 4 末段课程（含定向源）。",
|
|
||||||
"payload": {
|
|
||||||
"problem": 4, "stage": 4, "timesteps": 120000, "seed": 420001,
|
|
||||||
"envs": 4, "vec_env": "subproc", "device": "auto",
|
|
||||||
"config_profile": "production", "reward_profile": "time",
|
|
||||||
"model": "output/hrl/ui/q4-main/q4.zip",
|
|
||||||
"init": "output/hrl/ui/q3-main/q3.zip", "eval_episodes": 5,
|
|
||||||
"rollout_steps": 256, "batch_size": 256, "n_epochs": 10,
|
|
||||||
"max_wall_s": 7200.0, "verbose": 1,
|
|
||||||
},
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
# 指标字典：键来自训练脚本 --progress-json 的单行 JSON（短名）与 SB3 表格
|
|
||||||
# （归一化去前缀）。前端按此顺序绘图，收敛判据在 server.summarize_convergence。
|
|
||||||
METRIC_CATALOG: list[dict[str, Any]] = [
|
|
||||||
{
|
|
||||||
"key": "ep_rew_mean", "label": "平均回合奖励", "role": "primary",
|
|
||||||
"color": "#4f8cff", "format": "number", "better": "up",
|
|
||||||
"help": "最近若干回合的平均回报；主要收敛判据。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "ep_len_mean", "label": "平均回合长度", "role": "secondary",
|
|
||||||
"color": "#38bdf8", "format": "number", "better": "neutral",
|
|
||||||
"help": "回合步数；与奖励同时上升通常代表策略更完整而非更快。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "approx_kl", "label": "近似 KL", "role": "stability",
|
|
||||||
"color": "#f59e0b", "format": "sci", "better": "down",
|
|
||||||
"help": "策略更新幅度；> 0.05 视为更新过猛。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "explained_variance", "label": "解释方差", "role": "stability",
|
|
||||||
"color": "#34d399", "format": "number", "better": "up",
|
|
||||||
"help": "价值函数拟合优度；< 0 表示还不如均值预测。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "value_loss", "label": "价值损失", "role": "stability",
|
|
||||||
"color": "#f87171", "format": "sci", "better": "down",
|
|
||||||
"help": "critic 拟合误差。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "policy_gradient_loss", "label": "策略梯度损失", "role": "stability",
|
|
||||||
"color": "#c084fc", "format": "sci", "better": "neutral",
|
|
||||||
"help": "actor 的代理损失；量级小且符号可变属正常。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "entropy_loss", "label": "策略熵", "role": "exploration",
|
|
||||||
"color": "#fbbf24", "format": "number", "better": "neutral",
|
|
||||||
"help": "探索度；若长期贴近 0 说明策略过早确定。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "n_updates", "label": "更新次数", "role": "throughput",
|
|
||||||
"color": "#93a1bd", "format": "integer", "better": "up",
|
|
||||||
"help": "已完成的 PPO 更新轮数。",
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"key": "fps", "label": "吞吐 FPS", "role": "throughput",
|
|
||||||
"color": "#22d3ee", "format": "number", "better": "up",
|
|
||||||
"help": "每秒环境步数，反映硬件与并行配置是否吃满。",
|
|
||||||
},
|
|
||||||
]
|
|
||||||
|
|
||||||
_METRIC_BY_KEY = {item["key"]: item for item in METRIC_CATALOG}
|
|
||||||
|
|
||||||
|
|
||||||
def metric_keys() -> list[str]:
|
|
||||||
return [item["key"] for item in METRIC_CATALOG]
|
|
||||||
|
|
||||||
|
|
||||||
def metric_meta(key: str) -> dict[str, Any]:
|
|
||||||
return _METRIC_BY_KEY.get(key, {"key": key, "label": key, "role": "other",
|
|
||||||
"color": "#93a1bd", "format": "number",
|
|
||||||
"better": "neutral", "help": ""})
|
|
||||||
|
|
||||||
|
|
||||||
def config_schema() -> dict[str, Any]:
|
|
||||||
"""返回给前端的完整配置契约（字段 + 分组 + 预设 + 指标字典）。"""
|
|
||||||
return {
|
|
||||||
"fields": CONFIG_FIELDS,
|
|
||||||
"groups": CONFIG_GROUPS,
|
|
||||||
"presets": [{"key": item["key"], "label": item["label"],
|
|
||||||
"note": item["note"], "payload": dict(item["payload"])}
|
|
||||||
for item in PRESETS],
|
|
||||||
"metrics": METRIC_CATALOG,
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
def preset_payloads() -> dict[str, dict[str, Any]]:
|
|
||||||
return {item["key"]: dict(item["payload"]) for item in PRESETS}
|
|
||||||
@@ -1,977 +0,0 @@
|
|||||||
"""训练前端服务器：在评测前端同一套「本地回环 HTTP + 静态白名单」模式上，
|
|
||||||
额外托管一个受控的训练子进程，并把 stdout/JSON 进度流做成轮询 API。
|
|
||||||
|
|
||||||
设计边界：
|
|
||||||
- 只监听 127.0.0.1，只允许白名单静态文件；
|
|
||||||
- 只允许一个训练任务，命令由服务端用参数列表构造，不经过 shell；
|
|
||||||
- 支持启动/暂停/继续/停止/状态查询；暂停与停止通过**控制文件**下发，
|
|
||||||
POSIX 与 Windows 行为一致（详见 :mod:`mcm26b.process_control`）；
|
|
||||||
- 训练进程通过 ``--progress-json`` 输出单行 JSON 进度，服务端同时兼容
|
|
||||||
Stable-Baselines3 的表格输出和 Monitor CSV。
|
|
||||||
|
|
||||||
界面契约（供 ``static/`` 使用）：
|
|
||||||
- ``GET /api/train/schema`` 字段/分组/预设/指标字典（``schema.py`` 单一出处）
|
|
||||||
- ``GET /api/train/state`` 快照；``after_metric=N`` 只回增量指标
|
|
||||||
- ``POST /api/train/start`` 以部分字段启动（其余取默认值）
|
|
||||||
- ``POST /api/train/pause`` 请求训练进程在 step 边界挂起
|
|
||||||
- ``POST /api/train/resume`` 解除挂起
|
|
||||||
- ``POST /api/train/stop`` 请求优雅停止，超时后强杀进程树
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import dataclasses
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import re
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
import threading
|
|
||||||
import time
|
|
||||||
from collections import deque
|
|
||||||
from dataclasses import dataclass
|
|
||||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
|
||||||
from pathlib import Path
|
|
||||||
from typing import Any
|
|
||||||
from urllib.parse import parse_qs, urlsplit
|
|
||||||
|
|
||||||
from ..process_control import (DEFAULT_GRACE_S, spawn_kwargs,
|
|
||||||
terminate_process_tree, write_control,
|
|
||||||
write_json_atomic)
|
|
||||||
from .analysis import (MAX_LOG_LINES, MAX_METRICS, describe_process, finite_float,
|
|
||||||
process_pid, summarize_convergence)
|
|
||||||
from .demo import DEMO_SCENARIOS, demo_snapshot
|
|
||||||
from .schema import config_schema
|
|
||||||
|
|
||||||
__all__ = ["TrainingJob", "TrainingUI", "run_training_ui", "build_command",
|
|
||||||
"summarize_convergence", "describe_process", "discover_recovery"]
|
|
||||||
|
|
||||||
STATIC_DIR = Path(__file__).resolve().parent / "static"
|
|
||||||
REPO_ROOT = Path(__file__).resolve().parents[3]
|
|
||||||
TRAIN_SCRIPT = REPO_ROOT / "scripts" / "train_hrl_high.py"
|
|
||||||
# 控制文件集中放在 output/ 下（已 gitignore），job 之间互不干扰
|
|
||||||
CONTROL_DIR = REPO_ROOT / "output" / "hrl" / "ui" / "control"
|
|
||||||
# 从发出停止请求到强杀之间留给训练脚本收尾的秒数
|
|
||||||
STOP_GRACE_S = DEFAULT_GRACE_S
|
|
||||||
STATIC_FILES = {
|
|
||||||
"/": ("index.html", "text/html; charset=utf-8"),
|
|
||||||
"/index.html": ("index.html", "text/html; charset=utf-8"),
|
|
||||||
"/app.js": ("app.js", "application/javascript; charset=utf-8"),
|
|
||||||
"/charts.js": ("charts.js", "application/javascript; charset=utf-8"),
|
|
||||||
"/style.css": ("style.css", "text/css; charset=utf-8"),
|
|
||||||
}
|
|
||||||
|
|
||||||
_SB3_SECTIONS = ("rollout/", "time/", "train/", "system/")
|
|
||||||
# 路径/字符串参数只拒绝控制字符（含 \r\n\t 与 DEL），不做 shell 元字符过滤：
|
|
||||||
# 命令走参数列表、不经过 shell，这里只是防日志与命令行被注入控制序列。
|
|
||||||
# 注意必须用单反斜杠转义——写成 r"...\\x00" 会变成"字面反斜杠到反斜杠"的
|
|
||||||
# 字符类，从而把所有含 `\` 或看起来像路径分隔的输入误判为非法。
|
|
||||||
_SAFE_ARG_RE = re.compile(r"^[^\x00-\x1f\x7f]{1,512}$")
|
|
||||||
|
|
||||||
|
|
||||||
def _normalize_metric(key: str) -> str:
|
|
||||||
"""把 SB3 的 ``rollout/ep_rew_mean`` 与进度 JSON 的 ``ep_rew_mean``
|
|
||||||
归一化到同一短名，便于前端绘制。"""
|
|
||||||
key = key.strip().replace(" ", "_")
|
|
||||||
if "/" in key:
|
|
||||||
key = key.rsplit("/", 1)[-1]
|
|
||||||
return key
|
|
||||||
|
|
||||||
|
|
||||||
@dataclass(frozen=True)
|
|
||||||
class TrainConfig:
|
|
||||||
problem: int = 3
|
|
||||||
stage: int = 1
|
|
||||||
timesteps: int = 2048
|
|
||||||
seed: int = 320001
|
|
||||||
envs: int = 1
|
|
||||||
vec_env: str = "dummy"
|
|
||||||
device: str = "auto"
|
|
||||||
config_profile: str = "production"
|
|
||||||
reward_profile: str = "time"
|
|
||||||
model: str = "output/hrl/ui/q3.zip"
|
|
||||||
init: str | None = None
|
|
||||||
resume: bool = False
|
|
||||||
force: bool = False
|
|
||||||
eval_episodes: int = 2
|
|
||||||
rollout_steps: int = 128
|
|
||||||
batch_size: int = 128
|
|
||||||
n_epochs: int = 3
|
|
||||||
max_wall_s: float = 1200.0
|
|
||||||
local_selector: str = "geometry"
|
|
||||||
local_model: str | None = None
|
|
||||||
# 落盘/恢复：默认值与 scripts/train_hrl_high.py 保持一致
|
|
||||||
checkpoint_every: int = 20000
|
|
||||||
checkpoint_keep: int = 3
|
|
||||||
recovery_every: int = 2000
|
|
||||||
verbose: int = 1
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
def from_payload(cls, payload: dict[str, Any]) -> "TrainConfig":
|
|
||||||
if not isinstance(payload, dict):
|
|
||||||
raise ValueError("请求体必须是 JSON 对象")
|
|
||||||
defaults = cls()
|
|
||||||
data: dict[str, Any] = {}
|
|
||||||
# 用 dataclasses.fields 而不是 cls.__dataclass_fields__：后者是内部
|
|
||||||
# 实现细节，pylint 看不到它，也无法保证未来版本仍存在。
|
|
||||||
for field in dataclasses.fields(cls):
|
|
||||||
if field.name in payload and payload[field.name] is not None:
|
|
||||||
data[field.name] = payload[field.name]
|
|
||||||
cfg = cls(**{**defaults.__dict__, **data})
|
|
||||||
cfg.validate()
|
|
||||||
return cfg
|
|
||||||
|
|
||||||
def validate(self) -> None:
|
|
||||||
if self.problem not in (3, 4):
|
|
||||||
raise ValueError("problem 必须是 3 或 4")
|
|
||||||
if self.stage not in (1, 2, 3, 4):
|
|
||||||
raise ValueError("stage 必须在 1..4")
|
|
||||||
if not 1 <= int(self.timesteps) <= 10_000_000:
|
|
||||||
raise ValueError("timesteps 必须在 1..10,000,000")
|
|
||||||
if bool(self.resume) and bool(self.force):
|
|
||||||
raise ValueError("--resume 与 --force 互斥：续训不可能同时覆盖")
|
|
||||||
if not 1 <= int(self.envs) <= 32:
|
|
||||||
raise ValueError("envs 必须在 1..32")
|
|
||||||
if self.vec_env not in ("dummy", "subproc"):
|
|
||||||
raise ValueError("vec_env 必须是 dummy 或 subproc")
|
|
||||||
if self.device not in ("auto", "cpu", "cuda", "mps"):
|
|
||||||
raise ValueError("device 必须是 auto/cpu/cuda/mps")
|
|
||||||
if self.config_profile not in ("production", "coarse-pretrain"):
|
|
||||||
raise ValueError("config_profile 非法")
|
|
||||||
if self.reward_profile not in ("time", "prompt"):
|
|
||||||
raise ValueError("reward_profile 非法")
|
|
||||||
if self.local_selector not in ("geometry", "rl"):
|
|
||||||
raise ValueError("local_selector 非法")
|
|
||||||
if not 1 <= int(self.rollout_steps) <= int(self.timesteps):
|
|
||||||
raise ValueError("rollout_steps 必须在 1..timesteps")
|
|
||||||
if not 1 <= int(self.n_epochs) <= 100:
|
|
||||||
raise ValueError("n_epochs 必须在 1..100")
|
|
||||||
if not 0.0 < float(self.max_wall_s) <= 7200.0:
|
|
||||||
raise ValueError("max_wall_s 必须在 0..7200 秒")
|
|
||||||
if int(self.eval_episodes) < 0:
|
|
||||||
raise ValueError("eval_episodes 不能为负")
|
|
||||||
if not 0 <= int(self.checkpoint_every) <= 10_000_000:
|
|
||||||
raise ValueError("checkpoint_every 必须在 0..10,000,000")
|
|
||||||
if not 1 <= int(self.checkpoint_keep) <= 100:
|
|
||||||
raise ValueError("checkpoint_keep 必须在 1..100")
|
|
||||||
if not 0 <= int(self.recovery_every) <= 10_000_000:
|
|
||||||
raise ValueError("recovery_every 必须在 0..10,000,000")
|
|
||||||
for label, value in (("model", self.model), ("init", self.init),
|
|
||||||
("local_model", self.local_model)):
|
|
||||||
if value is None:
|
|
||||||
continue
|
|
||||||
if not isinstance(value, str) or not _SAFE_ARG_RE.match(value):
|
|
||||||
raise ValueError(f"{label} 路径含非法字符")
|
|
||||||
|
|
||||||
@property
|
|
||||||
def model_path(self) -> Path:
|
|
||||||
return (REPO_ROOT / self.model).resolve() if not Path(self.model).is_absolute() \
|
|
||||||
else Path(self.model).resolve()
|
|
||||||
|
|
||||||
def command(self) -> list[str]:
|
|
||||||
return build_command(self)
|
|
||||||
|
|
||||||
|
|
||||||
def _int_query(params: dict[str, list[str]], name: str, default: int | None,
|
|
||||||
low: int, high: int, *, optional: bool = False) -> int | None:
|
|
||||||
"""解析整数查询参数；越界或非整数一律报错，不用静默钳制掩盖调用错误。"""
|
|
||||||
if name not in params:
|
|
||||||
return None if optional else default
|
|
||||||
raw = (params[name] or [""])[0].strip()
|
|
||||||
if raw == "":
|
|
||||||
return None if optional else default
|
|
||||||
try:
|
|
||||||
value = int(raw)
|
|
||||||
except ValueError:
|
|
||||||
raise ValueError(f"{name} 必须是整数") from None
|
|
||||||
if not low <= value <= high:
|
|
||||||
raise ValueError(f"{name} 必须在 {low}..{high}")
|
|
||||||
return value
|
|
||||||
|
|
||||||
|
|
||||||
def build_command(config: TrainConfig) -> list[str]:
|
|
||||||
if not TRAIN_SCRIPT.is_file():
|
|
||||||
raise FileNotFoundError(f"训练脚本不存在: {TRAIN_SCRIPT}")
|
|
||||||
cmd = [
|
|
||||||
sys.executable, str(TRAIN_SCRIPT),
|
|
||||||
"--problem", str(config.problem),
|
|
||||||
"--stage", str(config.stage),
|
|
||||||
"--timesteps", str(config.timesteps),
|
|
||||||
"--seed", str(config.seed),
|
|
||||||
"--envs", str(config.envs),
|
|
||||||
"--vec-env", config.vec_env,
|
|
||||||
"--device", config.device,
|
|
||||||
"--config-profile", config.config_profile,
|
|
||||||
"--reward-profile", config.reward_profile,
|
|
||||||
"--model", str(config.model_path),
|
|
||||||
"--eval-episodes", str(config.eval_episodes),
|
|
||||||
"--rollout-steps", str(config.rollout_steps),
|
|
||||||
"--batch-size", str(config.batch_size),
|
|
||||||
"--n-epochs", str(config.n_epochs),
|
|
||||||
"--max-wall-s", str(config.max_wall_s),
|
|
||||||
"--local-selector", config.local_selector,
|
|
||||||
"--checkpoint-every", str(config.checkpoint_every),
|
|
||||||
"--checkpoint-keep", str(config.checkpoint_keep),
|
|
||||||
"--recovery-every", str(config.recovery_every),
|
|
||||||
"--progress-json",
|
|
||||||
"--verbose", str(config.verbose),
|
|
||||||
]
|
|
||||||
if config.resume:
|
|
||||||
cmd.append("--resume")
|
|
||||||
if config.force:
|
|
||||||
cmd.append("--force")
|
|
||||||
if config.init:
|
|
||||||
cmd.extend(["--init", str((REPO_ROOT / config.init).resolve()
|
|
||||||
if not Path(config.init).is_absolute()
|
|
||||||
else config.init)])
|
|
||||||
if config.local_model:
|
|
||||||
cmd.extend(["--local-model", str(config.local_model)])
|
|
||||||
return cmd
|
|
||||||
|
|
||||||
|
|
||||||
def discover_recovery(run_dir: Path, model_path: Path) -> dict[str, Any]:
|
|
||||||
"""找出"现在就能继续训练"的那份产物。
|
|
||||||
|
|
||||||
优先级：
|
|
||||||
1. ``<run_dir>/resume.json``——训练脚本写的权威指针（校验目标文件真实存在）；
|
|
||||||
2. 否则按 mtime 取最新：最终模型 / 滚动恢复点 / 历史 checkpoint / 异常兜底。
|
|
||||||
|
|
||||||
只认**已完整落盘**的文件：模型的 zip 与 json 都是原子替换写的，所以不会
|
|
||||||
出现"文件在、但打不开"的半截快照。
|
|
||||||
"""
|
|
||||||
run_dir = Path(run_dir)
|
|
||||||
model_path = Path(model_path)
|
|
||||||
hint_file = run_dir / "resume.json"
|
|
||||||
hint = _read_json_file(hint_file)
|
|
||||||
|
|
||||||
candidates: list[tuple[float, str, str]] = []
|
|
||||||
|
|
||||||
def _consider(path: Path, reason: str) -> None:
|
|
||||||
try:
|
|
||||||
stat = path.stat()
|
|
||||||
except OSError:
|
|
||||||
return
|
|
||||||
if stat.st_size > 0:
|
|
||||||
candidates.append((stat.st_mtime, str(path), reason))
|
|
||||||
|
|
||||||
_consider(model_path, "final_model")
|
|
||||||
ckpt_dir = run_dir / "checkpoints"
|
|
||||||
if ckpt_dir.is_dir():
|
|
||||||
_consider(ckpt_dir / f"{model_path.stem}_ckpt_latest.zip", "recovery")
|
|
||||||
for path in ckpt_dir.glob(f"{model_path.stem}_ckpt_[0-9]*.zip"):
|
|
||||||
if path.stem.rsplit("_", 1)[-1].isdigit():
|
|
||||||
_consider(path, "checkpoint")
|
|
||||||
_consider(run_dir / f"{model_path.stem}_emergency.zip", "emergency")
|
|
||||||
|
|
||||||
state: dict[str, Any] = {
|
|
||||||
"can_resume": False, "model": None, "timesteps": None,
|
|
||||||
"reason": "none", "updated_unix_s": None, "source": "none",
|
|
||||||
"hint_file": str(hint_file), "candidates": len(candidates),
|
|
||||||
}
|
|
||||||
if hint and hint.get("model") and Path(str(hint["model"])).is_file():
|
|
||||||
state.update({
|
|
||||||
"model": str(hint["model"]),
|
|
||||||
"timesteps": _int_or_none(hint.get("timesteps")),
|
|
||||||
"reason": str(hint.get("reason") or "resume_hint"),
|
|
||||||
"updated_unix_s": _float_or_none(hint.get("created_unix_s")),
|
|
||||||
"source": "resume.json",
|
|
||||||
})
|
|
||||||
elif candidates:
|
|
||||||
mtime, path, reason = max(candidates, key=lambda item: item[0])
|
|
||||||
state.update({
|
|
||||||
"model": path, "reason": reason,
|
|
||||||
"updated_unix_s": mtime, "source": "scan",
|
|
||||||
})
|
|
||||||
if state["model"]:
|
|
||||||
state["can_resume"] = Path(str(state["model"])).is_file()
|
|
||||||
return state
|
|
||||||
|
|
||||||
|
|
||||||
def _read_json_file(path: Path) -> dict[str, Any] | None:
|
|
||||||
"""读 JSON 文件；缺失或损坏返回 None（恢复指针不是关键路径，不能因此崩）。"""
|
|
||||||
try:
|
|
||||||
payload = json.loads(Path(path).read_text(encoding="utf-8"))
|
|
||||||
except (OSError, ValueError, UnicodeDecodeError):
|
|
||||||
return None
|
|
||||||
return payload if isinstance(payload, dict) else None
|
|
||||||
|
|
||||||
|
|
||||||
def _int_or_none(value: Any) -> int | None:
|
|
||||||
try:
|
|
||||||
return int(value)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
def _float_or_none(value: Any) -> float | None:
|
|
||||||
try:
|
|
||||||
return float(value)
|
|
||||||
except (TypeError, ValueError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
|
|
||||||
class TrainingJob:
|
|
||||||
"""单个训练子进程的生命周期、日志和指标缓存。
|
|
||||||
|
|
||||||
``control_dir`` 是控制文件的落地目录；默认与模型同目录，服务端则统一放到
|
|
||||||
``output/hrl/ui/control``（已被 gitignore）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, config: TrainConfig, control_dir: Path | None = None):
|
|
||||||
self.config = config
|
|
||||||
self.job_id = f"job-{int(time.time() * 1000)}"
|
|
||||||
self.control_path = (Path(control_dir) if control_dir is not None
|
|
||||||
else config.model_path.parent) / \
|
|
||||||
f"{self.job_id}.control.json"
|
|
||||||
# 控制文件是暂停/停止的唯一可靠通道：Windows 上没有 SIGSTOP，
|
|
||||||
# 且 os.kill(SIGTERM) 会退化成硬杀，信号无法承载"优雅收尾"语义。
|
|
||||||
self.command = config.command() + ["--control-file", str(self.control_path)]
|
|
||||||
self.created_at = time.time()
|
|
||||||
self.started_at: float | None = None
|
|
||||||
self.finished_at: float | None = None
|
|
||||||
self.returncode: int | None = None
|
|
||||||
# created/running/paused/stopping/stopped/exited/failed
|
|
||||||
self.status = "created"
|
|
||||||
self.error: str | None = None
|
|
||||||
self.forced_kill = False # 停止请求超时后是否动用了强杀
|
|
||||||
self.process: subprocess.Popen | None = None
|
|
||||||
self.logs: deque[str] = deque(maxlen=MAX_LOG_LINES)
|
|
||||||
self.metrics: deque[dict[str, Any]] = deque(maxlen=MAX_METRICS)
|
|
||||||
self.monitor: list[dict[str, Any]] = []
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
self._reader: threading.Thread | None = None
|
|
||||||
self._stopper: threading.Thread | None = None
|
|
||||||
self._table_sections: list[tuple[str, dict[str, Any]]] = []
|
|
||||||
self._table_steps: float | None = None # 当前表的 total_timesteps
|
|
||||||
self._pending_table: dict[str, Any] | None = None # 未见刻度的半张表
|
|
||||||
self._last_table_steps: float | None = None # 上一张表已知刻度（截断兜底）
|
|
||||||
self._last_json_steps: float | None = None # 最近一条 progress JSON 的步数
|
|
||||||
self._last_monitor_sig: tuple[int, int] | None = None
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ lifecycle
|
|
||||||
def start(self) -> None:
|
|
||||||
with self._lock:
|
|
||||||
if self.process is not None and self.process.poll() is None:
|
|
||||||
raise RuntimeError("已有训练任务在运行")
|
|
||||||
# 先落一份"不暂停、不停止"的初始控制文件，训练脚本首步就能读到；
|
|
||||||
# 否则脚本要等到第一次轮询才区分"文件缺失"与"无指令"。
|
|
||||||
write_control(self.control_path, paused=False, stop=False)
|
|
||||||
env = os.environ.copy()
|
|
||||||
env["PYTHONUNBUFFERED"] = "1"
|
|
||||||
env["HRL_UI_JOB_ID"] = self.job_id
|
|
||||||
self.process = subprocess.Popen( # pylint: disable=consider-using-with
|
|
||||||
self.command,
|
|
||||||
cwd=str(REPO_ROOT),
|
|
||||||
stdout=subprocess.PIPE,
|
|
||||||
stderr=subprocess.STDOUT,
|
|
||||||
text=True,
|
|
||||||
bufsize=1,
|
|
||||||
env=env,
|
|
||||||
**spawn_kwargs(),
|
|
||||||
)
|
|
||||||
self.started_at = time.time()
|
|
||||||
self.status = "running"
|
|
||||||
self._reader = threading.Thread(
|
|
||||||
target=self._read_loop, name=f"{self.job_id}-reader", daemon=True)
|
|
||||||
self._reader.start()
|
|
||||||
|
|
||||||
def pause(self) -> bool:
|
|
||||||
"""请求训练进程在下一个 step 边界挂起（跨平台：控制文件）。"""
|
|
||||||
with self._lock:
|
|
||||||
process = self.process
|
|
||||||
if process is None or process.poll() is not None:
|
|
||||||
return False
|
|
||||||
if self.status not in ("running", "paused"):
|
|
||||||
return False
|
|
||||||
write_control(self.control_path, paused=True, stop=False)
|
|
||||||
self.status = "paused"
|
|
||||||
return True
|
|
||||||
|
|
||||||
def resume(self) -> bool:
|
|
||||||
"""解除挂起；训练从暂停的那个 step 继续。"""
|
|
||||||
with self._lock:
|
|
||||||
process = self.process
|
|
||||||
if process is None or process.poll() is not None:
|
|
||||||
return False
|
|
||||||
if self.status not in ("paused", "running"):
|
|
||||||
return False
|
|
||||||
write_control(self.control_path, paused=False, stop=False)
|
|
||||||
self.status = "running"
|
|
||||||
return True
|
|
||||||
|
|
||||||
def stop(self) -> bool:
|
|
||||||
"""请求优雅停止；超时后由看门狗强杀整棵进程树。
|
|
||||||
|
|
||||||
控制文件先于信号：Windows 上 SIGTERM 等价于 TerminateProcess，训练
|
|
||||||
脚本"保存模型 + 写元数据"的收尾分支不会执行；控制文件则两个平台都能
|
|
||||||
命中同一段协作式收尾。强杀放到独立线程，避免 HTTP 请求被 grace 等待卡住。
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
process = self.process
|
|
||||||
if process is None or process.poll() is not None:
|
|
||||||
return False
|
|
||||||
write_control(self.control_path, paused=False, stop=True)
|
|
||||||
self.status = "stopping"
|
|
||||||
if self._stopper is None or not self._stopper.is_alive():
|
|
||||||
self._stopper = threading.Thread(
|
|
||||||
target=self._stop_watchdog, args=(process,),
|
|
||||||
name=f"{self.job_id}-stopper", daemon=True)
|
|
||||||
self._stopper.start()
|
|
||||||
return True
|
|
||||||
|
|
||||||
def _stop_watchdog(self, process: subprocess.Popen) -> None:
|
|
||||||
"""等 grace 让协作式停止生效，超时强杀并记下恢复点。"""
|
|
||||||
forced = terminate_process_tree(process, grace_s=STOP_GRACE_S)
|
|
||||||
with self._lock:
|
|
||||||
self.forced_kill = forced
|
|
||||||
if forced and not self.error:
|
|
||||||
self.error = (f"训练进程未在 {STOP_GRACE_S:.0f}s 内响应停止请求，"
|
|
||||||
f"已强制结束进程树")
|
|
||||||
if forced:
|
|
||||||
self._write_forced_stop(process.pid)
|
|
||||||
|
|
||||||
def _write_forced_stop(self, pid: int) -> None:
|
|
||||||
"""强杀后落一份 `<模型目录>/forced_stop.json`，写明"从哪继续"。
|
|
||||||
|
|
||||||
强杀没有给训练脚本任何收尾机会，所以这里由前端补上审计与恢复指引：
|
|
||||||
磁盘上仍有原子落盘的滚动恢复点 / 历史 checkpoint，``recovery.model``
|
|
||||||
就是可直接 ``--resume`` 的目标。
|
|
||||||
"""
|
|
||||||
run_dir = self.config.model_path.parent
|
|
||||||
try:
|
|
||||||
recovery = discover_recovery(run_dir, self.config.model_path)
|
|
||||||
write_json_atomic(run_dir / "forced_stop.json", {
|
|
||||||
"event": "forced_stop",
|
|
||||||
"job_id": self.job_id,
|
|
||||||
"pid": int(pid),
|
|
||||||
"created_unix_s": time.time(),
|
|
||||||
"reason": "grace_timeout",
|
|
||||||
"grace_s": STOP_GRACE_S,
|
|
||||||
"recovery": recovery,
|
|
||||||
"note": ("训练进程未响应控制文件，已强杀整棵进程树；"
|
|
||||||
"把 recovery.model 作为 --resume 目标即可继续"
|
|
||||||
"（timesteps 为追加步数）"),
|
|
||||||
})
|
|
||||||
except OSError as exc: # 标记文件写不了也不能影响停止流程
|
|
||||||
with self._lock:
|
|
||||||
if not self.error:
|
|
||||||
self.error = f"强杀后写恢复标记失败：{exc}"
|
|
||||||
|
|
||||||
def wait_for_exit(self, timeout: float | None = None) -> bool:
|
|
||||||
"""等待停止流程收尾（优雅退出或强杀）；供前端进程退出前调用。
|
|
||||||
|
|
||||||
训练子进程在独立进程组里，前端先退出并不会带走它。若不在这里等，
|
|
||||||
前端一退出，子进程会成为孤儿——虽然控制文件仍会让它自行收尾，但
|
|
||||||
日志与退出码就没人接了。
|
|
||||||
"""
|
|
||||||
thread = self._stopper
|
|
||||||
if thread is not None:
|
|
||||||
thread.join(timeout)
|
|
||||||
process = self.process
|
|
||||||
return process is None or process.poll() is not None
|
|
||||||
|
|
||||||
def _read_loop(self) -> None:
|
|
||||||
process = self.process
|
|
||||||
assert process is not None and process.stdout is not None
|
|
||||||
try:
|
|
||||||
for raw in process.stdout:
|
|
||||||
line = raw.rstrip("\n")
|
|
||||||
self._on_line(line)
|
|
||||||
process.wait()
|
|
||||||
except Exception as exc: # pragma: no cover - 子进程读取异常
|
|
||||||
with self._lock:
|
|
||||||
self.error = f"{type(exc).__name__}: {exc}"
|
|
||||||
finally:
|
|
||||||
with self._lock:
|
|
||||||
self.finished_at = time.time()
|
|
||||||
if self.returncode is None:
|
|
||||||
self.returncode = process.poll()
|
|
||||||
if self.status == "stopping":
|
|
||||||
self.status = "stopped"
|
|
||||||
elif self.returncode == 0:
|
|
||||||
self.status = "exited"
|
|
||||||
else:
|
|
||||||
self.status = "failed"
|
|
||||||
self._flush_table()
|
|
||||||
# 最后半张表（Ctrl+C / 崩溃时没打印 total_timesteps）按末尾
|
|
||||||
# 已知刻度兜底提交，宁可标注来源也不丢最后一段曲线
|
|
||||||
self._flush_pending_with_fallback()
|
|
||||||
|
|
||||||
def _on_line(self, line: str) -> None:
|
|
||||||
with self._lock:
|
|
||||||
self.logs.append(line)
|
|
||||||
self._sync_control_events(line)
|
|
||||||
self._parse_progress_json(line)
|
|
||||||
self._parse_sb3_table(line)
|
|
||||||
self._refresh_monitor()
|
|
||||||
|
|
||||||
def _sync_control_events(self, line: str) -> None:
|
|
||||||
"""用训练脚本的回显校正前端状态。
|
|
||||||
|
|
||||||
正常情况下状态由前端自己设置；但外部直接改控制文件、或脚本比前端更早
|
|
||||||
感知到指令时，以脚本回显的 ``paused`` / ``resumed`` / ``stop_requested``
|
|
||||||
为准更真实。
|
|
||||||
"""
|
|
||||||
stripped = line.strip()
|
|
||||||
if not stripped.startswith("{") or '"event"' not in stripped:
|
|
||||||
return
|
|
||||||
try:
|
|
||||||
payload = json.loads(stripped)
|
|
||||||
except ValueError:
|
|
||||||
return
|
|
||||||
if not isinstance(payload, dict):
|
|
||||||
return
|
|
||||||
event = payload.get("event")
|
|
||||||
if event not in ("paused", "resumed", "stop_requested"):
|
|
||||||
return
|
|
||||||
with self._lock:
|
|
||||||
if event == "paused" and self.status == "running":
|
|
||||||
self.status = "paused"
|
|
||||||
elif event == "resumed" and self.status == "paused":
|
|
||||||
self.status = "running"
|
|
||||||
elif event == "stop_requested" and self.status in ("running", "paused"):
|
|
||||||
self.status = "stopping"
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ parsers
|
|
||||||
def _parse_progress_json(self, line: str) -> None:
|
|
||||||
stripped = line.strip()
|
|
||||||
if not stripped.startswith("{") or "progress" not in stripped:
|
|
||||||
return
|
|
||||||
try:
|
|
||||||
payload = json.loads(stripped)
|
|
||||||
except ValueError:
|
|
||||||
return
|
|
||||||
if not isinstance(payload, dict) or payload.get("event") != "progress":
|
|
||||||
return
|
|
||||||
point = {_normalize_metric(str(key)): value
|
|
||||||
for key, value in payload.items() if key != "event"}
|
|
||||||
steps = finite_float(point.get("timesteps"))
|
|
||||||
if steps is None:
|
|
||||||
steps = finite_float(point.get("total_timesteps"))
|
|
||||||
if steps is not None:
|
|
||||||
with self._lock:
|
|
||||||
self._last_json_steps = steps
|
|
||||||
self._append_metric(point)
|
|
||||||
|
|
||||||
def _parse_sb3_table(self, line: str) -> None:
|
|
||||||
"""兼容 SB3 verbose=1 的 ``| key | value |`` 表格。
|
|
||||||
|
|
||||||
SB3 每张表按 rollout/ → time/ → train/ → system/ 分段，``total_timesteps``
|
|
||||||
出现在 time/ 段——也就是要等同一张表的 rollout/ 行先读进来之后才拿到
|
|
||||||
横轴。因此不能在段头立刻提交：那会把还没有刻度的 rollout/ 行丢掉，或
|
|
||||||
把它们算到上一张表的刻度上。
|
|
||||||
|
|
||||||
规则：
|
|
||||||
- 每次"回到 rollout/ 段"视为新一张表开始，先把上一张已封闭的表提交；
|
|
||||||
- 表格分隔线表示本表结束，此时才允许用上一张表的刻度兜底（截断表）；
|
|
||||||
- 从未封闭过的半张表一直挂到进程结束才兜底提交，避免用旧刻度冒充。
|
|
||||||
"""
|
|
||||||
if not line.startswith("|"):
|
|
||||||
if line.startswith("-"):
|
|
||||||
self._flush_table()
|
|
||||||
return
|
|
||||||
cells = [cell.strip() for cell in line.split("|")]
|
|
||||||
if len(cells) < 4:
|
|
||||||
return
|
|
||||||
raw_key, raw_value = cells[1], cells[2]
|
|
||||||
if raw_key in _SB3_SECTIONS:
|
|
||||||
if cells[0] != "":
|
|
||||||
return
|
|
||||||
with self._lock:
|
|
||||||
if raw_key == _SB3_SECTIONS[0] and self._table_sections:
|
|
||||||
self._flush_table()
|
|
||||||
if not self._table_sections or self._table_sections[-1][0] != raw_key:
|
|
||||||
self._table_sections.append((raw_key, {}))
|
|
||||||
return
|
|
||||||
key = _normalize_metric(raw_key)
|
|
||||||
value = finite_float(raw_value)
|
|
||||||
if not key or value is None:
|
|
||||||
return
|
|
||||||
with self._lock:
|
|
||||||
if not self._table_sections:
|
|
||||||
self._table_sections.append((_SB3_SECTIONS[0], {}))
|
|
||||||
self._table_sections[-1][1][key] = value
|
|
||||||
if key == "total_timesteps":
|
|
||||||
self._table_steps = value
|
|
||||||
|
|
||||||
def _flush_table(self) -> None:
|
|
||||||
"""提交缓存中的 SB3 表。
|
|
||||||
|
|
||||||
刻度优先级：本表的 ``total_timesteps`` > 上一张表的已知刻度（仅在遇到
|
|
||||||
分隔线即"确认整表结束"时才允许兜底，见 ``_on_line``）。完全没有刻度
|
|
||||||
就丢弃——伪造横轴会在曲线上画出重复的幽灵点。与 JSON 进度行同一步数时
|
|
||||||
跳过表格版本（JSON 字段更全），表格只用于补 clip_fraction/learning_rate。
|
|
||||||
"""
|
|
||||||
with self._lock:
|
|
||||||
sections = self._table_sections
|
|
||||||
steps = self._table_steps
|
|
||||||
self._table_sections = []
|
|
||||||
self._table_steps = None
|
|
||||||
last_json = self._last_json_steps
|
|
||||||
if not sections:
|
|
||||||
return
|
|
||||||
table: dict[str, Any] = {}
|
|
||||||
for _name, items in sections:
|
|
||||||
table.update(items)
|
|
||||||
if steps is None:
|
|
||||||
steps = finite_float(table.get("total_timesteps"))
|
|
||||||
if steps is None:
|
|
||||||
# 半张表：交给调用方决定是否已确认结束（终止时按末尾兜底提交）
|
|
||||||
with self._lock:
|
|
||||||
self._pending_table = table
|
|
||||||
return
|
|
||||||
with self._lock:
|
|
||||||
self._last_table_steps = steps
|
|
||||||
if last_json is not None:
|
|
||||||
tolerance = max(1.0, 0.25 * steps / max(1.0, self.config.rollout_steps))
|
|
||||||
if abs(steps - last_json) <= tolerance:
|
|
||||||
return
|
|
||||||
if "timesteps" not in table:
|
|
||||||
table["timesteps"] = steps
|
|
||||||
self._append_metric(table)
|
|
||||||
|
|
||||||
def _flush_pending_with_fallback(self) -> None:
|
|
||||||
"""进程结束时提交最后半张表：用上一次已知刻度兜底，避免丢数据。"""
|
|
||||||
with self._lock:
|
|
||||||
pending = self._pending_table
|
|
||||||
self._pending_table = None
|
|
||||||
steps = self._last_table_steps
|
|
||||||
if not pending or steps is None:
|
|
||||||
return
|
|
||||||
if "timesteps" not in pending:
|
|
||||||
pending = {**pending, "timesteps": steps}
|
|
||||||
self._append_metric(pending)
|
|
||||||
|
|
||||||
def _append_metric(self, point: dict[str, Any]) -> None:
|
|
||||||
clean: dict[str, Any] = {}
|
|
||||||
for key, value in point.items():
|
|
||||||
number = finite_float(value)
|
|
||||||
clean[key] = number if number is not None else value
|
|
||||||
# 补齐 timesteps：progress JSON 优先，SB3 表用 total_timesteps。
|
|
||||||
if "timesteps" not in clean and "total_timesteps" in clean:
|
|
||||||
clean["timesteps"] = clean["total_timesteps"]
|
|
||||||
with self._lock:
|
|
||||||
if self.metrics and clean == self.metrics[-1]:
|
|
||||||
return
|
|
||||||
self.metrics.append(clean)
|
|
||||||
|
|
||||||
def _refresh_monitor(self) -> None:
|
|
||||||
"""训练脚本写入的 Monitor CSV：作为逐 episode 奖励的备用来源。"""
|
|
||||||
parent = self.config.model_path.parent
|
|
||||||
files = sorted(parent.glob("high_curve_*_env*.monitor.csv")) \
|
|
||||||
if parent.is_dir() else []
|
|
||||||
if not files:
|
|
||||||
return
|
|
||||||
signature = (len(files), max(path.stat().st_mtime_ns for path in files))
|
|
||||||
if signature == self._last_monitor_sig:
|
|
||||||
return
|
|
||||||
rows: list[dict[str, Any]] = []
|
|
||||||
for path in files:
|
|
||||||
try:
|
|
||||||
lines = path.read_text(encoding="utf-8", errors="replace").splitlines()
|
|
||||||
except OSError:
|
|
||||||
continue
|
|
||||||
header: list[str] | None = None
|
|
||||||
for line in lines:
|
|
||||||
if line.startswith("#") or not line.strip():
|
|
||||||
continue
|
|
||||||
if header is None:
|
|
||||||
header = [item.strip() for item in line.split(",")]
|
|
||||||
continue
|
|
||||||
values = line.split(",")
|
|
||||||
if len(values) < len(header):
|
|
||||||
continue
|
|
||||||
row = {}
|
|
||||||
valid = True
|
|
||||||
for key, raw in zip(header, values):
|
|
||||||
number = finite_float(raw)
|
|
||||||
if number is None:
|
|
||||||
valid = False
|
|
||||||
break
|
|
||||||
row[key] = number
|
|
||||||
if valid:
|
|
||||||
rows.append(row)
|
|
||||||
with self._lock:
|
|
||||||
self.monitor = rows[-MAX_METRICS:]
|
|
||||||
self._last_monitor_sig = signature
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ snapshot
|
|
||||||
def snapshot(self, *, log_tail: int = 400, metric_limit: int = 2000,
|
|
||||||
after_metric: int | None = None) -> dict[str, Any]:
|
|
||||||
with self._lock:
|
|
||||||
now = self.finished_at or time.time()
|
|
||||||
elapsed = (now - self.started_at) if self.started_at else 0.0
|
|
||||||
logs = list(self.logs)[-max(1, int(log_tail)):]
|
|
||||||
total = len(self.metrics)
|
|
||||||
if after_metric is None:
|
|
||||||
metrics = list(self.metrics)[-max(1, int(metric_limit)):]
|
|
||||||
offset = total - len(metrics)
|
|
||||||
else:
|
|
||||||
start = max(0, min(int(after_metric), total))
|
|
||||||
metrics = list(self.metrics)[start:]
|
|
||||||
offset = start
|
|
||||||
# metric_end 是本批点在序列中的绝对末端下标：服务端 deque 满 5000
|
|
||||||
# 后会丢弃最老的点，客户端只有拿到这个值才能区分"续接"与"服务端
|
|
||||||
# 已滑动窗口"，否则同一批点会被重复追加。
|
|
||||||
metric_end = offset + len(metrics)
|
|
||||||
monitor = self.monitor[-max(1, int(metric_limit)):]
|
|
||||||
config = dict(self.config.__dict__)
|
|
||||||
command = list(self.command)
|
|
||||||
model_path = str(self.config.model_path)
|
|
||||||
payload = {
|
|
||||||
"job_id": self.job_id,
|
|
||||||
"status": self.status,
|
|
||||||
"pid": process_pid(self.process),
|
|
||||||
"created_at": self.created_at,
|
|
||||||
"started_at": self.started_at,
|
|
||||||
"finished_at": self.finished_at,
|
|
||||||
"elapsed_s": round(elapsed, 3),
|
|
||||||
"returncode": self.returncode,
|
|
||||||
"error": self.error,
|
|
||||||
"forced_kill": self.forced_kill,
|
|
||||||
"command": command,
|
|
||||||
"model": model_path,
|
|
||||||
"config": config,
|
|
||||||
"log_lines": logs,
|
|
||||||
"log_total": len(self.logs),
|
|
||||||
"metrics": metrics,
|
|
||||||
"metric_offset": offset,
|
|
||||||
"metric_end": metric_end,
|
|
||||||
"metric_total": total,
|
|
||||||
"monitor": monitor,
|
|
||||||
}
|
|
||||||
payload["convergence"] = summarize_convergence(metrics, monitor)
|
|
||||||
payload["process"] = describe_process(payload)
|
|
||||||
# 恢复指针在锁外做文件扫描：即使训练被强杀，这里也能给出可续训的产物
|
|
||||||
payload["recovery"] = discover_recovery(
|
|
||||||
self.config.model_path.parent, self.config.model_path)
|
|
||||||
return payload
|
|
||||||
|
|
||||||
|
|
||||||
class TrainingUI:
|
|
||||||
"""训练前端管理器：持有唯一 TrainingJob 与 HTTP 服务器。"""
|
|
||||||
|
|
||||||
def __init__(self, ui_port: int = 8036):
|
|
||||||
self.ui_port = int(ui_port)
|
|
||||||
self.job: TrainingJob | None = None
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
try:
|
|
||||||
self.httpd = ThreadingHTTPServer(("127.0.0.1", self.ui_port), _UIHandler)
|
|
||||||
except OSError as exc:
|
|
||||||
raise OSError(
|
|
||||||
f"无法在 127.0.0.1:{self.ui_port} 启动训练前端（{exc}）；"
|
|
||||||
f"换一个 --ui-port 或先结束占用该端口的进程") from exc
|
|
||||||
self.httpd.daemon_threads = True
|
|
||||||
self.httpd.training_ui = self
|
|
||||||
self._serving = False
|
|
||||||
|
|
||||||
# ------------------------------------------------------------ API
|
|
||||||
def start(self, payload: dict[str, Any]) -> dict[str, Any]:
|
|
||||||
config = TrainConfig.from_payload(payload)
|
|
||||||
with self._lock:
|
|
||||||
if self.job is not None and self.job.status in (
|
|
||||||
"created", "running", "paused", "stopping"):
|
|
||||||
raise RuntimeError("已有训练任务在运行，请先停止")
|
|
||||||
self.job = TrainingJob(config, control_dir=CONTROL_DIR)
|
|
||||||
try:
|
|
||||||
self.job.start()
|
|
||||||
except Exception:
|
|
||||||
self.job.status = "failed"
|
|
||||||
self.job.error = "训练进程启动失败"
|
|
||||||
raise
|
|
||||||
# 与 snapshot() 同形：HTTP 层直接把返回值当响应体，不再二次拆包
|
|
||||||
return {"job": self.job.snapshot(log_tail=1)}
|
|
||||||
|
|
||||||
def pause(self) -> bool:
|
|
||||||
with self._lock:
|
|
||||||
return bool(self.job and self.job.pause())
|
|
||||||
|
|
||||||
def resume(self) -> bool:
|
|
||||||
with self._lock:
|
|
||||||
return bool(self.job and self.job.resume())
|
|
||||||
|
|
||||||
def stop(self) -> bool:
|
|
||||||
with self._lock:
|
|
||||||
return bool(self.job and self.job.stop())
|
|
||||||
|
|
||||||
def snapshot(self, *, log_tail: int, metric_limit: int,
|
|
||||||
after_metric: int | None = None) -> dict[str, Any]:
|
|
||||||
with self._lock:
|
|
||||||
if self.job is None:
|
|
||||||
return {"job": None, "message": "尚未启动训练"}
|
|
||||||
return {"job": self.job.snapshot(log_tail=log_tail,
|
|
||||||
metric_limit=metric_limit,
|
|
||||||
after_metric=after_metric)}
|
|
||||||
|
|
||||||
@property
|
|
||||||
def is_serving(self) -> bool:
|
|
||||||
return self._serving
|
|
||||||
|
|
||||||
def serve_forever(self) -> None:
|
|
||||||
self._serving = True
|
|
||||||
try:
|
|
||||||
self.httpd.serve_forever()
|
|
||||||
finally:
|
|
||||||
self._serving = False
|
|
||||||
|
|
||||||
def shutdown(self) -> None:
|
|
||||||
job = self.job
|
|
||||||
if job is not None and job.stop():
|
|
||||||
# 留出收尾落盘的时间，避免前端先退出把训练子进程留成孤儿
|
|
||||||
job.wait_for_exit(timeout=STOP_GRACE_S + 5.0)
|
|
||||||
if self._serving:
|
|
||||||
self.httpd.shutdown()
|
|
||||||
self.httpd.server_close()
|
|
||||||
|
|
||||||
|
|
||||||
class _UIHandler(BaseHTTPRequestHandler):
|
|
||||||
protocol_version = "HTTP/1.1"
|
|
||||||
server_version = "MCM26B-Training-UI/1.0"
|
|
||||||
|
|
||||||
def log_message(self, format, *args): # noqa: A002 # pylint: disable=redefined-builtin
|
|
||||||
"""覆写 stdlib 签名以吞掉访问日志；参数名沿用 ``format`` 不能改。"""
|
|
||||||
|
|
||||||
@property
|
|
||||||
def ui(self) -> TrainingUI:
|
|
||||||
return self.server.training_ui # type: ignore[attr-defined]
|
|
||||||
|
|
||||||
def do_GET(self):
|
|
||||||
parts = urlsplit(self.path)
|
|
||||||
if parts.path in STATIC_FILES:
|
|
||||||
self._serve_static(parts.path)
|
|
||||||
elif parts.path == "/api/train/schema":
|
|
||||||
self._send_json(200, config_schema())
|
|
||||||
elif parts.path == "/api/train/state":
|
|
||||||
params = parse_qs(parts.query)
|
|
||||||
if (params.get("preview") or ["0"])[0] in ("1", "true", "yes"):
|
|
||||||
# 离线预览：返回合成快照，不触碰任何训练进程
|
|
||||||
scenario = (params.get("scenario") or ["rising"])[0]
|
|
||||||
if scenario not in DEMO_SCENARIOS:
|
|
||||||
self._send_json(400, {"error": f"未知预览场景 {scenario}"})
|
|
||||||
return
|
|
||||||
self._send_json(200, {"job": demo_snapshot(log_tail=400,
|
|
||||||
scenario=scenario)})
|
|
||||||
return
|
|
||||||
try:
|
|
||||||
log_tail = _int_query(params, "log_tail", 400, 1, MAX_LOG_LINES)
|
|
||||||
metric_limit = _int_query(params, "metric_limit", 2000, 1,
|
|
||||||
MAX_METRICS)
|
|
||||||
after = _int_query(params, "after_metric", None, 0,
|
|
||||||
MAX_METRICS * 2, optional=True)
|
|
||||||
except ValueError as exc:
|
|
||||||
self._send_json(400, {"error": str(exc)})
|
|
||||||
return
|
|
||||||
self._send_json(200, self.ui.snapshot(
|
|
||||||
log_tail=log_tail, metric_limit=metric_limit,
|
|
||||||
after_metric=after))
|
|
||||||
else:
|
|
||||||
self._send_json(404, {"error": f"未知路径 {parts.path}"})
|
|
||||||
|
|
||||||
def do_POST(self):
|
|
||||||
parts = urlsplit(self.path)
|
|
||||||
payload = self._read_json()
|
|
||||||
if payload is None:
|
|
||||||
self._send_json(400, {"error": "请求体必须是合法 JSON"})
|
|
||||||
return
|
|
||||||
if not isinstance(payload, dict):
|
|
||||||
self._send_json(400, {"error": "请求体必须是 JSON 对象"})
|
|
||||||
return
|
|
||||||
try:
|
|
||||||
if parts.path == "/api/train/start":
|
|
||||||
self._send_json(200, self.ui.start(payload))
|
|
||||||
elif parts.path == "/api/train/pause":
|
|
||||||
self._send_json(200, {"paused": self.ui.pause()})
|
|
||||||
elif parts.path == "/api/train/resume":
|
|
||||||
self._send_json(200, {"resumed": self.ui.resume()})
|
|
||||||
elif parts.path == "/api/train/stop":
|
|
||||||
self._send_json(200, {"stopped": self.ui.stop()})
|
|
||||||
else:
|
|
||||||
self._send_json(404, {"error": f"未知路径 {parts.path}"})
|
|
||||||
except (ValueError, RuntimeError, FileNotFoundError,
|
|
||||||
PermissionError) as exc:
|
|
||||||
self._send_json(409, {"error": str(exc)})
|
|
||||||
except OSError as exc:
|
|
||||||
self._send_json(409, {"error": f"操作系统错误：{exc}"})
|
|
||||||
except Exception as exc: # pragma: no cover - 兜底，避免连接被静默中断
|
|
||||||
self._send_json(500, {"error": f"服务端内部错误：{type(exc).__name__}: {exc}"})
|
|
||||||
|
|
||||||
def _serve_static(self, path: str) -> None:
|
|
||||||
filename, content_type = STATIC_FILES[path]
|
|
||||||
data = (STATIC_DIR / filename).read_bytes()
|
|
||||||
self.send_response(200)
|
|
||||||
self.send_header("Content-Type", content_type)
|
|
||||||
self.send_header("Content-Length", str(len(data)))
|
|
||||||
self.send_header("Cache-Control", "no-store")
|
|
||||||
self.end_headers()
|
|
||||||
self.wfile.write(data)
|
|
||||||
|
|
||||||
def _read_json(self) -> dict[str, Any] | None:
|
|
||||||
try:
|
|
||||||
length = int(self.headers.get("Content-Length") or 0)
|
|
||||||
except ValueError:
|
|
||||||
return None
|
|
||||||
if length <= 0 or length > 1_000_000:
|
|
||||||
return {}
|
|
||||||
data = self.rfile.read(length)
|
|
||||||
try:
|
|
||||||
return json.loads(data.decode("utf-8"))
|
|
||||||
except (ValueError, UnicodeDecodeError):
|
|
||||||
return None
|
|
||||||
|
|
||||||
def _send_json(self, status: int, payload: dict[str, Any]) -> None:
|
|
||||||
data = json.dumps(payload, ensure_ascii=False, allow_nan=False).encode("utf-8")
|
|
||||||
self.send_response(status)
|
|
||||||
self.send_header("Content-Type", "application/json; charset=utf-8")
|
|
||||||
self.send_header("Content-Length", str(len(data)))
|
|
||||||
self.send_header("Cache-Control", "no-store")
|
|
||||||
self.end_headers()
|
|
||||||
self.wfile.write(data)
|
|
||||||
|
|
||||||
|
|
||||||
def run_training_ui(argv=None) -> int:
|
|
||||||
import argparse
|
|
||||||
|
|
||||||
parser = argparse.ArgumentParser(description="HRL 模型训练前端（本地回环）")
|
|
||||||
parser.add_argument("--ui-port", type=int, default=8036)
|
|
||||||
parser.add_argument("--no-browser", action="store_true")
|
|
||||||
parser.add_argument("--print-schema", action="store_true",
|
|
||||||
help="打印配置契约（字段/分组/预设/指标）后退出")
|
|
||||||
parser.add_argument("--print-preview", action="store_true",
|
|
||||||
help="打印离线示例快照 JSON 后退出")
|
|
||||||
args = parser.parse_args(argv)
|
|
||||||
if args.print_schema:
|
|
||||||
print(json.dumps(config_schema(), ensure_ascii=False, indent=2))
|
|
||||||
return 0
|
|
||||||
if args.print_preview:
|
|
||||||
print(json.dumps({"job": demo_snapshot(log_tail=200)},
|
|
||||||
ensure_ascii=False, indent=2))
|
|
||||||
return 0
|
|
||||||
try:
|
|
||||||
ui = TrainingUI(ui_port=args.ui_port)
|
|
||||||
except OSError as exc:
|
|
||||||
print(f"[训练前端] {exc}", file=sys.stderr)
|
|
||||||
return 2
|
|
||||||
url = f"http://127.0.0.1:{args.ui_port}/"
|
|
||||||
# flush=True：界面常用 `> log 2>&1 &` 方式后台启动，不刷新会看不到启动横幅
|
|
||||||
print(f"训练前端已启动：{url}", flush=True)
|
|
||||||
print(f" 配置契约：{url}api/train/schema", flush=True)
|
|
||||||
print(f" 状态快照：{url}api/train/state", flush=True)
|
|
||||||
print(f" 离线预览：{url}api/train/state?preview=1", flush=True)
|
|
||||||
print(" 界面只监听 127.0.0.1；Ctrl+C 退出并停止训练子进程。", flush=True)
|
|
||||||
if not args.no_browser:
|
|
||||||
try:
|
|
||||||
import webbrowser
|
|
||||||
webbrowser.open(url)
|
|
||||||
except Exception:
|
|
||||||
pass
|
|
||||||
try:
|
|
||||||
ui.serve_forever()
|
|
||||||
except KeyboardInterrupt:
|
|
||||||
pass
|
|
||||||
finally:
|
|
||||||
ui.shutdown()
|
|
||||||
return 0
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__": # pragma: no cover
|
|
||||||
raise SystemExit(run_training_ui())
|
|
||||||
@@ -1,841 +0,0 @@
|
|||||||
/* 训练前端主逻辑：纯原生 JS，无外部依赖。
|
|
||||||
职责分工：
|
|
||||||
- 表单由 /api/train/schema 的字段元数据生成，不在前端复制校验规则；
|
|
||||||
- 训练数值全部取自 /api/train/state 快照（服务端已算好进度、吞吐、
|
|
||||||
收敛判读），前端只做格式化与渲染，避免两套算法对不上；
|
|
||||||
- 指标增量拉取（after_metric 游标），日志按尾部行数拉取。 */
|
|
||||||
"use strict";
|
|
||||||
|
|
||||||
const $ = (id) => document.getElementById(id);
|
|
||||||
const REFRESH_MS = 1500;
|
|
||||||
// 与服务端 MAX_METRICS 对齐：本地缓存上限，避免长时间训练把内存吃满
|
|
||||||
const METRIC_LIMIT = 5000;
|
|
||||||
|
|
||||||
const state = {
|
|
||||||
schema: null,
|
|
||||||
preview: false, // 离线预览：不再启动进程，只渲染示例快照
|
|
||||||
polling: false,
|
|
||||||
saving: false, // 配置表单是否处于"训练中"锁定态
|
|
||||||
metrics: [], // 本地累积的进度点（按 timesteps 递增）
|
|
||||||
metricCache: [], // metrics 的同一个数组（显式持有，避免 slice/concat 时丢引用）
|
|
||||||
metricNext: 0, // 本地已吃到的下一个下标（= 服务端下次应返回的 offset）
|
|
||||||
metricTotal: 0, // 服务端累计进度点总数（增量拉取的游标）
|
|
||||||
chartKey: "ep_rew_mean",
|
|
||||||
jobKey: null, // job_id 变化时清空本地缓存
|
|
||||||
lastLogSignature: "",
|
|
||||||
lastStatus: null, // 最近一次快照的状态：决定暂停按钮是"暂停"还是"继续"
|
|
||||||
lastJob: null, // 最近一次快照：恢复按钮需要读 job.recovery
|
|
||||||
};
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 工具
|
|
||||||
function escapeHtml(value) {
|
|
||||||
return String(value == null ? "" : value)
|
|
||||||
.replace(/[&<>"']/g, (c) => ({ "&": "&", "<": "<", ">": ">",
|
|
||||||
'"': """, "'": "'" }[c]));
|
|
||||||
}
|
|
||||||
function fmtNumber(value, digits) {
|
|
||||||
if (value === null || value === undefined || value === "" || Number.isNaN(Number(value))) return "—";
|
|
||||||
const num = Number(value);
|
|
||||||
const abs = Math.abs(num);
|
|
||||||
if (abs !== 0 && (abs < 1e-3 || abs >= 1e5)) return num.toExponential(2);
|
|
||||||
const d = digits === undefined ? (abs >= 100 ? 1 : abs >= 1 ? 2 : 4) : digits;
|
|
||||||
return num.toFixed(d);
|
|
||||||
}
|
|
||||||
function fmtInt(value) {
|
|
||||||
if (value === null || value === undefined || Number.isNaN(Number(value))) return "—";
|
|
||||||
return Math.round(Number(value)).toLocaleString("en-US");
|
|
||||||
}
|
|
||||||
function fmtDuration(seconds) {
|
|
||||||
if (seconds === null || seconds === undefined || !isFinite(seconds) || seconds < 0) return "—";
|
|
||||||
const s = Math.round(seconds);
|
|
||||||
const h = Math.floor(s / 3600), m = Math.floor((s % 3600) / 60), sec = s % 60;
|
|
||||||
if (h > 0) return `${h}h${String(m).padStart(2, "0")}m`;
|
|
||||||
if (m > 0) return `${m}m${String(sec).padStart(2, "0")}s`;
|
|
||||||
return `${sec}s`;
|
|
||||||
}
|
|
||||||
function fmtClock(epochSeconds) {
|
|
||||||
if (!epochSeconds) return "—";
|
|
||||||
const d = new Date(epochSeconds * 1000);
|
|
||||||
return `${String(d.getHours()).padStart(2, "0")}:${String(d.getMinutes()).padStart(2, "0")}:${String(d.getSeconds()).padStart(2, "0")}`;
|
|
||||||
}
|
|
||||||
function chip(id, label, value, tone) {
|
|
||||||
const el = $(id);
|
|
||||||
if (!el) return;
|
|
||||||
el.className = `chip ${tone || "gray"}`;
|
|
||||||
el.innerHTML = `${label}<b>${escapeHtml(value)}</b>`;
|
|
||||||
}
|
|
||||||
function setText(id, text) { const el = $(id); if (el) el.textContent = text; }
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 启动
|
|
||||||
async function boot() {
|
|
||||||
try {
|
|
||||||
const res = await fetch("/api/train/schema");
|
|
||||||
if (!res.ok) throw new Error(`HTTP ${res.status}`);
|
|
||||||
state.schema = await res.json();
|
|
||||||
} catch (err) {
|
|
||||||
state.schema = { fields: [], groups: [], presets: [], metrics: [] };
|
|
||||||
showBanner("bad", `无法读取配置契约（${err.message}）：请确认已通过 python -m mcm26b.training_ui 启动前端。`);
|
|
||||||
}
|
|
||||||
buildForm();
|
|
||||||
await poll();
|
|
||||||
bindChartInteractions();
|
|
||||||
setInterval(poll, REFRESH_MS);
|
|
||||||
}
|
|
||||||
|
|
||||||
function showBanner(tone, html) {
|
|
||||||
const el = $("banner");
|
|
||||||
if (!html) { el.classList.add("hidden"); el.innerHTML = ""; return; }
|
|
||||||
el.className = `banner ${tone || ""}`;
|
|
||||||
el.innerHTML = html;
|
|
||||||
el.classList.remove("hidden");
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 表单
|
|
||||||
function buildForm() {
|
|
||||||
const presets = $("presets");
|
|
||||||
presets.innerHTML = "";
|
|
||||||
for (const preset of state.schema.presets || []) {
|
|
||||||
const btn = document.createElement("button");
|
|
||||||
btn.type = "button";
|
|
||||||
btn.className = "preset";
|
|
||||||
btn.innerHTML = `${escapeHtml(preset.label)}<span class="note">${escapeHtml(preset.note || "")}</span>`;
|
|
||||||
btn.onclick = () => { applyValues(preset.payload); $("form-error").textContent = ""; };
|
|
||||||
presets.appendChild(btn);
|
|
||||||
}
|
|
||||||
|
|
||||||
const wrap = $("form-groups");
|
|
||||||
wrap.innerHTML = "";
|
|
||||||
for (const group of state.schema.groups || []) {
|
|
||||||
const box = document.createElement("fieldset");
|
|
||||||
box.className = "group";
|
|
||||||
const fields = (state.schema.fields || []).filter((f) => f.group === group.key);
|
|
||||||
box.innerHTML = `<legend>${escapeHtml(group.label)}</legend>` +
|
|
||||||
`<p class="ghint">${escapeHtml(group.hint || "")}</p>`;
|
|
||||||
for (const field of fields) box.appendChild(buildField(field));
|
|
||||||
wrap.appendChild(box);
|
|
||||||
}
|
|
||||||
applyValues({});
|
|
||||||
}
|
|
||||||
|
|
||||||
function buildField(field) {
|
|
||||||
const row = document.createElement("div");
|
|
||||||
row.className = "field";
|
|
||||||
row.id = `field-${field.key}`;
|
|
||||||
const label = document.createElement("label");
|
|
||||||
label.className = "flabel";
|
|
||||||
label.innerHTML = `<span>${escapeHtml(field.label)}</span>` +
|
|
||||||
(field.unit ? `<span class="unit">${escapeHtml(field.unit)}</span>` : "");
|
|
||||||
let control;
|
|
||||||
if (field.kind === "select") {
|
|
||||||
control = document.createElement("select");
|
|
||||||
for (const choice of field.choices || []) {
|
|
||||||
const option = document.createElement("option");
|
|
||||||
option.value = String(choice.value);
|
|
||||||
option.textContent = choice.label;
|
|
||||||
control.appendChild(option);
|
|
||||||
}
|
|
||||||
} else if (field.kind === "bool") {
|
|
||||||
control = document.createElement("input");
|
|
||||||
control.type = "checkbox";
|
|
||||||
row.classList.add("check");
|
|
||||||
} else {
|
|
||||||
control = document.createElement("input");
|
|
||||||
control.type = field.kind === "text" ? "text" : "number";
|
|
||||||
if (field.kind === "number") control.step = field.step || "any";
|
|
||||||
if (field.min !== undefined) control.min = field.min;
|
|
||||||
if (field.max !== undefined) control.max = field.max;
|
|
||||||
}
|
|
||||||
control.id = `cfg-${field.key}`;
|
|
||||||
control.dataset.key = field.key;
|
|
||||||
control.addEventListener("input", () => { row.classList.add("dirty"); row.classList.remove("invalid"); });
|
|
||||||
row.appendChild(label);
|
|
||||||
row.appendChild(control);
|
|
||||||
if (field.help) {
|
|
||||||
const help = document.createElement("span");
|
|
||||||
help.className = "fhelp";
|
|
||||||
help.textContent = field.help;
|
|
||||||
row.appendChild(help);
|
|
||||||
}
|
|
||||||
return row;
|
|
||||||
}
|
|
||||||
|
|
||||||
function setControl(field, value) {
|
|
||||||
const control = $(`cfg-${field.key}`);
|
|
||||||
if (!control) return;
|
|
||||||
if (field.kind === "bool") { control.checked = Boolean(value); return; }
|
|
||||||
if (value === null || value === undefined) { control.value = ""; return; }
|
|
||||||
control.value = String(value);
|
|
||||||
}
|
|
||||||
|
|
||||||
function applyValues(values) {
|
|
||||||
for (const field of state.schema.fields || []) {
|
|
||||||
const has = Object.prototype.hasOwnProperty.call(values, field.key);
|
|
||||||
setControl(field, has ? values[field.key] : field.default);
|
|
||||||
const row = $(`field-${field.key}`);
|
|
||||||
if (row) row.classList.remove("dirty", "invalid");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function collectValues() {
|
|
||||||
const out = {};
|
|
||||||
for (const field of state.schema.fields || []) {
|
|
||||||
const control = $(`cfg-${field.key}`);
|
|
||||||
if (!control) continue;
|
|
||||||
if (field.kind === "bool") { out[field.key] = control.checked; continue; }
|
|
||||||
const raw = String(control.value || "").trim();
|
|
||||||
if (raw === "") { out[field.key] = null; continue; }
|
|
||||||
if (field.kind === "integer") out[field.key] = Number.parseInt(raw, 10);
|
|
||||||
else if (field.kind === "number") out[field.key] = Number.parseFloat(raw);
|
|
||||||
else if (field.kind === "select") {
|
|
||||||
const choices = field.choices || [];
|
|
||||||
const match = choices.find((c) => String(c.value) === raw);
|
|
||||||
out[field.key] = match ? match.value : raw;
|
|
||||||
} else out[field.key] = raw;
|
|
||||||
}
|
|
||||||
return out;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* 与后端 validate() 对齐的前置提示：只做提示，不阻止提交后端的权威判定 */
|
|
||||||
function validateLocally(values) {
|
|
||||||
const problems = [];
|
|
||||||
for (const field of state.schema.fields || []) {
|
|
||||||
const value = values[field.key];
|
|
||||||
if (value === null || value === undefined) continue;
|
|
||||||
if (field.kind === "integer" || field.kind === "number") {
|
|
||||||
if (!isFinite(value)) { problems.push(`${field.label} 不是有效数字`); markInvalid(field.key, true); continue; }
|
|
||||||
if (field.min !== undefined && value < field.min) { problems.push(`${field.label} 不能小于 ${field.min}`); markInvalid(field.key, true); continue; }
|
|
||||||
if (field.max !== undefined && value > field.max) { problems.push(`${field.label} 不能大于 ${field.max}`); markInvalid(field.key, true); continue; }
|
|
||||||
markInvalid(field.key, false);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (values.rollout_steps && values.timesteps && values.rollout_steps > values.timesteps) {
|
|
||||||
problems.push("采样步数 n_steps 不能大于总步数 timesteps");
|
|
||||||
markInvalid("rollout_steps", true);
|
|
||||||
}
|
|
||||||
if (values.batch_size && values.rollout_steps && values.envs
|
|
||||||
&& values.batch_size > values.rollout_steps * values.envs) {
|
|
||||||
problems.push("批大小 batch_size 大于 n_steps × envs，PPO 每轮数据不足以分批");
|
|
||||||
markInvalid("batch_size", true);
|
|
||||||
}
|
|
||||||
return problems;
|
|
||||||
}
|
|
||||||
|
|
||||||
function markInvalid(key, bad) {
|
|
||||||
const row = $(`field-${key}`);
|
|
||||||
if (row) row.classList.toggle("invalid", Boolean(bad));
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 轮询
|
|
||||||
async function poll() {
|
|
||||||
if (state.polling) return;
|
|
||||||
if (state.preview) return;
|
|
||||||
state.polling = true;
|
|
||||||
try {
|
|
||||||
const query = new URLSearchParams({
|
|
||||||
log_tail: $("log-tail").value || "400",
|
|
||||||
metric_limit: String(METRIC_LIMIT),
|
|
||||||
});
|
|
||||||
if (state.metricTotal > 0) query.set("after_metric", String(state.metricTotal));
|
|
||||||
const res = await fetch(`/api/train/state?${query.toString()}`);
|
|
||||||
const payload = await res.json();
|
|
||||||
if (!res.ok) throw new Error(payload.error || `HTTP ${res.status}`);
|
|
||||||
chip("chip-conn", "连接", "已连接", "ok");
|
|
||||||
render(payload.job);
|
|
||||||
} catch (err) {
|
|
||||||
chip("chip-conn", "连接", "断开", "bad");
|
|
||||||
} finally {
|
|
||||||
state.polling = false;
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
/* 合并一次快照返回的进度点。服务端用 metric_offset 给出这批点的绝对起点，
|
|
||||||
客户端只需与"本地已吃到的下一个下标"（metricNext）对账：
|
|
||||||
- offset === metricNext：正常续接（增量拉取），concat；
|
|
||||||
- offset > metricNext：本地缓存与这批点之间缺了若干点（例如页面刷新后
|
|
||||||
服务端窗口已前移），保留本地旧点会让曲线出现空洞，因此整体替换；
|
|
||||||
- offset < metricNext：本地缓存比服务端更新（服务端重启/计数回退），
|
|
||||||
忽略这批过期点，不动本地缓存。
|
|
||||||
判据是"绝对下标"而不是长度：游标是服务端总数，本地可能被裁剪得更短，
|
|
||||||
拿长度比较会把同一批点重复追加（本函数注释即为此 bug 的回归说明）。
|
|
||||||
下标必须显式转 Number：JSON 里若出现字符串形式的数字，JS 的 >= 会退化成
|
|
||||||
字典序比较（"14" >= "014" 为 false），静默走错分支。 */
|
|
||||||
function absorbMetrics(job) {
|
|
||||||
const incoming = Array.isArray(job.metrics) ? job.metrics : [];
|
|
||||||
const rawOffset = Number(job.metric_offset);
|
|
||||||
const offset = Number.isFinite(rawOffset) ? rawOffset : null;
|
|
||||||
const cache = state.metricCache || [];
|
|
||||||
if (incoming.length && offset !== null) {
|
|
||||||
if (!cache.length || offset > state.metricNext) {
|
|
||||||
// 首批，或与本地之间有缺口（服务端窗口前移）：整体替换，避免曲线断档
|
|
||||||
state.metricCache = incoming.slice();
|
|
||||||
state.metricNext = offset + incoming.length;
|
|
||||||
} else if (offset === state.metricNext) {
|
|
||||||
state.metricCache = cache.concat(incoming);
|
|
||||||
state.metricNext = offset + incoming.length;
|
|
||||||
} else {
|
|
||||||
return; // 过期批次：本地已有更新的点，直接忽略
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (state.metricCache.length > METRIC_LIMIT) {
|
|
||||||
state.metricCache = state.metricCache.slice(state.metricCache.length - METRIC_LIMIT);
|
|
||||||
}
|
|
||||||
state.metrics = state.metricCache;
|
|
||||||
const total = Number(job.metric_total);
|
|
||||||
if (Number.isFinite(total)) state.metricTotal = total;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 渲染
|
|
||||||
function render(job) {
|
|
||||||
if (!job) {
|
|
||||||
state.lastJob = null;
|
|
||||||
renderIdle();
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
state.lastJob = job;
|
|
||||||
if (job.job_id !== state.jobKey) {
|
|
||||||
state.jobKey = job.job_id;
|
|
||||||
state.metrics = [];
|
|
||||||
state.metricTotal = 0;
|
|
||||||
state.lastLogSignature = "";
|
|
||||||
}
|
|
||||||
absorbMetrics(job);
|
|
||||||
|
|
||||||
renderProcess(job);
|
|
||||||
renderConvergence(job);
|
|
||||||
renderCharts();
|
|
||||||
renderLog(job);
|
|
||||||
renderEpisodes(job);
|
|
||||||
renderBanner(job);
|
|
||||||
renderRecovery(job);
|
|
||||||
renderJobLine(job);
|
|
||||||
}
|
|
||||||
|
|
||||||
function renderIdle() {
|
|
||||||
chip("chip-process", "进程", "未启动", "gray");
|
|
||||||
chip("chip-progress", "进度", "—", "gray");
|
|
||||||
chip("chip-elapsed", "墙钟", "—", "gray");
|
|
||||||
chip("chip-device", "设备", "—", "gray");
|
|
||||||
setText("process-hint", "启动任务后显示子进程与步进进度");
|
|
||||||
setText("steps-done", "0"); setText("steps-total", "0");
|
|
||||||
setText("progress-percent", "0.0%");
|
|
||||||
$("bar-fill").style.width = "0%";
|
|
||||||
setText("eta", "—"); setText("eta-clock", "—"); setText("metric-total", "0");
|
|
||||||
for (const id of ["stat-pid", "stat-fps", "stat-elapsed", "stat-rc"]) setText(id, "—");
|
|
||||||
setText("stat-logs", "0");
|
|
||||||
$("timeline").innerHTML = "";
|
|
||||||
setText("command-preview", "");
|
|
||||||
state.lastStatus = null;
|
|
||||||
const pauseBtn = $("btn-pause");
|
|
||||||
pauseBtn.disabled = true;
|
|
||||||
pauseBtn.textContent = "暂停训练";
|
|
||||||
chip("chip-converge", "判读", "等待数据", "gray");
|
|
||||||
$("converge-notes").innerHTML = "";
|
|
||||||
$("metric-grid").innerHTML = '<p class="hint">尚未启动训练，暂无参数。</p>';
|
|
||||||
setFormLocked(false);
|
|
||||||
}
|
|
||||||
|
|
||||||
function renderJobLine(job) {
|
|
||||||
const cfg = job.config || {};
|
|
||||||
const parts = [
|
|
||||||
`任务 ${job.job_id}`, `问题 ${cfg.problem}`, `阶段 ${cfg.stage}`,
|
|
||||||
`口径 ${cfg.reward_profile}/${cfg.config_profile}`,
|
|
||||||
`seed ${cfg.seed}`, `envs ${cfg.envs}·${cfg.vec_env}`,
|
|
||||||
];
|
|
||||||
setText("job-line", parts.join(" · "));
|
|
||||||
chip("chip-device", "设备", `${cfg.device || "?"}`, "cyan");
|
|
||||||
}
|
|
||||||
|
|
||||||
function renderProcess(job) {
|
|
||||||
const process = job.process || {};
|
|
||||||
const tone = process.tone || "gray";
|
|
||||||
chip("chip-process", "进程", process.label || job.status, tone);
|
|
||||||
chip("chip-elapsed", "墙钟", fmtDuration(process.elapsed_s), tone === "gray" ? "gray" : "cyan");
|
|
||||||
|
|
||||||
const percent = process.percent;
|
|
||||||
setText("chip-progress-value", "");
|
|
||||||
chip("chip-progress", "进度",
|
|
||||||
percent === null || percent === undefined ? "—" : `${percent.toFixed(1)}%`,
|
|
||||||
percent !== null && percent >= 100 ? "ok" : "cyan");
|
|
||||||
setText("steps-done", fmtInt(process.trained_steps ?? process.done_steps));
|
|
||||||
setText("steps-total", fmtInt(process.total_steps));
|
|
||||||
setText("progress-percent", percent === null || percent === undefined ? "—" : `${percent.toFixed(2)}%`);
|
|
||||||
const fill = $("bar-fill");
|
|
||||||
fill.style.width = `${percent === null || percent === undefined ? 0 : percent}%`;
|
|
||||||
fill.className = "bar-fill" + (tone === "bad" ? " bad" : tone === "warn" ? " warn" : tone === "ok" && percent >= 100 ? " ok" : "");
|
|
||||||
setText("process-hint", process.hint || "");
|
|
||||||
setText("eta", fmtDuration(process.remaining_s));
|
|
||||||
setText("eta-clock", fmtClock(process.eta_epoch_s));
|
|
||||||
setText("metric-total", fmtInt(process.metric_total));
|
|
||||||
|
|
||||||
setText("stat-pid", process.terminal ? "已退出" : (job.pid ? `PID ${job.pid}` : "—"));
|
|
||||||
setText("stat-pgid", process.terminal ? "已回收" : "独立会话");
|
|
||||||
const fps = state.metrics.length ? lastFinite(state.metrics, "fps") : null;
|
|
||||||
setText("stat-fps", fps === null ? "—" : fmtNumber(fps, 1));
|
|
||||||
setText("stat-elapsed", fmtDuration(process.elapsed_s));
|
|
||||||
const rc = $("stat-rc");
|
|
||||||
rc.textContent = job.returncode === null || job.returncode === undefined ? "—" : String(job.returncode);
|
|
||||||
rc.className = "v" + (job.returncode === 0 ? " ok" : job.returncode ? " bad" : "");
|
|
||||||
setText("stat-logs", `${fmtInt(process.log_total)} / ${fmtInt(process.log_capacity)}`);
|
|
||||||
|
|
||||||
renderTimeline(process);
|
|
||||||
setText("command-preview", job.command ? `$ ${job.command.join(" ")}` : "");
|
|
||||||
state.lastStatus = job.status;
|
|
||||||
setFormLocked(job.status === "running" || job.status === "created"
|
|
||||||
|| job.status === "paused" || job.status === "stopping");
|
|
||||||
$("btn-stop").disabled = !(job.status === "running" || job.status === "created"
|
|
||||||
|| job.status === "paused");
|
|
||||||
const pauseBtn = $("btn-pause");
|
|
||||||
pauseBtn.disabled = !(job.status === "running" || job.status === "paused");
|
|
||||||
pauseBtn.textContent = job.status === "paused" ? "继续训练" : "暂停训练";
|
|
||||||
}
|
|
||||||
|
|
||||||
function renderTimeline(process) {
|
|
||||||
const wrap = $("timeline");
|
|
||||||
const nodes = process.timeline || [];
|
|
||||||
if (!nodes.length) { wrap.innerHTML = ""; return; }
|
|
||||||
const active = nodes.findIndex((n) => n.state === "active");
|
|
||||||
if (wrap.dataset.signature !== nodes.map((n) => n.label).join("|")) {
|
|
||||||
wrap.innerHTML = "";
|
|
||||||
nodes.forEach((node, index) => {
|
|
||||||
if (index > 0) {
|
|
||||||
const link = document.createElement("span");
|
|
||||||
link.className = "link";
|
|
||||||
link.dataset.link = String(index);
|
|
||||||
wrap.appendChild(link);
|
|
||||||
}
|
|
||||||
const el = document.createElement("span");
|
|
||||||
el.className = "node";
|
|
||||||
el.dataset.node = node.key;
|
|
||||||
el.innerHTML = `<span class="dot"></span><span class="txt">${escapeHtml(node.label)}</span>`;
|
|
||||||
wrap.appendChild(el);
|
|
||||||
});
|
|
||||||
wrap.dataset.signature = nodes.map((n) => n.label).join("|");
|
|
||||||
}
|
|
||||||
nodes.forEach((node, index) => {
|
|
||||||
const el = wrap.querySelector(`[data-node="${node.key}"]`);
|
|
||||||
if (!el) return;
|
|
||||||
let cls = "node";
|
|
||||||
if (node.state === "done") cls += " done";
|
|
||||||
else if (node.state === "active") cls += " active";
|
|
||||||
if (node.state === "active" && process.tone === "bad") cls += " failed";
|
|
||||||
el.className = cls;
|
|
||||||
const link = wrap.querySelector(`[data-link="${index}"]`);
|
|
||||||
if (link) link.className = "link" + (index <= active ? " done" : "");
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
function setFormLocked(locked) {
|
|
||||||
state.saving = locked;
|
|
||||||
for (const field of state.schema.fields || []) {
|
|
||||||
const control = $(`cfg-${field.key}`);
|
|
||||||
if (control) control.disabled = locked;
|
|
||||||
}
|
|
||||||
$("btn-start").disabled = locked;
|
|
||||||
$("btn-start").textContent = locked ? "训练中…" : "开始训练";
|
|
||||||
}
|
|
||||||
|
|
||||||
function renderBanner(job) {
|
|
||||||
const process = job.process || {};
|
|
||||||
if (state.preview) {
|
|
||||||
showBanner("warn", "离线预览模式：正在渲染示例快照，<b>不会</b>启动训练进程。点「刷新」可返回实时轮询。");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
if (job.status === "failed") {
|
|
||||||
showBanner("bad", `训练异常结束（退出码 <b>${job.returncode === null ? "?" : job.returncode}</b>）：${escapeHtml(job.error || "见日志尾部")}。`);
|
|
||||||
} else if (job.status === "paused") {
|
|
||||||
showBanner("warn", "训练已暂停：进程与显存仍在，但停在 step 边界不再推进。点「继续训练」从暂停处恢复，或点「停止训练」保存当前模型后退出。");
|
|
||||||
} else if (job.status === "stopped") {
|
|
||||||
showBanner("warn", job.forced_kill
|
|
||||||
? "停止请求超时，已强制结束进程树；最后一段模型可能未保存，请检查 checkpoint 目录。"
|
|
||||||
: "已按请求停止训练（控制文件协作式停止），脚本已在 step 边界保存当前模型与元数据。");
|
|
||||||
} else if (job.status === "exited") {
|
|
||||||
showBanner("ok", "训练与评估流程正常结束，模型与元数据已写入输出目录。");
|
|
||||||
} else if (process.remaining_s !== null && process.remaining_s !== undefined
|
|
||||||
&& process.eta_epoch_s === null && job.status === "running") {
|
|
||||||
showBanner("warn", `尚未产出可估算吞吐的进度点，剩余时间按墙钟上限回退估计：<b>${fmtDuration(process.remaining_s)}</b>。`);
|
|
||||||
} else {
|
|
||||||
showBanner("", "");
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 恢复
|
|
||||||
function renderRecovery(job) {
|
|
||||||
const recovery = job.recovery || {};
|
|
||||||
const box = $("recovery-box");
|
|
||||||
if (!box) return;
|
|
||||||
if (!recovery.model || recovery.reason === "none") {
|
|
||||||
box.classList.add("hidden");
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
box.classList.remove("hidden");
|
|
||||||
const steps = recovery.timesteps === null || recovery.timesteps === undefined
|
|
||||||
? "步数未知" : `${fmtInt(recovery.timesteps)} 步`;
|
|
||||||
const when = recovery.updated_unix_s ? fmtClock(recovery.updated_unix_s) : "—";
|
|
||||||
setText("recovery-text",
|
|
||||||
`可恢复产物：${steps} · 来源 ${recovery.reason || "?"} · 更新 ${when} · ${recovery.model}`);
|
|
||||||
const button = $("btn-recover");
|
|
||||||
// 离线预览里的恢复信息是合成的，不允许真的去续训
|
|
||||||
button.disabled = !recovery.can_resume || Boolean(state.preview);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 收敛
|
|
||||||
function renderConvergence(job) {
|
|
||||||
const convergence = job.convergence || {};
|
|
||||||
chip("chip-converge", "判读", convergence.status || "等待数据", convergence.tone || "gray");
|
|
||||||
const notes = convergence.notes || [];
|
|
||||||
$("converge-notes").innerHTML = notes.length
|
|
||||||
? notes.map((note) => `<li>${escapeHtml(note)}</li>`).join("")
|
|
||||||
: "<li>暂无判读说明。</li>";
|
|
||||||
|
|
||||||
const grid = $("metric-grid");
|
|
||||||
grid.innerHTML = "";
|
|
||||||
const catalog = state.schema.metrics || [];
|
|
||||||
for (const metric of catalog) {
|
|
||||||
const points = seriesOf(metric.key);
|
|
||||||
const values = points.map((p) => p[1]);
|
|
||||||
const value = values.length ? values[values.length - 1] : null;
|
|
||||||
const prev = values.length > 1 ? values[values.length - 2] : null;
|
|
||||||
const delta = value !== null && prev !== null ? value - prev : null;
|
|
||||||
const card = document.createElement("div");
|
|
||||||
card.className = "metric-card";
|
|
||||||
const alert = isAlert(metric.key, value);
|
|
||||||
if (alert) card.classList.add("alert");
|
|
||||||
const trendClass = delta === null || Math.abs(delta) < 1e-12 ? "flat"
|
|
||||||
: delta > 0 ? "up" : "down";
|
|
||||||
const arrow = delta === null || Math.abs(delta) < 1e-12 ? "→" : delta > 0 ? "▲" : "▼";
|
|
||||||
card.innerHTML =
|
|
||||||
`<div class="mhead"><span class="mname">${escapeHtml(metric.label)}</span>` +
|
|
||||||
`<span class="mvalue">${escapeHtml(fmtValue(metric, value))}</span></div>` +
|
|
||||||
`<canvas class="spark" data-spark="${metric.key}"></canvas>` +
|
|
||||||
`<div class="mfoot"><span class="trend ${trendClass}" data-delta="${metric.key}">${arrow} ${escapeHtml(delta === null ? "—" : fmtValue(metric, delta))}</span>` +
|
|
||||||
`<span>${points.length} 点</span></div>`;
|
|
||||||
if (alert) card.title = alert;
|
|
||||||
grid.appendChild(card);
|
|
||||||
const canvas = card.querySelector("canvas");
|
|
||||||
if (canvas && values.length >= 2) {
|
|
||||||
window.Charts.sparkline(canvas, values.slice(-120), { color: metric.color });
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
function isAlert(key, value) {
|
|
||||||
if (value === null || value === undefined) return null;
|
|
||||||
if (key === "approx_kl" && value > 0.05) return "近似 KL 超过 0.05：策略更新幅度过大";
|
|
||||||
if (key === "explained_variance" && value < 0) return "解释方差为负：价值函数尚未拟合";
|
|
||||||
if (key === "entropy_loss" && Math.abs(value) < 1e-3) return "策略熵接近 0：探索可能过早停止";
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
function fmtValue(metric, value) {
|
|
||||||
if (value === null || value === undefined || !isFinite(value)) return "—";
|
|
||||||
if (metric.format === "integer") return fmtInt(value);
|
|
||||||
return fmtNumber(value);
|
|
||||||
}
|
|
||||||
|
|
||||||
function seriesOf(key) {
|
|
||||||
const points = [];
|
|
||||||
for (const row of state.metrics) {
|
|
||||||
const value = Number(row[key]);
|
|
||||||
if (!isFinite(value)) continue;
|
|
||||||
const x = Number(row.timesteps ?? row.total_timesteps ?? points.length + 1);
|
|
||||||
points.push([isFinite(x) ? x : points.length + 1, value]);
|
|
||||||
}
|
|
||||||
return points;
|
|
||||||
}
|
|
||||||
function lastFinite(rows, key) {
|
|
||||||
for (let i = rows.length - 1; i >= 0; i--) {
|
|
||||||
const value = Number(rows[i][key]);
|
|
||||||
if (isFinite(value)) return value;
|
|
||||||
}
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 图表
|
|
||||||
function renderCharts() {
|
|
||||||
const catalog = state.schema.metrics || [];
|
|
||||||
const tabs = $("chart-tabs");
|
|
||||||
if (tabs.dataset.signature !== catalog.map((m) => m.key).join("|")) {
|
|
||||||
tabs.innerHTML = "";
|
|
||||||
for (const metric of catalog) {
|
|
||||||
const btn = document.createElement("button");
|
|
||||||
btn.type = "button";
|
|
||||||
btn.className = "chart-tab";
|
|
||||||
btn.dataset.key = metric.key;
|
|
||||||
btn.textContent = metric.label;
|
|
||||||
btn.onclick = () => { state.chartKey = metric.key; renderCharts(); };
|
|
||||||
tabs.appendChild(btn);
|
|
||||||
}
|
|
||||||
tabs.dataset.signature = catalog.map((m) => m.key).join("|");
|
|
||||||
}
|
|
||||||
for (const btn of tabs.querySelectorAll(".chart-tab")) {
|
|
||||||
btn.classList.toggle("active", btn.dataset.key === state.chartKey);
|
|
||||||
}
|
|
||||||
|
|
||||||
const windowSize = Number($("chart-window").value || 0);
|
|
||||||
const primary = catalog.find((m) => m.role === "primary") || { key: "ep_rew_mean", label: "奖励", color: "#4f8cff" };
|
|
||||||
const rewardPoints = slice(seriesOf(primary.key), windowSize);
|
|
||||||
const layout = window.Charts.line($("chart-reward"), {
|
|
||||||
series: [{ name: primary.label, points: rewardPoints, color: primary.color, fill: true }],
|
|
||||||
xFormat: (v) => `${fmtInt(v)}步`,
|
|
||||||
emptyText: "等待首个 rollout 的 ep_rew_mean……",
|
|
||||||
});
|
|
||||||
setText("chart-title", primary.label);
|
|
||||||
setText("chart-sub", describeSeries(rewardPoints, primary));
|
|
||||||
|
|
||||||
const metric = catalog.find((m) => m.key === state.chartKey) || primary;
|
|
||||||
const metricPoints = slice(seriesOf(metric.key), windowSize);
|
|
||||||
const refs = [];
|
|
||||||
if (metric.key === "approx_kl") refs.push({ value: 0.05, color: "#f87171", label: "过猛阈值 0.05" });
|
|
||||||
if (metric.key === "explained_variance") refs.push({ value: 0, color: "#93a1bd", label: "0（不如均值预测）" });
|
|
||||||
const layout2 = window.Charts.line($("chart-metric"), {
|
|
||||||
series: [{ name: metric.label, points: metricPoints, color: metric.color, fill: true }],
|
|
||||||
refs: refs,
|
|
||||||
xFormat: (v) => `${fmtInt(v)}步`,
|
|
||||||
emptyText: `等待 ${metric.key} 数据……`,
|
|
||||||
});
|
|
||||||
setText("chart2-title", metric.label);
|
|
||||||
setText("chart2-sub", describeSeries(metricPoints, metric));
|
|
||||||
state.layouts = { reward: layout, metric: layout2, metricKey: metric.key, primary: primary };
|
|
||||||
}
|
|
||||||
|
|
||||||
function slice(points, size) {
|
|
||||||
if (!size || points.length <= size) return points;
|
|
||||||
return points.slice(points.length - size);
|
|
||||||
}
|
|
||||||
|
|
||||||
function describeSeries(points, metric) {
|
|
||||||
if (!points.length) return "（无数据）";
|
|
||||||
if (points.length === 1) return `仅 1 点 · 最新 ${fmtValue(metric, points[0][1])}`;
|
|
||||||
const first = points[0], last = points[points.length - 1];
|
|
||||||
const delta = last[1] - first[1];
|
|
||||||
const sign = delta > 0 ? "+" : "";
|
|
||||||
return `${points.length} 点 · 窗口内 ${sign}${fmtValue(metric, delta)} · ` +
|
|
||||||
`最新 ${fmtValue(metric, last[1])} @ ${fmtInt(last[0])} 步`;
|
|
||||||
}
|
|
||||||
|
|
||||||
function bindChartInteractions() {
|
|
||||||
for (const id of ["chart-reward", "chart-metric"]) {
|
|
||||||
const canvas = $(id);
|
|
||||||
if (!canvas) continue;
|
|
||||||
const overlay = document.createElement("canvas");
|
|
||||||
overlay.className = "chart-overlay";
|
|
||||||
overlay.style.width = "100%"; overlay.style.height = "100%";
|
|
||||||
if (canvas.parentElement) {
|
|
||||||
canvas.parentElement.style.position = "relative";
|
|
||||||
canvas.parentElement.appendChild(overlay);
|
|
||||||
}
|
|
||||||
const move = (event) => {
|
|
||||||
const show = $("chart-cursor").checked;
|
|
||||||
const rect = canvas.getBoundingClientRect();
|
|
||||||
const which = id === "chart-reward" ? "reward" : "metric";
|
|
||||||
const layout = state.layouts && state.layouts[which];
|
|
||||||
const ctx = overlay.getContext("2d");
|
|
||||||
const dpr = window.devicePixelRatio || 1;
|
|
||||||
if (overlay.width !== Math.round(rect.width * dpr) || overlay.height !== Math.round(rect.height * dpr)) {
|
|
||||||
overlay.width = Math.round(rect.width * dpr);
|
|
||||||
overlay.height = Math.round(rect.height * dpr);
|
|
||||||
}
|
|
||||||
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
|
|
||||||
ctx.clearRect(0, 0, rect.width, rect.height);
|
|
||||||
if (!layout || !show || !isFinite(layout.bounds.x0)) return;
|
|
||||||
window.Charts.drawCursor(overlay, layout, {
|
|
||||||
mouse: { x: event.clientX - rect.left, y: event.clientY - rect.top },
|
|
||||||
xFormat: (v) => `${fmtInt(v)}步`,
|
|
||||||
});
|
|
||||||
};
|
|
||||||
canvas.addEventListener("mousemove", move);
|
|
||||||
canvas.addEventListener("mouseleave", () => {
|
|
||||||
const ctx = overlay.getContext("2d");
|
|
||||||
ctx.clearRect(0, 0, overlay.width, overlay.height);
|
|
||||||
});
|
|
||||||
}
|
|
||||||
$("chart-window").addEventListener("change", renderCharts);
|
|
||||||
$("chart-cursor").addEventListener("change", renderCharts);
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 日志
|
|
||||||
function renderLog(job) {
|
|
||||||
const lines = job.log_lines || [];
|
|
||||||
const filter = ($("log-filter").value || "").trim().toLowerCase();
|
|
||||||
const showJson = $("log-json").checked;
|
|
||||||
const errorsOnly = $("log-errors-only").checked;
|
|
||||||
const view = $("log-view");
|
|
||||||
const signature = `${lines.length}|${job.log_total}|${filter}|${showJson}|${errorsOnly}|${lines[lines.length - 1] || ""}`;
|
|
||||||
if (signature === state.lastLogSignature) return;
|
|
||||||
state.lastLogSignature = signature;
|
|
||||||
|
|
||||||
const follow = $("log-follow").checked;
|
|
||||||
const atBottom = view.scrollTop + view.clientHeight >= view.scrollHeight - 24;
|
|
||||||
const keep = follow || atBottom;
|
|
||||||
|
|
||||||
const html = [];
|
|
||||||
let shown = 0;
|
|
||||||
for (let i = 0; i < lines.length; i++) {
|
|
||||||
const raw = lines[i];
|
|
||||||
const lower = raw.toLowerCase();
|
|
||||||
if (filter && !lower.includes(filter)) continue;
|
|
||||||
if (errorsOnly && !/error|traceback|failed|exception|错误|失败/.test(lower)) continue;
|
|
||||||
if (!showJson && raw.trim().startsWith("{") && raw.includes('"event"')) continue;
|
|
||||||
shown++;
|
|
||||||
const cls = classify(raw);
|
|
||||||
html.push(`<span class="l ${cls}">${highlight(raw, filter)}</span>`);
|
|
||||||
}
|
|
||||||
view.innerHTML = html.length ? html.join("") : '<span class="l dim">（无匹配日志行）</span>';
|
|
||||||
if (keep) view.scrollTop = view.scrollHeight;
|
|
||||||
setText("log-shown", `显示 ${shown} / 共 ${fmtInt(job.log_total)} 行`);
|
|
||||||
setText("log-meta", `stdout/stderr 合并捕获 · 服务端保留最近 ${fmtInt(job.process ? job.process.log_capacity : 4000)} 行`);
|
|
||||||
}
|
|
||||||
|
|
||||||
function classify(line) {
|
|
||||||
const text = line.toLowerCase();
|
|
||||||
if (/error|traceback|exception|failed|错误|失败/.test(text)) return "err";
|
|
||||||
if (/warn|deprecat|警告/.test(text)) return "warn";
|
|
||||||
if (line.trim().startsWith("{")) return "json";
|
|
||||||
if (/^\s*[-=]{3,}|已保存|success|complete|评估|metadata/.test(text)) return "meta";
|
|
||||||
if (!line.trim()) return "dim";
|
|
||||||
return "";
|
|
||||||
}
|
|
||||||
|
|
||||||
function highlight(line, filter) {
|
|
||||||
const escaped = escapeHtml(line);
|
|
||||||
if (!filter) return escaped;
|
|
||||||
try {
|
|
||||||
return escaped.replace(new RegExp(`(${filter.replace(/[.*+?^${}()|[\]\\]/g, "\\$&")})`, "gi"),
|
|
||||||
"<mark>$1</mark>");
|
|
||||||
} catch (err) { return escaped; }
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 逐回合
|
|
||||||
function renderEpisodes(job) {
|
|
||||||
const rows = job.monitor || [];
|
|
||||||
const tbody = $("episode-table").querySelector("tbody");
|
|
||||||
const tail = rows.slice(-200).reverse();
|
|
||||||
tbody.innerHTML = tail.map((row, index) => {
|
|
||||||
const seq = rows.length - index;
|
|
||||||
return `<tr><td>${seq}</td><td>${fmtNumber(row.r)}</td>` +
|
|
||||||
`<td>${fmtNumber(row.l, 0)}</td><td>${fmtNumber(row.t, 3)}</td></tr>`;
|
|
||||||
}).join("") || '<tr><td colspan="4" class="hint">尚无逐回合记录（Monitor CSV 随训练写入）</td></tr>';
|
|
||||||
}
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 交互
|
|
||||||
async function submitPayload(payload) {
|
|
||||||
const err = $("form-error");
|
|
||||||
err.textContent = "";
|
|
||||||
if (state.saving) return false; // 训练中表单锁定，续训按钮同样不许重复提交
|
|
||||||
if (payload.resume && !payload.model) {
|
|
||||||
err.textContent = "续训需要填写模型输出路径";
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
const res = await fetch("/api/train/start", {
|
|
||||||
method: "POST", headers: { "Content-Type": "application/json" },
|
|
||||||
body: JSON.stringify(payload),
|
|
||||||
});
|
|
||||||
const body = await res.json().catch(() => ({}));
|
|
||||||
if (!res.ok) {
|
|
||||||
err.textContent = body.error || `启动失败（HTTP ${res.status}）`;
|
|
||||||
return false;
|
|
||||||
}
|
|
||||||
state.preview = false;
|
|
||||||
state.jobKey = null;
|
|
||||||
state.metrics = [];
|
|
||||||
state.metricCache = [];
|
|
||||||
state.metricNext = 0;
|
|
||||||
state.metricTotal = 0;
|
|
||||||
await poll();
|
|
||||||
return true;
|
|
||||||
}
|
|
||||||
|
|
||||||
$("config-form").addEventListener("submit", async (event) => {
|
|
||||||
event.preventDefault();
|
|
||||||
if (state.saving) return;
|
|
||||||
const err = $("form-error");
|
|
||||||
err.textContent = "";
|
|
||||||
const values = collectValues();
|
|
||||||
const problems = validateLocally(values);
|
|
||||||
if (problems.length) { err.textContent = problems[0]; return; }
|
|
||||||
const payload = {};
|
|
||||||
for (const [key, value] of Object.entries(values)) {
|
|
||||||
if (value !== null) payload[key] = value;
|
|
||||||
}
|
|
||||||
await submitPayload(payload);
|
|
||||||
});
|
|
||||||
|
|
||||||
$("btn-recover").addEventListener("click", async () => {
|
|
||||||
const recovery = (state.lastJob && state.lastJob.recovery) || {};
|
|
||||||
if (!recovery.can_resume || !recovery.model) return;
|
|
||||||
if (!confirm(`用最近的可恢复产物续训？\n${recovery.model}\n`
|
|
||||||
+ "当前表单里的 timesteps 会作为**追加**步数。")) return;
|
|
||||||
// 同步回表单，用户随后可微调步数再手动点「开始训练」
|
|
||||||
const modelField = $("cfg-model");
|
|
||||||
if (modelField) modelField.value = recovery.model;
|
|
||||||
const resumeField = $("cfg-resume");
|
|
||||||
if (resumeField) resumeField.checked = true;
|
|
||||||
const values = collectValues();
|
|
||||||
values.model = recovery.model;
|
|
||||||
values.resume = true;
|
|
||||||
values.force = false; // --resume 与 --force 互斥
|
|
||||||
await submitPayload(values);
|
|
||||||
});
|
|
||||||
|
|
||||||
$("btn-pause").addEventListener("click", async () => {
|
|
||||||
const resuming = state.lastStatus === "paused";
|
|
||||||
const res = await fetch(resuming ? "/api/train/resume" : "/api/train/pause", {
|
|
||||||
method: "POST", headers: { "Content-Type": "application/json" }, body: "{}",
|
|
||||||
});
|
|
||||||
const body = await res.json().catch(() => ({}));
|
|
||||||
if (!res.ok) {
|
|
||||||
showBanner("bad", `${resuming ? "继续" : "暂停"}失败：${escapeHtml(body.error || res.status)}`);
|
|
||||||
} else if (!resuming && body.paused === false) {
|
|
||||||
showBanner("warn", "当前没有可暂停的训练进程（可能已结束）。");
|
|
||||||
}
|
|
||||||
await poll();
|
|
||||||
});
|
|
||||||
|
|
||||||
$("btn-stop").addEventListener("click", async () => {
|
|
||||||
if (!confirm("确认停止当前训练？\n将通过控制文件请求优雅停止，脚本先保存当前模型与元数据再退出；"
|
|
||||||
+ "若超时未响应则强制结束进程树。")) return;
|
|
||||||
const res = await fetch("/api/train/stop", {
|
|
||||||
method: "POST", headers: { "Content-Type": "application/json" }, body: "{}",
|
|
||||||
});
|
|
||||||
const body = await res.json().catch(() => ({}));
|
|
||||||
if (!res.ok) showBanner("bad", `停止失败：${escapeHtml(body.error || res.status)}`);
|
|
||||||
await poll();
|
|
||||||
});
|
|
||||||
|
|
||||||
$("btn-refresh").addEventListener("click", async () => {
|
|
||||||
state.preview = false;
|
|
||||||
showBanner("", "");
|
|
||||||
await poll();
|
|
||||||
});
|
|
||||||
|
|
||||||
$("btn-preview").addEventListener("click", async () => {
|
|
||||||
try {
|
|
||||||
const scenario = $("preview-scenario") ? $("preview-scenario").value : "rising";
|
|
||||||
const res = await fetch(`/api/train/state?preview=1&scenario=${encodeURIComponent(scenario)}&log_tail=400`);
|
|
||||||
const payload = await res.json();
|
|
||||||
if (!payload.job) throw new Error(payload.error || "服务端未返回示例快照");
|
|
||||||
state.preview = true;
|
|
||||||
state.jobKey = null;
|
|
||||||
state.metrics = [];
|
|
||||||
state.metricCache = [];
|
|
||||||
state.metricNext = 0;
|
|
||||||
state.metricTotal = 0;
|
|
||||||
render(payload.job);
|
|
||||||
} catch (err) {
|
|
||||||
showBanner("bad", `离线预览不可用：${escapeHtml(err.message)}`);
|
|
||||||
}
|
|
||||||
});
|
|
||||||
|
|
||||||
$("btn-reset").addEventListener("click", () => {
|
|
||||||
applyValues({});
|
|
||||||
$("form-error").textContent = "";
|
|
||||||
});
|
|
||||||
|
|
||||||
for (const id of ["log-filter", "log-json", "log-errors-only", "log-tail"]) {
|
|
||||||
$(id).addEventListener("input", () => {
|
|
||||||
state.lastLogSignature = "";
|
|
||||||
poll();
|
|
||||||
});
|
|
||||||
}
|
|
||||||
|
|
||||||
boot();
|
|
||||||
@@ -1,255 +0,0 @@
|
|||||||
/* 训练前端图表：原生 canvas 折线图，无外部依赖（离线可用，与评测前端同风格）。
|
|
||||||
只做三件事：坐标轴自适应、多序列折线、悬停读数；不引入缩放/图例等交互，
|
|
||||||
因为训练曲线每 2 秒整体重绘一次，重量级图表库在这里只会增加风险。
|
|
||||||
|
|
||||||
对外接口（挂到 globalThis.Charts）：
|
|
||||||
Charts.fit(canvas) 按 DPR 调整画布像素
|
|
||||||
Charts.line(canvas, {series, refs, ...}) 折线图
|
|
||||||
Charts.sparkline(canvas, values, opts) 指标卡片内的小趋势线
|
|
||||||
数据约定：series = [{ points: [[x, y], ...], color, width, fill }] */
|
|
||||||
(function (global) {
|
|
||||||
"use strict";
|
|
||||||
|
|
||||||
var AXIS = "#2a3550", GRID = "rgba(42,53,80,.55)", DIM = "#93a1bd", TEXT = "#e6ecf8";
|
|
||||||
var FONT = '11px ui-monospace, SFMono-Regular, Menlo, monospace';
|
|
||||||
|
|
||||||
function fit(canvas) {
|
|
||||||
var dpr = global.devicePixelRatio || 1;
|
|
||||||
var rect = canvas.getBoundingClientRect ? canvas.getBoundingClientRect() : null;
|
|
||||||
var cssWidth = (rect && rect.width) || canvas.clientWidth || canvas.width || 640;
|
|
||||||
var cssHeight = (rect && rect.height) || canvas.clientHeight || canvas.height || 200;
|
|
||||||
var w = Math.max(160, Math.round(cssWidth * dpr));
|
|
||||||
var h = Math.max(80, Math.round(cssHeight * dpr));
|
|
||||||
if (canvas.width !== w || canvas.height !== h) { canvas.width = w; canvas.height = h; }
|
|
||||||
var ctx = canvas.getContext("2d");
|
|
||||||
ctx.setTransform(dpr, 0, 0, dpr, 0, 0);
|
|
||||||
return { ctx: ctx, w: w / dpr, h: h / dpr };
|
|
||||||
}
|
|
||||||
|
|
||||||
/* 第 0 层：数值格式（科学计数法只用于真正小而大的量级） */
|
|
||||||
function fmt(value) {
|
|
||||||
if (value === null || value === undefined || Number.isNaN(value)) return "—";
|
|
||||||
var abs = Math.abs(value);
|
|
||||||
if (abs !== 0 && (abs < 1e-3 || abs >= 1e5)) return value.toExponential(2);
|
|
||||||
if (abs >= 100) return value.toFixed(1);
|
|
||||||
if (abs >= 1) return value.toFixed(2);
|
|
||||||
if (abs >= 0.01) return value.toFixed(3);
|
|
||||||
return value.toFixed(4);
|
|
||||||
}
|
|
||||||
|
|
||||||
function niceTicks(min, max, count) {
|
|
||||||
if (!isFinite(min) || !isFinite(max)) return [0, 1];
|
|
||||||
if (min === max) { var pad = Math.abs(min) > 1e-9 ? Math.abs(min) * 0.1 : 0.5;
|
|
||||||
min -= pad; max += pad; }
|
|
||||||
var span = max - min;
|
|
||||||
var raw = span / Math.max(1, count);
|
|
||||||
var mag = Math.pow(10, Math.floor(Math.log10(raw)));
|
|
||||||
var norm = raw / mag;
|
|
||||||
var step = (norm <= 1 ? 1 : norm <= 2 ? 2 : norm <= 5 ? 5 : 10) * mag;
|
|
||||||
var start = Math.ceil(min / step) * step;
|
|
||||||
var ticks = [];
|
|
||||||
for (var v = start; v <= max + step * 1e-6 && ticks.length < 40; v += step) {
|
|
||||||
ticks.push(Math.abs(v) < step * 1e-9 ? 0 : v);
|
|
||||||
}
|
|
||||||
if (!ticks.length) ticks = [min, max];
|
|
||||||
return ticks;
|
|
||||||
}
|
|
||||||
|
|
||||||
function bounds(series) {
|
|
||||||
var xs = [], ys = [];
|
|
||||||
for (var i = 0; i < series.length; i++) {
|
|
||||||
var pts = series[i].points || [];
|
|
||||||
for (var j = 0; j < pts.length; j++) {
|
|
||||||
var p = pts[j];
|
|
||||||
if (!p || !isFinite(p[0]) || !isFinite(p[1])) continue;
|
|
||||||
xs.push(p[0]); ys.push(p[1]);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if (!xs.length) return null;
|
|
||||||
var x0 = Math.min.apply(null, xs), x1 = Math.max.apply(null, xs);
|
|
||||||
var y0 = Math.min.apply(null, ys), y1 = Math.max.apply(null, ys);
|
|
||||||
if (x0 === x1) { x0 -= 0.5; x1 += 0.5; }
|
|
||||||
if (y0 === y1) { var pad = Math.abs(y0) > 1e-9 ? Math.abs(y0) * 0.08 : 0.5;
|
|
||||||
y0 -= pad; y1 += pad; }
|
|
||||||
return { x0: x0, x1: x1, y0: y0, y1: y1 };
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 折线图 */
|
|
||||||
function line(canvas, options) {
|
|
||||||
var opts = options || {};
|
|
||||||
var axes = fit(canvas), ctx = axes.ctx, W = axes.w, H = axes.h;
|
|
||||||
var padL = opts.padLeft || 62, padR = 14, padT = 10, padB = 24;
|
|
||||||
ctx.clearRect(0, 0, W, H);
|
|
||||||
|
|
||||||
var series = (opts.series || []).filter(function (s) {
|
|
||||||
return s && s.points && s.points.length;
|
|
||||||
});
|
|
||||||
var box = bounds(series);
|
|
||||||
if (!box) {
|
|
||||||
ctx.fillStyle = DIM; ctx.font = FONT;
|
|
||||||
ctx.fillText(opts.emptyText || "等待数据……", padL, H / 2);
|
|
||||||
return null;
|
|
||||||
}
|
|
||||||
var yTicks = niceTicks(box.y0, box.y1, 4);
|
|
||||||
var yMin = Math.min(box.y0, yTicks[0]), yMax = Math.max(box.y1, yTicks[yTicks.length - 1]);
|
|
||||||
var plotW = Math.max(10, W - padL - padR), plotH = Math.max(10, H - padT - padB);
|
|
||||||
var sx = function (x) { return padL + (x - box.x0) / (box.x1 - box.x0) * plotW; };
|
|
||||||
var sy = function (y) { return padT + (1 - (y - yMin) / (yMax - yMin)) * plotH; };
|
|
||||||
|
|
||||||
// 网格与刻度
|
|
||||||
ctx.font = FONT; ctx.lineWidth = 1;
|
|
||||||
for (var i = 0; i < yTicks.length; i++) {
|
|
||||||
var y = sy(yTicks[i]);
|
|
||||||
ctx.strokeStyle = GRID; ctx.beginPath();
|
|
||||||
ctx.moveTo(padL, y + 0.5); ctx.lineTo(padL + plotW, y + 0.5); ctx.stroke();
|
|
||||||
ctx.fillStyle = DIM; ctx.textAlign = "right";
|
|
||||||
ctx.fillText(fmt(yTicks[i]), padL - 6, y + 3.5);
|
|
||||||
}
|
|
||||||
ctx.textAlign = "center";
|
|
||||||
for (var t = 0; t <= 4; t++) {
|
|
||||||
var xv = box.x0 + (box.x1 - box.x0) * (t / 4);
|
|
||||||
var x = sx(xv);
|
|
||||||
ctx.fillStyle = DIM;
|
|
||||||
ctx.fillText(opts.xFormat ? opts.xFormat(xv) : fmt(xv), x, H - 7);
|
|
||||||
}
|
|
||||||
ctx.textAlign = "left";
|
|
||||||
ctx.strokeStyle = AXIS; ctx.beginPath();
|
|
||||||
ctx.moveTo(padL + 0.5, padT); ctx.lineTo(padL + 0.5, padT + plotH);
|
|
||||||
ctx.lineTo(padL + plotW, padT + plotH); ctx.stroke();
|
|
||||||
|
|
||||||
// 参考线（例如 KL 的 0.05 阈值、解释方差的 0）
|
|
||||||
var refs = opts.refs || [];
|
|
||||||
for (var r = 0; r < refs.length; r++) {
|
|
||||||
var ref = refs[r];
|
|
||||||
if (ref.value === undefined || ref.value < yMin || ref.value > yMax) continue;
|
|
||||||
var ry = sy(ref.value);
|
|
||||||
ctx.save();
|
|
||||||
ctx.setLineDash([4, 4]); ctx.strokeStyle = ref.color || "#f59e0b";
|
|
||||||
ctx.beginPath(); ctx.moveTo(padL, ry); ctx.lineTo(padL + plotW, ry); ctx.stroke();
|
|
||||||
ctx.restore();
|
|
||||||
if (ref.label) {
|
|
||||||
ctx.fillStyle = ref.color || "#f59e0b"; ctx.font = FONT;
|
|
||||||
ctx.fillText(ref.label, padL + 6, ry - 4);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
// 序列
|
|
||||||
for (var s = 0; s < series.length; s++) {
|
|
||||||
var item = series[s];
|
|
||||||
var pts = item.points;
|
|
||||||
ctx.strokeStyle = item.color || "#4f8cff";
|
|
||||||
ctx.lineWidth = item.width || 1.6;
|
|
||||||
ctx.lineJoin = "round";
|
|
||||||
ctx.beginPath();
|
|
||||||
for (var k = 0; k < pts.length; k++) {
|
|
||||||
var px = sx(pts[k][0]), py = sy(pts[k][1]);
|
|
||||||
if (k === 0) ctx.moveTo(px, py); else ctx.lineTo(px, py);
|
|
||||||
}
|
|
||||||
ctx.stroke();
|
|
||||||
if (item.fill && pts.length > 1) {
|
|
||||||
ctx.save();
|
|
||||||
ctx.globalAlpha = 0.12; ctx.fillStyle = item.color || "#4f8cff";
|
|
||||||
ctx.lineTo(sx(pts[pts.length - 1][0]), padT + plotH);
|
|
||||||
ctx.lineTo(sx(pts[0][0]), padT + plotH);
|
|
||||||
ctx.closePath(); ctx.fill(); ctx.restore();
|
|
||||||
}
|
|
||||||
// 末点高亮：一眼看到最新值
|
|
||||||
var last = pts[pts.length - 1];
|
|
||||||
ctx.fillStyle = item.color || "#4f8cff";
|
|
||||||
ctx.beginPath(); ctx.arc(sx(last[0]), sy(last[1]), 2.6, 0, Math.PI * 2); ctx.fill();
|
|
||||||
}
|
|
||||||
|
|
||||||
return { bounds: box, sx: sx, sy: sy, padL: padL, padT: padT,
|
|
||||||
plotW: plotW, plotH: plotH, yMin: yMin, yMax: yMax, series: series };
|
|
||||||
}
|
|
||||||
|
|
||||||
/* 悬停读数：把鼠标 x 映射到最接近的数据点，返回一次绘制的覆盖层信息 */
|
|
||||||
function drawCursor(canvas, layout, options) {
|
|
||||||
var opts = options || {};
|
|
||||||
if (!layout || !layout.series || !layout.series.length) return;
|
|
||||||
var axes = fit(canvas), ctx = axes.ctx;
|
|
||||||
var mouse = opts.mouse;
|
|
||||||
if (!mouse || mouse.x < layout.padL || mouse.x > layout.padL + layout.plotW) return;
|
|
||||||
var target = layout.bounds.x0 +
|
|
||||||
(mouse.x - layout.padL) / layout.plotW * (layout.bounds.x1 - layout.bounds.x0);
|
|
||||||
var lines = [], snapX = target;
|
|
||||||
for (var s = 0; s < layout.series.length; s++) {
|
|
||||||
var pts = layout.series[s].points;
|
|
||||||
if (!pts.length) continue;
|
|
||||||
// 数据已按 x 递增，二分定位最近点
|
|
||||||
var lo = 0, hi = pts.length - 1, best = pts[hi];
|
|
||||||
while (lo <= hi) {
|
|
||||||
var mid = (lo + hi) >> 1;
|
|
||||||
if (pts[mid][0] === target) { best = pts[mid]; break; }
|
|
||||||
if (pts[mid][0] < target) { best = pts[mid]; lo = mid + 1; }
|
|
||||||
else { hi = mid - 1; }
|
|
||||||
}
|
|
||||||
var next = pts[Math.min(pts.length - 1, lo)];
|
|
||||||
if (next && Math.abs(next[0] - target) < Math.abs(best[0] - target)) best = next;
|
|
||||||
snapX = best[0];
|
|
||||||
lines.push({ label: layout.series[s].name || "", color: layout.series[s].color,
|
|
||||||
value: best[1] });
|
|
||||||
}
|
|
||||||
var cx = layout.sx(snapX);
|
|
||||||
ctx.save();
|
|
||||||
ctx.setLineDash([3, 3]); ctx.strokeStyle = "rgba(230,236,248,.35)";
|
|
||||||
ctx.beginPath(); ctx.moveTo(cx, layout.padT);
|
|
||||||
ctx.lineTo(cx, layout.padT + layout.plotH); ctx.stroke();
|
|
||||||
ctx.restore();
|
|
||||||
for (var i = 0; i < lines.length; i++) {
|
|
||||||
var item = lines[i];
|
|
||||||
var cy = layout.sy(item.value);
|
|
||||||
ctx.fillStyle = item.color || "#4f8cff";
|
|
||||||
ctx.beginPath(); ctx.arc(cx, cy, 3.2, 0, Math.PI * 2); ctx.fill();
|
|
||||||
ctx.strokeStyle = "rgba(11,15,26,.9)"; ctx.lineWidth = 1.4; ctx.stroke();
|
|
||||||
}
|
|
||||||
var text = (opts.xFormat ? opts.xFormat(snapX) : fmt(snapX)) + " · " +
|
|
||||||
lines.map(function (item) { return (item.label ? item.label + " " : "") + fmt(item.value); })
|
|
||||||
.join(" ");
|
|
||||||
ctx.font = FONT;
|
|
||||||
var tw = ctx.measureText(text).width + 14;
|
|
||||||
var bx = Math.min(cx + 8, layout.padL + layout.plotW - tw);
|
|
||||||
bx = Math.max(layout.padL + 2, bx);
|
|
||||||
var by = layout.padT + 4;
|
|
||||||
ctx.fillStyle = "rgba(23,30,46,.94)"; ctx.strokeStyle = AXIS;
|
|
||||||
ctx.beginPath();
|
|
||||||
if (ctx.roundRect) { ctx.roundRect(bx, by, tw, 20, 6); ctx.fill(); ctx.stroke(); }
|
|
||||||
else { ctx.fillRect(bx, by, tw, 20); ctx.strokeRect(bx, by, tw, 20); }
|
|
||||||
ctx.fillStyle = TEXT; ctx.textAlign = "left";
|
|
||||||
ctx.fillText(text, bx + 7, by + 14);
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 迷你趋势线 */
|
|
||||||
function sparkline(canvas, values, options) {
|
|
||||||
var opts = options || {};
|
|
||||||
var axes = fit(canvas), ctx = axes.ctx, W = axes.w, H = axes.h;
|
|
||||||
ctx.clearRect(0, 0, W, H);
|
|
||||||
var clean = (values || []).filter(function (v) { return isFinite(v); });
|
|
||||||
if (clean.length < 2) {
|
|
||||||
ctx.strokeStyle = GRID; ctx.beginPath();
|
|
||||||
ctx.moveTo(0, H - 1.5); ctx.lineTo(W, H - 1.5); ctx.stroke();
|
|
||||||
return;
|
|
||||||
}
|
|
||||||
var lo = Math.min.apply(null, clean), hi = Math.max.apply(null, clean);
|
|
||||||
if (lo === hi) { lo -= 0.5; hi += 0.5; }
|
|
||||||
var pad = 2;
|
|
||||||
var X = function (i) { return (i / (clean.length - 1)) * W; };
|
|
||||||
var Y = function (v) { return H - pad - (v - lo) / (hi - lo) * (H - pad * 2); };
|
|
||||||
ctx.strokeStyle = opts.color || "#4f8cff"; ctx.lineWidth = 1.4;
|
|
||||||
ctx.beginPath();
|
|
||||||
for (var i = 0; i < clean.length; i++) {
|
|
||||||
if (i === 0) ctx.moveTo(X(i), Y(clean[i])); else ctx.lineTo(X(i), Y(clean[i]));
|
|
||||||
}
|
|
||||||
ctx.stroke();
|
|
||||||
if (opts.fill !== false) {
|
|
||||||
ctx.save(); ctx.globalAlpha = 0.14; ctx.fillStyle = opts.color || "#4f8cff";
|
|
||||||
ctx.lineTo(W, H); ctx.lineTo(0, H); ctx.closePath(); ctx.fill(); ctx.restore();
|
|
||||||
}
|
|
||||||
var lastX = X(clean.length - 1), lastY = Y(clean[clean.length - 1]);
|
|
||||||
ctx.fillStyle = opts.color || "#4f8cff";
|
|
||||||
ctx.beginPath(); ctx.arc(lastX - 1.5, lastY, 2.2, 0, Math.PI * 2); ctx.fill();
|
|
||||||
}
|
|
||||||
|
|
||||||
global.Charts = { fit: fit, line: line, drawCursor: drawCursor,
|
|
||||||
sparkline: sparkline, fmt: fmt };
|
|
||||||
})(typeof globalThis !== "undefined" ? globalThis : this);
|
|
||||||
@@ -1,180 +0,0 @@
|
|||||||
<!doctype html>
|
|
||||||
<html lang="zh-CN">
|
|
||||||
<head>
|
|
||||||
<meta charset="utf-8">
|
|
||||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
|
||||||
<title>模型训练前端 · CUMCM 2026 B HRL</title>
|
|
||||||
<link rel="stylesheet" href="/style.css">
|
|
||||||
</head>
|
|
||||||
<body>
|
|
||||||
<header class="topbar">
|
|
||||||
<div class="brand">
|
|
||||||
<h1>模型训练前端 <span class="sub">HRL 高层 MaskablePPO</span></h1>
|
|
||||||
<p class="tagline" id="job-line">尚未启动训练任务</p>
|
|
||||||
</div>
|
|
||||||
<div class="topstatus">
|
|
||||||
<span class="chip gray" id="chip-process">进程<b>未启动</b></span>
|
|
||||||
<span class="chip gray" id="chip-progress">进度<b>—</b></span>
|
|
||||||
<span class="chip gray" id="chip-elapsed">墙钟<b>—</b></span>
|
|
||||||
<span class="chip gray" id="chip-device">设备<b>—</b></span>
|
|
||||||
<span class="chip gray" id="chip-conn">连接<b>探测中</b></span>
|
|
||||||
</div>
|
|
||||||
<div class="topactions">
|
|
||||||
<button id="btn-refresh" type="button" title="立即拉取一次快照">刷新</button>
|
|
||||||
<label class="inline" title="离线预览：读取示例快照，用于无训练进程时检查界面">
|
|
||||||
预览
|
|
||||||
<select id="preview-scenario">
|
|
||||||
<option value="rising">上升</option>
|
|
||||||
<option value="plateau">平台期</option>
|
|
||||||
<option value="kl-spike">KL 过猛</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
<button id="btn-preview" type="button" title="渲染示例快照，不启动任何训练进程">离线预览</button>
|
|
||||||
<button id="btn-pause" type="button" disabled>暂停训练</button>
|
|
||||||
<button id="btn-stop" class="danger" type="button" disabled>停止训练</button>
|
|
||||||
</div>
|
|
||||||
</header>
|
|
||||||
|
|
||||||
<main>
|
|
||||||
<div id="banner" class="banner hidden"></div>
|
|
||||||
|
|
||||||
<div id="recovery-box" class="recovery hidden">
|
|
||||||
<span class="hint" id="recovery-text">—</span>
|
|
||||||
<button id="btn-recover" type="button" title="把最近一次完整落盘的产物作为 --resume 目标继续训练">用最近检查点续训</button>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<!-- =============================================== 1. 执行进度与进程 -->
|
|
||||||
<section class="panel" id="panel-process">
|
|
||||||
<div class="panel-head">
|
|
||||||
<h2>执行进度与进程</h2>
|
|
||||||
<span class="hint" id="process-hint">启动任务后显示子进程与步进进度</span>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="progress-row">
|
|
||||||
<div class="progress-main">
|
|
||||||
<div class="progress-labels">
|
|
||||||
<span><b id="steps-done">0</b> / <span id="steps-total">0</span> 环境步</span>
|
|
||||||
<span class="hint" id="progress-percent">0.0%</span>
|
|
||||||
</div>
|
|
||||||
<div class="bar"><div class="bar-fill" id="bar-fill"></div></div>
|
|
||||||
<div class="progress-foot">
|
|
||||||
<span class="hint">剩余估计 <b id="eta">—</b></span>
|
|
||||||
<span class="hint">预计结束 <b id="eta-clock">—</b></span>
|
|
||||||
<span class="hint">进度点 <b id="metric-total">0</b></span>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div class="stat-grid">
|
|
||||||
<div class="stat"><span class="k">子进程 PID</span><b class="v" id="stat-pid">—</b></div>
|
|
||||||
<div class="stat"><span class="k">进程组 / 会话</span><b class="v" id="stat-pgid">独立</b></div>
|
|
||||||
<div class="stat"><span class="k">吞吐 FPS</span><b class="v" id="stat-fps">—</b></div>
|
|
||||||
<div class="stat"><span class="k">已用墙钟</span><b class="v" id="stat-elapsed">—</b></div>
|
|
||||||
<div class="stat"><span class="k">退出码</span><b class="v" id="stat-rc">—</b></div>
|
|
||||||
<div class="stat"><span class="k">日志行数</span><b class="v" id="stat-logs">0</b></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div class="timeline" id="timeline"></div>
|
|
||||||
<p class="hint" id="command-preview"></p>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<!-- =============================================== 2. 训练配置 -->
|
|
||||||
<section class="panel" id="panel-config">
|
|
||||||
<div class="panel-head">
|
|
||||||
<h2>训练配置</h2>
|
|
||||||
<span class="hint">字段与后端 <code>TrainConfig</code> 同源；留空字段取默认值</span>
|
|
||||||
</div>
|
|
||||||
<div class="presets" id="presets"></div>
|
|
||||||
<form id="config-form" autocomplete="off">
|
|
||||||
<div class="form-groups" id="form-groups"></div>
|
|
||||||
<div class="form-foot">
|
|
||||||
<button id="btn-start" class="primary" type="submit">开始训练</button>
|
|
||||||
<button id="btn-reset" type="button">恢复默认</button>
|
|
||||||
<span class="form-error" id="form-error"></span>
|
|
||||||
</div>
|
|
||||||
</form>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<!-- =============================================== 3. 参数收敛 -->
|
|
||||||
<section class="panel" id="panel-convergence">
|
|
||||||
<div class="panel-head">
|
|
||||||
<h2>参数收敛状态</h2>
|
|
||||||
<span class="hint">判据为最近窗口的斜率与波动，仅供人工复核</span>
|
|
||||||
</div>
|
|
||||||
<div class="converge-head">
|
|
||||||
<span class="chip gray" id="chip-converge">判读<b>等待数据</b></span>
|
|
||||||
<ul class="notes" id="converge-notes"></ul>
|
|
||||||
</div>
|
|
||||||
<div class="metric-grid" id="metric-grid"></div>
|
|
||||||
|
|
||||||
<div class="chart-toolbar">
|
|
||||||
<div class="chart-tabs" id="chart-tabs"></div>
|
|
||||||
<label class="inline">最近点数
|
|
||||||
<select id="chart-window">
|
|
||||||
<option value="0">全部</option>
|
|
||||||
<option value="40">40</option>
|
|
||||||
<option value="120" selected>120</option>
|
|
||||||
<option value="400">400</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
<label class="inline"><input type="checkbox" id="chart-cursor" checked> 悬停读数</label>
|
|
||||||
</div>
|
|
||||||
<div class="chart-stack">
|
|
||||||
<figure class="chart-figure">
|
|
||||||
<figcaption><b id="chart-title">平均回合奖励</b><span class="hint" id="chart-sub"></span></figcaption>
|
|
||||||
<canvas id="chart-reward" height="240"></canvas>
|
|
||||||
</figure>
|
|
||||||
<figure class="chart-figure">
|
|
||||||
<figcaption><b id="chart2-title">近似 KL</b><span class="hint" id="chart2-sub"></span></figcaption>
|
|
||||||
<canvas id="chart-metric" height="180"></canvas>
|
|
||||||
</figure>
|
|
||||||
</div>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<!-- =============================================== 4. 实时运行日志 -->
|
|
||||||
<section class="panel" id="panel-log">
|
|
||||||
<div class="panel-head">
|
|
||||||
<h2>实时运行日志</h2>
|
|
||||||
<span class="hint" id="log-meta">stdout/stderr 合并捕获</span>
|
|
||||||
</div>
|
|
||||||
<div class="log-toolbar">
|
|
||||||
<input id="log-filter" type="search" placeholder="过滤关键字，如 rollout / progress / error">
|
|
||||||
<label class="inline"><input type="checkbox" id="log-json" checked> 显示进度 JSON</label>
|
|
||||||
<label class="inline"><input type="checkbox" id="log-errors-only"> 只看错误</label>
|
|
||||||
<label class="inline"><input type="checkbox" id="log-follow" checked> 自动跟随到底部</label>
|
|
||||||
<label class="inline">尾部行数
|
|
||||||
<select id="log-tail">
|
|
||||||
<option value="200">200</option>
|
|
||||||
<option value="400" selected>400</option>
|
|
||||||
<option value="1000">1000</option>
|
|
||||||
<option value="2000">2000</option>
|
|
||||||
</select>
|
|
||||||
</label>
|
|
||||||
<span class="hint" id="log-shown"></span>
|
|
||||||
</div>
|
|
||||||
<pre class="log-view" id="log-view">（暂无输出）</pre>
|
|
||||||
</section>
|
|
||||||
|
|
||||||
<!-- =============================================== 5. 逐回合奖励 -->
|
|
||||||
<section class="panel" id="panel-episodes">
|
|
||||||
<div class="panel-head">
|
|
||||||
<h2>逐回合记录</h2>
|
|
||||||
<span class="hint">来自训练脚本写入的 Monitor CSV（每回合奖励/长度/时刻）</span>
|
|
||||||
</div>
|
|
||||||
<div class="table-wrap">
|
|
||||||
<table id="episode-table">
|
|
||||||
<thead><tr><th>#</th><th>奖励 r</th><th>长度 l</th><th>时刻 t (s)</th></tr></thead>
|
|
||||||
<tbody></tbody>
|
|
||||||
</table>
|
|
||||||
</div>
|
|
||||||
</section>
|
|
||||||
</main>
|
|
||||||
|
|
||||||
<footer>
|
|
||||||
本地回环训练前端：与评测前端共用「白名单静态资源 + JSON 轮询 API」模式；
|
|
||||||
命令由服务端以参数列表构造（不经过 shell），停止时向独立进程组发送 SIGTERM。
|
|
||||||
</footer>
|
|
||||||
|
|
||||||
<script src="/charts.js"></script>
|
|
||||||
<script src="/app.js"></script>
|
|
||||||
</body>
|
|
||||||
</html>
|
|
||||||
@@ -1,230 +0,0 @@
|
|||||||
/* 训练前端样式：沿用评测前端（referee/static/style.css）的暗色变量与圆角面板
|
|
||||||
语言，保证两个界面像同一套系统的两个页面；本页信息密度更高：
|
|
||||||
进度条、进程时间线、参数卡片网格、曲线与日志终端。 */
|
|
||||||
|
|
||||||
:root {
|
|
||||||
--bg: #0f1420; --panel: #171e2e; --panel2: #1d2740; --line: #2a3550;
|
|
||||||
--text: #e6ecf8; --dim: #93a1bd; --accent: #4f8cff; --ok: #34d399;
|
|
||||||
--warn: #f59e0b; --bad: #f87171; --cyan: #38bdf8; --gray: #6b7280;
|
|
||||||
--mono: ui-monospace, SFMono-Regular, Menlo, Consolas, monospace;
|
|
||||||
}
|
|
||||||
* { box-sizing: border-box; }
|
|
||||||
body {
|
|
||||||
margin: 0; background: var(--bg); color: var(--text);
|
|
||||||
font: 14px/1.5 "PingFang SC", "Microsoft YaHei", system-ui, sans-serif;
|
|
||||||
}
|
|
||||||
code, .mono { background: var(--panel2); padding: 1px 6px; border-radius: 4px; color: var(--cyan); font-family: var(--mono); font-size: 12px; }
|
|
||||||
main { max-width: 1240px; margin: 0 auto; padding: 16px 22px 48px; }
|
|
||||||
footer { color: var(--dim); font-size: 12px; text-align: center; padding: 14px 22px; border-top: 1px solid var(--line); }
|
|
||||||
.hidden { display: none !important; }
|
|
||||||
.hint { color: var(--dim); font-size: 12px; }
|
|
||||||
h2 { font-size: 15px; margin: 0; }
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 顶栏 */
|
|
||||||
.topbar {
|
|
||||||
position: sticky; top: 0; z-index: 20;
|
|
||||||
display: flex; align-items: center; gap: 18px; flex-wrap: wrap;
|
|
||||||
padding: 12px 22px; border-bottom: 1px solid var(--line);
|
|
||||||
background: rgba(23,30,46,.96); backdrop-filter: blur(6px);
|
|
||||||
}
|
|
||||||
.brand h1 { font-size: 17px; margin: 0; }
|
|
||||||
.brand .sub { font-size: 12px; color: var(--dim); font-weight: normal; margin-left: 8px; }
|
|
||||||
.brand .tagline { margin: 3px 0 0; color: var(--dim); font-size: 12px; font-family: var(--mono); }
|
|
||||||
.topstatus { display: flex; gap: 8px; flex-wrap: wrap; margin-left: auto; }
|
|
||||||
.topactions { display: flex; gap: 8px; }
|
|
||||||
|
|
||||||
.chip {
|
|
||||||
background: var(--panel2); border: 1px solid var(--line); border-radius: 999px;
|
|
||||||
padding: 3px 12px; font-size: 12px; color: var(--dim); white-space: nowrap;
|
|
||||||
}
|
|
||||||
.chip b { color: var(--text); font-weight: 600; margin-left: 4px; }
|
|
||||||
.chip.gray b { color: var(--gray); }
|
|
||||||
.chip.ok b { color: var(--ok); }
|
|
||||||
.chip.warn b { color: var(--warn); }
|
|
||||||
.chip.bad b { color: var(--bad); }
|
|
||||||
.chip.cyan b { color: var(--cyan); }
|
|
||||||
|
|
||||||
button, .button {
|
|
||||||
background: var(--panel2); color: var(--text); border: 1px solid var(--line);
|
|
||||||
border-radius: 6px; padding: 7px 14px; font-size: 13px; cursor: pointer;
|
|
||||||
}
|
|
||||||
button:hover:enabled, .button:hover { border-color: var(--accent); }
|
|
||||||
button:disabled { opacity: .45; cursor: not-allowed; }
|
|
||||||
button.primary { background: var(--accent); border-color: var(--accent); color: #fff; }
|
|
||||||
button.primary:hover:enabled { filter: brightness(1.08); }
|
|
||||||
button.danger { background: transparent; border-color: var(--bad); color: var(--bad); }
|
|
||||||
button.danger:hover:enabled { background: rgba(248,113,113,.12); }
|
|
||||||
|
|
||||||
.banner {
|
|
||||||
border: 1px solid var(--bad); border-left: 4px solid var(--bad);
|
|
||||||
background: rgba(248,113,113,.08); border-radius: 8px;
|
|
||||||
padding: 10px 14px; margin: 14px 0 0; font-size: 13px;
|
|
||||||
}
|
|
||||||
.banner.ok { border-color: var(--ok); border-left-color: var(--ok); background: rgba(52,211,153,.08); }
|
|
||||||
.banner.warn { border-color: var(--warn); border-left-color: var(--warn); background: rgba(245,158,11,.08); }
|
|
||||||
.banner b { font-family: var(--mono); }
|
|
||||||
|
|
||||||
/* 恢复条：显示最近一份完整落盘的产物，并提供一键续训 */
|
|
||||||
.recovery {
|
|
||||||
display: flex; align-items: center; justify-content: space-between; gap: 12px;
|
|
||||||
border: 1px solid var(--cyan, #22d3ee); border-left: 4px solid var(--cyan, #22d3ee);
|
|
||||||
background: rgba(34,211,238,.07); border-radius: 8px;
|
|
||||||
padding: 8px 12px; margin: 10px 0 0; font-size: 13px;
|
|
||||||
}
|
|
||||||
.recovery .hint { overflow-wrap: anywhere; }
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 面板 */
|
|
||||||
.panel {
|
|
||||||
border: 1px solid var(--line); border-radius: 12px; background: var(--panel);
|
|
||||||
padding: 14px 16px; margin-top: 16px;
|
|
||||||
}
|
|
||||||
.panel-head { display: flex; align-items: baseline; gap: 12px; margin-bottom: 12px; flex-wrap: wrap; }
|
|
||||||
.panel-head h2 { color: var(--text); }
|
|
||||||
.panel-head h2::before {
|
|
||||||
content: ""; display: inline-block; width: 3px; height: 13px; margin-right: 8px;
|
|
||||||
background: var(--accent); border-radius: 2px; vertical-align: -1px;
|
|
||||||
}
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 进度 */
|
|
||||||
.progress-row { display: flex; gap: 20px; align-items: stretch; flex-wrap: wrap; }
|
|
||||||
.progress-main { flex: 1 1 380px; min-width: 320px; }
|
|
||||||
.progress-labels { display: flex; justify-content: space-between; font-size: 13px; margin-bottom: 6px; }
|
|
||||||
.progress-labels b { font-family: var(--mono); font-size: 15px; }
|
|
||||||
.bar { height: 12px; border-radius: 999px; background: #0b0f1a; border: 1px solid var(--line); overflow: hidden; }
|
|
||||||
.bar-fill {
|
|
||||||
height: 100%; width: 0%; border-radius: 999px;
|
|
||||||
background: linear-gradient(90deg, #2f6fe0, var(--accent) 60%, var(--cyan));
|
|
||||||
transition: width .3s ease;
|
|
||||||
}
|
|
||||||
.bar-fill.warn { background: linear-gradient(90deg, #b45309, var(--warn)); }
|
|
||||||
.bar-fill.bad { background: linear-gradient(90deg, #b91c1c, var(--bad)); }
|
|
||||||
.bar-fill.ok { background: linear-gradient(90deg, #059669, var(--ok)); }
|
|
||||||
.progress-foot { display: flex; gap: 18px; margin-top: 7px; flex-wrap: wrap; }
|
|
||||||
|
|
||||||
.stat-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(132px, 1fr)); gap: 8px; flex: 1 1 420px; }
|
|
||||||
.stat {
|
|
||||||
border: 1px solid var(--line); border-radius: 8px; background: var(--panel2);
|
|
||||||
padding: 7px 10px; display: flex; flex-direction: column; gap: 2px;
|
|
||||||
}
|
|
||||||
.stat .k { color: var(--dim); font-size: 11px; }
|
|
||||||
.stat .v { font-family: var(--mono); font-size: 15px; }
|
|
||||||
.stat .v.ok { color: var(--ok); }
|
|
||||||
.stat .v.warn { color: var(--warn); }
|
|
||||||
.stat .v.bad { color: var(--bad); }
|
|
||||||
|
|
||||||
.timeline { display: flex; align-items: center; gap: 0; margin-top: 16px; flex-wrap: wrap; }
|
|
||||||
.node { display: flex; align-items: center; gap: 8px; }
|
|
||||||
.node .dot {
|
|
||||||
width: 12px; height: 12px; border-radius: 50%; border: 2px solid var(--line);
|
|
||||||
background: var(--panel2); flex: none;
|
|
||||||
}
|
|
||||||
.node .txt { font-size: 12px; color: var(--dim); }
|
|
||||||
.node.done .dot { border-color: var(--ok); background: var(--ok); }
|
|
||||||
.node.done .txt { color: var(--text); }
|
|
||||||
.node.active .dot { border-color: var(--accent); background: var(--accent); box-shadow: 0 0 0 4px rgba(79,140,255,.18); }
|
|
||||||
.node.active .txt { color: var(--accent); font-weight: 600; }
|
|
||||||
.node.failed .dot { border-color: var(--bad); background: var(--bad); }
|
|
||||||
.node.failed .txt { color: var(--bad); }
|
|
||||||
.link { width: 34px; height: 2px; background: var(--line); flex: none; margin: 0 8px; }
|
|
||||||
.link.done { background: var(--ok); }
|
|
||||||
#command-preview { margin: 12px 0 0; font-family: var(--mono); word-break: break-all; line-height: 1.7; }
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 表单 */
|
|
||||||
.presets { display: flex; flex-wrap: wrap; gap: 8px; margin-bottom: 14px; }
|
|
||||||
.preset {
|
|
||||||
border: 1px solid var(--line); border-radius: 8px; background: var(--panel2);
|
|
||||||
padding: 7px 12px; font-size: 12px; cursor: pointer; color: var(--text);
|
|
||||||
}
|
|
||||||
.preset:hover { border-color: var(--accent); }
|
|
||||||
.preset .note { display: block; color: var(--dim); font-size: 11px; margin-top: 2px; }
|
|
||||||
.form-groups { display: grid; grid-template-columns: repeat(auto-fit, minmax(268px, 1fr)); gap: 14px; }
|
|
||||||
fieldset.group { border: 1px solid var(--line); border-radius: 10px; padding: 10px 12px 12px; margin: 0; background: #141b29; }
|
|
||||||
fieldset.group legend { color: var(--dim); font-size: 12px; padding: 0 6px; }
|
|
||||||
fieldset.group .ghint { color: var(--dim); font-size: 11px; margin: 0 0 8px; }
|
|
||||||
.field { display: flex; flex-direction: column; gap: 4px; margin-bottom: 10px; }
|
|
||||||
.field .flabel { font-size: 12px; color: var(--dim); display: flex; gap: 6px; align-items: baseline; }
|
|
||||||
.field .flabel .unit { color: var(--gray); font-size: 11px; }
|
|
||||||
.field input[type="text"], .field input[type="number"], .field select {
|
|
||||||
background: var(--panel2); border: 1px solid var(--line); color: var(--text);
|
|
||||||
border-radius: 6px; padding: 6px 9px; font-size: 13px; width: 100%;
|
|
||||||
font-family: var(--mono);
|
|
||||||
}
|
|
||||||
.field input:focus, .field select:focus, .log-toolbar input:focus { outline: none; border-color: var(--accent); }
|
|
||||||
.field .fhelp { font-size: 11px; color: var(--gray); }
|
|
||||||
.field.check { flex-direction: row; align-items: center; gap: 8px; }
|
|
||||||
.field.check input { width: auto; }
|
|
||||||
.field.dirty input, .field.dirty select { border-color: rgba(79,140,255,.65); }
|
|
||||||
.field.invalid input, .field.invalid select { border-color: var(--bad); }
|
|
||||||
.form-foot { display: flex; align-items: center; gap: 12px; margin-top: 14px; flex-wrap: wrap; }
|
|
||||||
.form-error { color: var(--bad); font-size: 12px; }
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 收敛 */
|
|
||||||
.converge-head { display: flex; gap: 14px; align-items: flex-start; flex-wrap: wrap; margin-bottom: 12px; }
|
|
||||||
.notes { margin: 0; padding-left: 18px; color: var(--dim); font-size: 12px; max-width: 760px; }
|
|
||||||
.notes li { margin: 2px 0; }
|
|
||||||
.metric-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(186px, 1fr)); gap: 10px; }
|
|
||||||
.metric-card {
|
|
||||||
border: 1px solid var(--line); border-radius: 10px; background: var(--panel2);
|
|
||||||
padding: 9px 11px; display: flex; flex-direction: column; gap: 5px;
|
|
||||||
}
|
|
||||||
.metric-card .mhead { display: flex; justify-content: space-between; align-items: baseline; gap: 8px; }
|
|
||||||
.metric-card .mname { font-size: 12px; color: var(--dim); }
|
|
||||||
.metric-card .mvalue { font-family: var(--mono); font-size: 17px; }
|
|
||||||
.metric-card .mfoot { display: flex; justify-content: space-between; font-size: 11px; color: var(--gray); gap: 6px; }
|
|
||||||
.metric-card canvas { width: 100%; height: 40px; display: block; }
|
|
||||||
.trend.up { color: var(--ok); } .trend.down { color: var(--warn); }
|
|
||||||
.trend.flat { color: var(--dim); } .trend.none { color: var(--gray); }
|
|
||||||
.metric-card.alert { border-color: rgba(248,113,113,.55); }
|
|
||||||
.metric-card.alert .mvalue { color: var(--bad); }
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 图表 */
|
|
||||||
.chart-toolbar { display: flex; align-items: center; gap: 14px; flex-wrap: wrap; margin: 16px 0 8px; }
|
|
||||||
.chart-tabs { display: flex; gap: 6px; flex-wrap: wrap; }
|
|
||||||
.chart-tab {
|
|
||||||
border: 1px solid var(--line); background: var(--transparent, transparent);
|
|
||||||
color: var(--dim); border-radius: 999px; padding: 4px 12px; font-size: 12px;
|
|
||||||
}
|
|
||||||
.chart-tab.active { border-color: var(--accent); color: var(--text); background: var(--panel2); }
|
|
||||||
label.inline { font-size: 12px; color: var(--dim); display: flex; gap: 6px; align-items: center; }
|
|
||||||
label.inline input[type="checkbox"] { width: auto; }
|
|
||||||
label.inline select, .log-toolbar select {
|
|
||||||
background: var(--panel2); color: var(--text); border: 1px solid var(--line);
|
|
||||||
border-radius: 6px; padding: 4px 8px; font-size: 12px;
|
|
||||||
}
|
|
||||||
.chart-stack { display: flex; flex-direction: column; gap: 14px; }
|
|
||||||
.chart-figure { margin: 0; border: 1px solid var(--line); border-radius: 10px; background: #0b0f1a; padding: 10px 12px 4px; }
|
|
||||||
.chart-figure figcaption { display: flex; gap: 10px; align-items: baseline; font-size: 12px; margin-bottom: 6px; }
|
|
||||||
.chart-figure figcaption b { color: var(--text); font-size: 13px; }
|
|
||||||
.chart-figure canvas { width: 100%; display: block; }
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 日志 */
|
|
||||||
.log-toolbar { display: flex; align-items: center; gap: 12px; flex-wrap: wrap; margin-bottom: 8px; }
|
|
||||||
.log-toolbar input[type="search"] {
|
|
||||||
background: var(--panel2); border: 1px solid var(--line); color: var(--text);
|
|
||||||
border-radius: 6px; padding: 6px 10px; font-size: 12px; min-width: 240px;
|
|
||||||
}
|
|
||||||
.log-view {
|
|
||||||
margin: 0; background: #0b0f1a; border: 1px solid var(--line); border-radius: 10px;
|
|
||||||
padding: 10px 12px; height: 320px; overflow: auto; font-family: var(--mono);
|
|
||||||
font-size: 12px; line-height: 1.55; white-space: pre-wrap; word-break: break-word;
|
|
||||||
}
|
|
||||||
.log-view .l { display: block; }
|
|
||||||
.log-view .l.err { color: var(--bad); }
|
|
||||||
.log-view .l.warn { color: var(--warn); }
|
|
||||||
.log-view .l.json { color: var(--cyan); }
|
|
||||||
.log-view .l.meta { color: var(--ok); }
|
|
||||||
.log-view .l.dim { color: var(--gray); }
|
|
||||||
.log-view mark { background: rgba(245,158,11,.28); color: inherit; border-radius: 2px; }
|
|
||||||
|
|
||||||
/* ------------------------------------------------------------ 表格 */
|
|
||||||
.table-wrap { border: 1px solid var(--line); border-radius: 10px; overflow: auto; max-height: 300px; }
|
|
||||||
table { width: 100%; border-collapse: collapse; font-size: 12px; }
|
|
||||||
th, td { text-align: left; padding: 6px 10px; border-bottom: 1px solid var(--line); white-space: nowrap; }
|
|
||||||
th { position: sticky; top: 0; background: var(--panel2); color: var(--dim); font-weight: 500; }
|
|
||||||
td { font-family: var(--mono); }
|
|
||||||
tbody tr:hover { background: rgba(79,140,255,.06); }
|
|
||||||
|
|
||||||
@media (max-width: 760px) {
|
|
||||||
.topstatus { margin-left: 0; }
|
|
||||||
.progress-row { gap: 12px; }
|
|
||||||
}
|
|
||||||
@@ -1,285 +0,0 @@
|
|||||||
/* 训练前端 JS 逻辑回归（由 pytest 经 node 调起，见 tests/test_training_ui_js.py）。
|
|
||||||
以 DOM 桩加载发布的 charts.js / app.js 原文，用服务端真实生成的示例快照
|
|
||||||
（services/demo.py 的 demo_snapshot JSON）驱动渲染，断言：
|
|
||||||
1) 表单由 schema 生成、字段数与后端契约一致；
|
|
||||||
2) 指标卡/曲线/日志/逐回合表都拿到数据并渲染，无异常抛出；
|
|
||||||
3) 三种收敛场景的判读文案正确落到顶部 chip；
|
|
||||||
4) 增量指标游标合并不丢点也不重点（曾是该前端的典型 bug）；
|
|
||||||
5) charts.js 的刻度与格式函数在极端输入下不抛异常。
|
|
||||||
曾回归出两个"加载即白屏"隐患：job.process 缺失时直接取下标、以及游标
|
|
||||||
与 metric_limit 同时生效时重复追加同批进度点。本文件防其复发。 */
|
|
||||||
"use strict";
|
|
||||||
const fs = require("fs");
|
|
||||||
const path = require("path");
|
|
||||||
|
|
||||||
const REPO = process.env.REPO;
|
|
||||||
if (!REPO) throw new Error("REPO env required");
|
|
||||||
const STATIC = path.join(REPO, "src", "mcm26b", "training_ui", "static");
|
|
||||||
const DEMO = process.env.DEMO_STATE;
|
|
||||||
if (!DEMO) throw new Error("DEMO_STATE env required");
|
|
||||||
|
|
||||||
const scenarios = JSON.parse(fs.readFileSync(DEMO, "utf8"));
|
|
||||||
const schema = JSON.parse(fs.readFileSync(process.env.SCHEMA, "utf8"));
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- DOM 桩
|
|
||||||
/* className 是唯一真源：代码里会直接赋值 className（见 chip()），也会用
|
|
||||||
classList.add/toggle；如果 classList 自带一个独立 Set，两者就会不同步，
|
|
||||||
测试会误报。这里让 classList 每次读写都落回 className。 */
|
|
||||||
function makeClassList(el) {
|
|
||||||
const read = () => String(el.className || "").split(/\s+/).filter(Boolean);
|
|
||||||
const write = (list) => { el.className = [...new Set(list)].join(" "); };
|
|
||||||
return {
|
|
||||||
add: (...c) => write(read().concat(c)),
|
|
||||||
remove: (...c) => write(read().filter((x) => !c.includes(x))),
|
|
||||||
toggle: (c, force) => {
|
|
||||||
const on = force === undefined ? !read().includes(c) : !!force;
|
|
||||||
write(on ? read().concat(c) : read().filter((x) => x !== c));
|
|
||||||
return on;
|
|
||||||
},
|
|
||||||
contains: (c) => read().includes(c),
|
|
||||||
};
|
|
||||||
}
|
|
||||||
const ctxStub = new Proxy({}, {
|
|
||||||
get(t, prop) {
|
|
||||||
if (!(prop in t)) {
|
|
||||||
t[prop] = typeof prop === "string" &&
|
|
||||||
/^(arc|beginPath|closePath|fill|stroke|moveTo|lineTo|fillText|strokeText|strokeRect|fillRect|clearRect|setLineDash|setTransform|save|restore|roundRect|measureText|scale|translate)$/.test(prop)
|
|
||||||
? (prop === "measureText" ? () => ({ width: 40 }) : () => {})
|
|
||||||
: function () {};
|
|
||||||
}
|
|
||||||
return t[prop];
|
|
||||||
},
|
|
||||||
set(t, prop, v) { t[prop] = v; return true; },
|
|
||||||
});
|
|
||||||
const allElements = [];
|
|
||||||
/* querySelectorAll 桩：支持 app.js 实际用到的选择器
|
|
||||||
[data-node="x"] / [data-link="n"] → 时间线节点与连接线
|
|
||||||
.chart-tab → 曲线页签（用于切换 active）
|
|
||||||
tbody → 逐回合表体
|
|
||||||
子元素按 tag 记账，因此 page.querySelector("tbody") 每次返回同一个对象。 */
|
|
||||||
function matchesSelector(el, sel) {
|
|
||||||
const attr = /^\[data-([a-z]+)="?([^"\]]*)"?\]$/.exec(sel);
|
|
||||||
if (attr) return el.dataset && String(el.dataset[attr[1]]) === attr[2];
|
|
||||||
if (sel.startsWith(".")) return String(el.className || "").split(/\s+/).includes(sel.slice(1));
|
|
||||||
if (el.tag === sel) return true;
|
|
||||||
const byTag = el.byTag && el.byTag[sel];
|
|
||||||
return Boolean(byTag);
|
|
||||||
}
|
|
||||||
function makeEl(id, tag) {
|
|
||||||
const el = {
|
|
||||||
id, tag: tag || id, textContent: "", innerHTML: "", value: "", checked: true,
|
|
||||||
disabled: false, className: "",
|
|
||||||
files: [], width: 600, height: 240, style: {}, handlers: {}, children: [],
|
|
||||||
byTag: {}, dataset: {}, scrollTop: 0, scrollHeight: 100, clientHeight: 100,
|
|
||||||
get classList() { return makeClassList(this); },
|
|
||||||
addEventListener(type, fn) { (this.handlers[type] ||= []).push(fn); },
|
|
||||||
removeEventListener() {}, click() {}, matches() { return false; },
|
|
||||||
appendChild(child) { this.children.push(child); return child; },
|
|
||||||
removeChild(child) { this.children = this.children.filter((c) => c !== child); },
|
|
||||||
setAttribute() {}, focus() {},
|
|
||||||
querySelector(sel) { return this.querySelectorAll(sel)[0] || this.childFor(sel); },
|
|
||||||
// 表单里的静态子元素（如 <tbody>）不在 children 里，按需惰性创建并缓存
|
|
||||||
childFor(sel) {
|
|
||||||
if (!this.byTag[sel]) this.byTag[sel] = makeEl(`${id}:${sel}`, sel);
|
|
||||||
return this.byTag[sel];
|
|
||||||
},
|
|
||||||
querySelectorAll(sel) {
|
|
||||||
const found = [];
|
|
||||||
const walk = (node) => {
|
|
||||||
for (const child of node.children || []) {
|
|
||||||
if (matchesSelector(child, sel)) found.push(child);
|
|
||||||
walk(child);
|
|
||||||
}
|
|
||||||
};
|
|
||||||
walk(this);
|
|
||||||
return found;
|
|
||||||
},
|
|
||||||
getContext() { return ctxStub; },
|
|
||||||
getBoundingClientRect() { return { left: 0, top: 0, width: 600, height: 240 }; },
|
|
||||||
};
|
|
||||||
allElements.push(el);
|
|
||||||
return el;
|
|
||||||
}
|
|
||||||
const els = new Map();
|
|
||||||
const $get = (id) => { if (!els.has(id)) els.set(id, makeEl(id)); return els.get(id); };
|
|
||||||
// 几个会被读取初值的控件（否则 value 为空字符串，Number("") === 0）
|
|
||||||
for (const [id, value] of [["log-tail", "400"], ["chart-window", "120"]]) $get(id).value = value;
|
|
||||||
$get("chart-cursor").checked = true;
|
|
||||||
$get("log-follow").checked = true;
|
|
||||||
$get("log-json").checked = true;
|
|
||||||
$get("log-errors-only").checked = false; // 只保留"显示进度 JSON"这一项过滤
|
|
||||||
$get("preview-scenario").value = "rising";
|
|
||||||
|
|
||||||
globalThis.requestAnimationFrame = (cb) => { cb(0); return 1; };
|
|
||||||
globalThis.cancelAnimationFrame = () => {};
|
|
||||||
globalThis.performance = { now: () => Date.now() };
|
|
||||||
globalThis.setInterval = () => 0;
|
|
||||||
globalThis.location = { port: "8036" };
|
|
||||||
globalThis.confirm = () => true;
|
|
||||||
globalThis.alert = () => {};
|
|
||||||
globalThis.addEventListener = () => {};
|
|
||||||
globalThis.devicePixelRatio = 2;
|
|
||||||
globalThis.window = globalThis;
|
|
||||||
globalThis.document = {
|
|
||||||
getElementById: $get,
|
|
||||||
createElement: (tag) => makeEl(tag),
|
|
||||||
addEventListener() {},
|
|
||||||
};
|
|
||||||
// 元素挂载计数：断言"指标卡确实被创建"而不是静默空转
|
|
||||||
let createdCount = 0;
|
|
||||||
const realCreate = globalThis.document.createElement;
|
|
||||||
globalThis.document.createElement = (tag) => { createdCount++; return realCreate(tag); };
|
|
||||||
|
|
||||||
globalThis.fetch = async (url) => {
|
|
||||||
const json = async () => {
|
|
||||||
if (url.startsWith("/api/train/schema")) return schema;
|
|
||||||
if (url.includes("state")) {
|
|
||||||
const match = /scenario=([a-z-]+)/.exec(url);
|
|
||||||
const key = match ? match[1] : "rising";
|
|
||||||
return { job: scenarios[key] };
|
|
||||||
}
|
|
||||||
throw new Error("unexpected fetch " + url);
|
|
||||||
};
|
|
||||||
return { ok: true, status: 200, json };
|
|
||||||
};
|
|
||||||
|
|
||||||
// ---------------------------------------------------------------- 装载发布代码
|
|
||||||
const chartsSrc = fs.readFileSync(path.join(STATIC, "charts.js"), "utf8");
|
|
||||||
const appSrc = fs.readFileSync(path.join(STATIC, "app.js"), "utf8");
|
|
||||||
new Function(chartsSrc)();
|
|
||||||
const appApi = new Function(
|
|
||||||
appSrc + "\n;return { seriesOf, classify, highlight, absorbMetrics, state, render, fmtNumber, fmtDuration };"
|
|
||||||
)();
|
|
||||||
// app.js 末尾会自动 boot()，这里等它把 schema/首次快照渲染完
|
|
||||||
const flush = async () => { for (let i = 0; i < 40; i++) await new Promise((r) => setImmediate(r)); };
|
|
||||||
function assert(cond, message) { if (!cond) throw new Error(message); }
|
|
||||||
function stripTags(html) { return String(html).replace(/<[^>]*>/g, ""); }
|
|
||||||
|
|
||||||
(async () => {
|
|
||||||
await flush();
|
|
||||||
|
|
||||||
// 1) schema 驱动表单
|
|
||||||
const groups = $get("form-groups").children;
|
|
||||||
assert(groups.length === schema.groups.length,
|
|
||||||
`表单分组数应为 ${schema.groups.length}，实际 ${groups.length}`);
|
|
||||||
const cfgField = $get("cfg-timesteps");
|
|
||||||
assert(cfgField && String(cfgField.value) === String(schema.fields.find((f) => f.key === "timesteps").default),
|
|
||||||
`默认值未写入表单: ${cfgField && cfgField.value}`);
|
|
||||||
assert($get("presets").children.length === schema.presets.length, "预设按钮数不匹配");
|
|
||||||
|
|
||||||
// 2) 首帧（rising 场景）渲染
|
|
||||||
const state = appApi.state;
|
|
||||||
assert(state.metrics.length > 0, "首次快照后本地进度点为空");
|
|
||||||
const chips = $get("chip-converge").innerHTML;
|
|
||||||
assert(chips.includes(scenarios.rising.convergence.status),
|
|
||||||
`收敛 chip 未渲染: ${chips}`);
|
|
||||||
assert($get("stat-pid").textContent === "PID 42424", `PID 未渲染: ${$get("stat-pid").textContent}`);
|
|
||||||
assert($get("metric-grid").children.length === schema.metrics.length,
|
|
||||||
`指标卡数量应为 ${schema.metrics.length}，实际 ${$get("metric-grid").children.length}`);
|
|
||||||
assert($get("chart-tabs").children.length === schema.metrics.length,
|
|
||||||
"曲线页签数量与指标字典不一致");
|
|
||||||
assert(stripTags($get("log-view").innerHTML).includes("progress"),
|
|
||||||
"日志区没有渲染进度行");
|
|
||||||
assert($get("episode-table").querySelector("tbody") !== null, "逐回合表缺少 tbody");
|
|
||||||
const tbody = $get("episode-table").querySelector("tbody");
|
|
||||||
assert(tbody.innerHTML.includes("<tr>"),
|
|
||||||
`逐回合表未渲染任何行: monitor=${(scenarios.rising.monitor || []).length}`);
|
|
||||||
assert(Number($get("steps-total").textContent.replace(/,/g, "")) === scenarios.rising.config.timesteps,
|
|
||||||
`总步数渲染异常: ${$get("steps-total").textContent}`);
|
|
||||||
|
|
||||||
// 3) 三种收敛场景切到对应 chip
|
|
||||||
const tones = { rising: "ok", plateau: "ok", "kl-spike": "bad" };
|
|
||||||
for (const key of Object.keys(scenarios)) {
|
|
||||||
const rendered = appApi.render(scenarios[key]);
|
|
||||||
void rendered;
|
|
||||||
const html = $get("chip-converge").innerHTML;
|
|
||||||
assert(html.includes(scenarios[key].convergence.status),
|
|
||||||
`场景 ${key} 的判读未落到 chip: ${html}`);
|
|
||||||
assert($get("chip-converge").classList.contains(tones[key]),
|
|
||||||
`场景 ${key} 的色调应为 ${tones[key]}`);
|
|
||||||
}
|
|
||||||
assert(String($get("chart2-title").textContent).length > 0, "第二张曲线标题为空");
|
|
||||||
|
|
||||||
// 4) 增量游标合并：续接 / 有缺口 / 过期批次 三条分支
|
|
||||||
const job = JSON.parse(JSON.stringify(scenarios.rising));
|
|
||||||
const total = job.metrics.length;
|
|
||||||
const reset = () => {
|
|
||||||
appApi.state.metrics = [];
|
|
||||||
appApi.state.metricCache = [];
|
|
||||||
appApi.state.metricNext = 0;
|
|
||||||
appApi.state.metricTotal = 0;
|
|
||||||
};
|
|
||||||
const feed = (metrics, offset) => appApi.absorbMetrics(
|
|
||||||
{ ...job, metrics, metric_offset: offset, metric_total: total });
|
|
||||||
const stamp = (i) => job.metrics[i].timesteps;
|
|
||||||
|
|
||||||
reset();
|
|
||||||
feed(job.metrics.slice(0, 10), 0);
|
|
||||||
assert(appApi.state.metrics.length === 10, `首批应 10 点，实际 ${appApi.state.metrics.length}`);
|
|
||||||
feed(job.metrics.slice(10, 14), 10);
|
|
||||||
assert(appApi.state.metrics.length === 14, `续接后应 14 点，实际 ${appApi.state.metrics.length}`);
|
|
||||||
assert(appApi.state.metricNext === 14, `metricNext 应为 14，实际 ${appApi.state.metricNext}`);
|
|
||||||
feed([], 14);
|
|
||||||
assert(appApi.state.metrics.length === 14, `无新点不得改变长度，实际 ${appApi.state.metrics.length}`);
|
|
||||||
|
|
||||||
// 有缺口（服务端窗口前移）：整体替换，避免曲线出现空洞
|
|
||||||
feed(job.metrics.slice(20, 30), 20);
|
|
||||||
assert(appApi.state.metrics.length === 10, `有缺口时应替换为 10 点，实际 ${appApi.state.metrics.length}`);
|
|
||||||
assert(appApi.state.metrics[0].timesteps === stamp(20), "替换后起点错位");
|
|
||||||
assert(appApi.state.metricNext === 30, `游标未随替换更新: ${appApi.state.metricNext}`);
|
|
||||||
|
|
||||||
// 过期批次（服务端计数回退）：忽略，本地不动
|
|
||||||
feed(job.metrics.slice(0, 5), 0);
|
|
||||||
assert(appApi.state.metrics.length === 10, `过期批次应被忽略，实际 ${appApi.state.metrics.length}`);
|
|
||||||
assert(appApi.state.metrics[0].timesteps === stamp(20), "过期批次污染了本地缓存");
|
|
||||||
|
|
||||||
// 字符串形式的 offset：必须按数值比较，不得退化成字典序
|
|
||||||
reset();
|
|
||||||
feed(job.metrics.slice(0, 10), "0");
|
|
||||||
feed(job.metrics.slice(10, 14), "10");
|
|
||||||
assert(appApi.state.metrics.length === 14,
|
|
||||||
`字符串 offset 应正常续接为 14 点，实际 ${appApi.state.metrics.length}`);
|
|
||||||
|
|
||||||
// 缺失 offset（旧版服务端）：退化为按到达顺序追加，不得丢点
|
|
||||||
reset();
|
|
||||||
appApi.absorbMetrics({ ...job, metrics: job.metrics.slice(0, 6) });
|
|
||||||
assert(appApi.state.metrics.length === 6, `缺 offset 时首批应 6 点，实际 ${appApi.state.metrics.length}`);
|
|
||||||
|
|
||||||
// 5) 日志分类与高亮
|
|
||||||
assert(appApi.classify("Traceback (most recent call last):") === "err", "Traceback 未归入错误行");
|
|
||||||
assert(appApi.classify("UserWarning: x") === "warn", "警告未归入 warn");
|
|
||||||
assert(appApi.classify('{"event": "progress"}') === "json", "进度 JSON 未归入 json");
|
|
||||||
assert(appApi.highlight("abc", "b").includes("<mark>b</mark>"), "过滤高亮未生效");
|
|
||||||
const marked = appApi.highlight("<script>", "script");
|
|
||||||
assert(marked.startsWith("<") && marked.endsWith(">") && marked.includes("<mark>"),
|
|
||||||
`命中过滤时也必须转义 HTML: ${marked}`);
|
|
||||||
assert(!appApi.highlight("<script>", "").includes("<script>"),
|
|
||||||
"未过滤时也应转义 HTML");
|
|
||||||
|
|
||||||
// 6) 序列抽取：跳过 None 且横轴用 timesteps
|
|
||||||
const series = appApi.seriesOf("ep_rew_mean");
|
|
||||||
assert(series.length > 0 && isFinite(series[0][0]) && isFinite(series[0][1]),
|
|
||||||
"seriesOf 产出非法坐标");
|
|
||||||
for (let i = 1; i < series.length; i++) {
|
|
||||||
assert(series[i][0] >= series[i - 1][0], "seriesOf 横轴未单调递增");
|
|
||||||
}
|
|
||||||
|
|
||||||
// 7) charts.js 边界输入不抛异常
|
|
||||||
const Charts = globalThis.Charts;
|
|
||||||
const canvas = makeEl("probe");
|
|
||||||
Charts.line(canvas, { series: [] });
|
|
||||||
Charts.line(canvas, { series: [{ points: [[1, 1]] }] });
|
|
||||||
Charts.line(canvas, { series: [{ points: [[1, 1], [1, 1]] }] });
|
|
||||||
Charts.line(canvas, { series: [{ points: [[0, 1e-12], [1, 1e12]] }] });
|
|
||||||
Charts.sparkline(canvas, []);
|
|
||||||
Charts.sparkline(canvas, [3, 3, 3, 3]);
|
|
||||||
assert(Charts.fmt(NaN) === "—", "fmt(NaN) 应为占位符");
|
|
||||||
assert(Charts.fmt(12345) === "12345.0", `五位数应保持定点: ${Charts.fmt(12345)}`);
|
|
||||||
assert(Charts.fmt(1.2e6).includes("e+6"), `六位数应转科学计数: ${Charts.fmt(1.2e6)}`);
|
|
||||||
assert(Charts.fmt(0.000012) === "1.20e-5", `极小值应转科学计数: ${Charts.fmt(0.000012)}`);
|
|
||||||
|
|
||||||
console.log("training-ui harness: ALL PASS");
|
|
||||||
})().catch((err) => {
|
|
||||||
console.error("training-ui harness FAILED:", err && err.stack || err);
|
|
||||||
process.exit(1);
|
|
||||||
});
|
|
||||||
@@ -1,286 +0,0 @@
|
|||||||
"""HRL 验收：槽位布局、观测/掩码同源性、整场环境闭环与奖励恒等式。
|
|
||||||
|
|
||||||
HRL 依赖属于可选组 [dependency-groups] rl；未安装 numpy/gymnasium 时
|
|
||||||
全部跳过（运行闭环不依赖它们）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
import math
|
|
||||||
from types import SimpleNamespace
|
|
||||||
import unittest
|
|
||||||
|
|
||||||
pytest = None
|
|
||||||
try:
|
|
||||||
import pytest
|
|
||||||
numpy = pytest.importorskip("numpy")
|
|
||||||
pytest.importorskip("gymnasium")
|
|
||||||
except Exception: # pragma: no cover - 无 RL 依赖环境
|
|
||||||
numpy = None
|
|
||||||
|
|
||||||
if numpy is not None:
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.hrl.constants import (CHANNEL_COUNT, CLEAR_OFFSET, CLEAR_SLOTS,
|
|
||||||
FALLBACK_OFFSET, FALLBACK_SLOTS, HIGH_GLOBAL_DIM,
|
|
||||||
HIGH_OBS_DIM, LOCATE_OFFSET,
|
|
||||||
LOCATE_SLOTS,
|
|
||||||
N_CANDIDATES, PER_CANDIDATE_DIM,
|
|
||||||
PER_CHANNEL_DIM, STATION_OFFSET, STATION_SLOTS)
|
|
||||||
from mcm26b.hrl.curriculum import STAGES, make_scene, training_config
|
|
||||||
from mcm26b.hrl.executor import LowLevelExecutor, build_low_observation
|
|
||||||
from mcm26b.hrl.hierarchical_env import HierarchicalEnv
|
|
||||||
from mcm26b.hrl.low_level_env import LowLevelEnv
|
|
||||||
from mcm26b.hrl.scheduler import HRLControllerMixin
|
|
||||||
|
|
||||||
CONFIG = ProblemConfig()
|
|
||||||
|
|
||||||
|
|
||||||
def _absent(info: dict) -> int:
|
|
||||||
return info.get("absent_proven", 0)
|
|
||||||
|
|
||||||
|
|
||||||
def _env(problem: int = 3, stage_index: int = 0, **kwargs) -> HierarchicalEnv:
|
|
||||||
return HierarchicalEnv(problem=problem, stage=STAGES[stage_index], **kwargs)
|
|
||||||
|
|
||||||
|
|
||||||
if numpy is not None:
|
|
||||||
class _CandidateHost(HRLControllerMixin):
|
|
||||||
"""只提供候选所需感知原语，槽位分配和掩码均调用生产实现。"""
|
|
||||||
|
|
||||||
def __init__(self, detected, certified=(), blocked=()):
|
|
||||||
self.config = CONFIG
|
|
||||||
self.problem = 4 if blocked else 3
|
|
||||||
self._certified = set(certified)
|
|
||||||
self._clear_failed = set()
|
|
||||||
self.blocked_probes = {
|
|
||||||
channel: [(0.0, 0.0)] * CONFIG.search.max_blocked_probes
|
|
||||||
for channel in blocked
|
|
||||||
}
|
|
||||||
region = SimpleNamespace(
|
|
||||||
polygon=[(1.0, 1.0), (9.0, 1.0), (9.0, 9.0), (1.0, 9.0)],
|
|
||||||
mec=lambda: ((5.0, 5.0), math.sqrt(32.0)))
|
|
||||||
self.belief = SimpleNamespace(
|
|
||||||
channels_by_status=lambda status: list(detected),
|
|
||||||
candidate_set=lambda channel: region)
|
|
||||||
self.ledger = SimpleNamespace(points=[])
|
|
||||||
|
|
||||||
def _best_certificate(self, channel):
|
|
||||||
return SimpleNamespace(certified=channel in self._certified,
|
|
||||||
center=(float(channel), 0.0))
|
|
||||||
|
|
||||||
def _select_point(self, channel):
|
|
||||||
return (float(channel), 1.0)
|
|
||||||
|
|
||||||
def _hrl_probe_point(self, region, channel):
|
|
||||||
return (float(channel), 2.0)
|
|
||||||
|
|
||||||
|
|
||||||
@unittest.skipIf(numpy is None, "RL 依赖组未安装")
|
|
||||||
class LayoutTests(unittest.TestCase):
|
|
||||||
def test_high_obs_dim_matches_component_contract(self):
|
|
||||||
expected = (HIGH_GLOBAL_DIM + CHANNEL_COUNT * PER_CHANNEL_DIM
|
|
||||||
+ N_CANDIDATES * PER_CANDIDATE_DIM)
|
|
||||||
self.assertEqual(HIGH_OBS_DIM, expected)
|
|
||||||
self.assertEqual(HIGH_OBS_DIM, 1248)
|
|
||||||
self.assertEqual(N_CANDIDATES, 120)
|
|
||||||
|
|
||||||
def test_observation_finite_and_deterministic(self):
|
|
||||||
env = _env(problem=3)
|
|
||||||
obs, _ = env.reset(seed=2026)
|
|
||||||
self.assertEqual(obs.shape, (HIGH_OBS_DIM,))
|
|
||||||
self.assertTrue(numpy.isfinite(obs).all())
|
|
||||||
again = env._loop.hrl_observation(env._current_candidates)
|
|
||||||
self.assertTrue(numpy.array_equal(obs, again))
|
|
||||||
|
|
||||||
def test_mask_nonempty_and_matches_candidate_slots(self):
|
|
||||||
env = _env(problem=3)
|
|
||||||
env.reset(seed=7)
|
|
||||||
for _ in range(6):
|
|
||||||
mask = env.action_masks()
|
|
||||||
self.assertTrue(mask.any())
|
|
||||||
slots = {cand.slot for cand in env._current_candidates}
|
|
||||||
self.assertEqual({int(i) for i in numpy.flatnonzero(mask)}, slots)
|
|
||||||
_obs, _r, term, trunc, _info = env.step(
|
|
||||||
int(env._loop.heuristic_action(env._current_candidates)))
|
|
||||||
if term or trunc:
|
|
||||||
break
|
|
||||||
|
|
||||||
def test_q4_probe_and_fallback_slots_exist_in_layout(self):
|
|
||||||
self.assertEqual(CLEAR_SLOTS, CHANNEL_COUNT)
|
|
||||||
self.assertEqual(LOCATE_SLOTS, CHANNEL_COUNT)
|
|
||||||
self.assertEqual(FALLBACK_SLOTS, CHANNEL_COUNT)
|
|
||||||
self.assertEqual(STATION_SLOTS, 49)
|
|
||||||
self.assertEqual(CLEAR_OFFSET, 0)
|
|
||||||
self.assertEqual(LOCATE_OFFSET, CLEAR_OFFSET + CLEAR_SLOTS)
|
|
||||||
self.assertEqual(STATION_OFFSET, LOCATE_OFFSET + LOCATE_SLOTS)
|
|
||||||
self.assertEqual(FALLBACK_OFFSET, STATION_OFFSET + STATION_SLOTS)
|
|
||||||
self.assertEqual(N_CANDIDATES - (FALLBACK_OFFSET + FALLBACK_SLOTS), 11)
|
|
||||||
|
|
||||||
def test_build_candidates_keeps_clear_and_locate(self):
|
|
||||||
host = _CandidateHost(detected=[1, 2, 3, 4, 17, 18, 19, 20],
|
|
||||||
certified=[17, 18, 19, 20])
|
|
||||||
candidates = host.build_candidates()
|
|
||||||
expected = {(CLEAR_OFFSET + channel - 1, "clear", channel)
|
|
||||||
for channel in range(17, 21)}
|
|
||||||
expected.update((LOCATE_OFFSET + channel - 1, "locate", channel)
|
|
||||||
for channel in range(1, 5))
|
|
||||||
self.assertEqual({(c.slot, c.kind, c.channel) for c in candidates}, expected)
|
|
||||||
self.assertEqual(len(candidates), 8)
|
|
||||||
mask = host.hrl_action_mask(candidates)
|
|
||||||
self.assertEqual(set(numpy.flatnonzero(mask)), {c.slot for c in candidates})
|
|
||||||
for channel in range(17, 21):
|
|
||||||
selected = host.candidate_by_slot(candidates, CLEAR_OFFSET + channel - 1)
|
|
||||||
self.assertEqual((selected.kind, selected.channel), ("clear", channel))
|
|
||||||
|
|
||||||
def test_build_candidates_fallback_covers_slots(self):
|
|
||||||
channels = list(range(17, 21))
|
|
||||||
host = _CandidateHost(detected=channels, blocked=channels)
|
|
||||||
candidates = host.build_candidates()
|
|
||||||
fallbacks = [c for c in candidates if c.kind == "fallback"]
|
|
||||||
self.assertEqual({c.channel for c in fallbacks}, set(channels))
|
|
||||||
self.assertEqual(len(candidates), 8) # 四个 probe 与四个 fallback 并存
|
|
||||||
for candidate in fallbacks:
|
|
||||||
self.assertEqual(candidate.slot, FALLBACK_OFFSET + candidate.channel - 1)
|
|
||||||
self.assertGreaterEqual(candidate.slot, FALLBACK_OFFSET)
|
|
||||||
self.assertLess(candidate.slot, FALLBACK_OFFSET + FALLBACK_SLOTS)
|
|
||||||
self.assertGreater(candidate.targets, 0)
|
|
||||||
mask = host.hrl_action_mask(candidates)
|
|
||||||
self.assertTrue(mask[FALLBACK_OFFSET + 19])
|
|
||||||
self.assertFalse(mask[FALLBACK_OFFSET + FALLBACK_SLOTS:].any())
|
|
||||||
|
|
||||||
|
|
||||||
@unittest.skipIf(numpy is None, "RL 依赖组未安装")
|
|
||||||
class HeuristicLoopTests(unittest.TestCase):
|
|
||||||
def test_q3_full_episode_clears_all_sources(self):
|
|
||||||
env = _env(problem=3, stage_index=3, max_episode_steps=500)
|
|
||||||
_, _ = env.reset(seed=11)
|
|
||||||
steps = 0
|
|
||||||
while True:
|
|
||||||
action = env._loop.heuristic_action(env._current_candidates)
|
|
||||||
_, _, term, trunc, info = env.step(action)
|
|
||||||
steps += 1
|
|
||||||
if term or trunc:
|
|
||||||
break
|
|
||||||
self.assertTrue(term)
|
|
||||||
self.assertGreaterEqual(info["cleared"], 10)
|
|
||||||
self.assertEqual(info["cleared"] + _absent(info), 20)
|
|
||||||
self.assertEqual(info["accounted"], 20)
|
|
||||||
self.assertEqual(info["anomalies"], [])
|
|
||||||
|
|
||||||
def test_q4_episode_accounts_all_channels(self):
|
|
||||||
env = _env(problem=4, stage_index=3, max_episode_steps=600)
|
|
||||||
env.reset(seed=21)
|
|
||||||
while True:
|
|
||||||
action = env._loop.heuristic_action(env._current_candidates)
|
|
||||||
_, _, term, trunc, info = env.step(action)
|
|
||||||
if term or trunc:
|
|
||||||
break
|
|
||||||
self.assertTrue(term)
|
|
||||||
self.assertEqual(info["accounted"], 20)
|
|
||||||
self.assertEqual(info["anomalies"], [])
|
|
||||||
self.assertGreaterEqual(info["cleared"], 10)
|
|
||||||
|
|
||||||
def test_reward_identity_from_outcome_terms(self):
|
|
||||||
"""非终止步奖励恒等于各分项加权和（prompt §5.2 的可折算实现）。"""
|
|
||||||
env = _env(problem=3, stage_index=1, max_episode_steps=50)
|
|
||||||
env.reset(seed=5)
|
|
||||||
reward_cfg = env.reward
|
|
||||||
for _ in range(8):
|
|
||||||
action = env._loop.heuristic_action(env._current_candidates)
|
|
||||||
_virtual_before = env._client.state.virtual_time_s
|
|
||||||
_, r, term, trunc, info = env.step(action)
|
|
||||||
outcome = env._last_outcome
|
|
||||||
if outcome.invalid:
|
|
||||||
self.assertAlmostEqual(r, -reward_cfg.invalid, places=6)
|
|
||||||
else:
|
|
||||||
expected = (-outcome.delta_virtual_s / reward_cfg.time_scale
|
|
||||||
+ reward_cfg.clear * len(outcome.clear_success)
|
|
||||||
+ reward_cfg.detect * len(outcome.new_detections)
|
|
||||||
+ reward_cfg.info_weight
|
|
||||||
* sum(outcome.info_gains.values()))
|
|
||||||
if outcome.redundant:
|
|
||||||
expected -= reward_cfg.redundant
|
|
||||||
if term:
|
|
||||||
expected += (reward_cfg.terminal_accounted
|
|
||||||
if info["accounted"] == 20
|
|
||||||
else -reward_cfg.terminal_unresolved)
|
|
||||||
self.assertAlmostEqual(r, expected, places=6)
|
|
||||||
if term or trunc:
|
|
||||||
break
|
|
||||||
|
|
||||||
def test_invalid_action_penalized_without_sim_call(self):
|
|
||||||
env = _env(problem=3, stage_index=0, max_episode_steps=50)
|
|
||||||
env.reset(seed=3)
|
|
||||||
mask = env.action_masks()
|
|
||||||
invalid_action = int(numpy.argmin(mask))
|
|
||||||
virtual_before = env._client.state.virtual_time_s
|
|
||||||
_, r, _, _, _ = env.step(invalid_action)
|
|
||||||
self.assertAlmostEqual(r, -env.reward.invalid, places=6)
|
|
||||||
self.assertAlmostEqual(env._client.state.virtual_time_s,
|
|
||||||
virtual_before, places=6)
|
|
||||||
|
|
||||||
|
|
||||||
@unittest.skipIf(numpy is None, "RL 依赖组未安装")
|
|
||||||
class LowLevelTests(unittest.TestCase):
|
|
||||||
def test_low_env_shapes_and_expert_reaches_goal(self):
|
|
||||||
env = LowLevelEnv(seed=99)
|
|
||||||
obs, info = env.reset(seed=99)
|
|
||||||
self.assertEqual(obs.shape, (10,))
|
|
||||||
start = tuple(info["position"])
|
|
||||||
goal = tuple(info["goal"])
|
|
||||||
total = 0
|
|
||||||
for _ in range(60):
|
|
||||||
obs, _, term, trunc, _ = env.step(env.expert_action())
|
|
||||||
total += 1
|
|
||||||
if term or trunc:
|
|
||||||
break
|
|
||||||
self.assertTrue(term, "直线专家应在步数上限内到达")
|
|
||||||
straight = math.dist(start, goal)
|
|
||||||
efficiency = straight / (total * env.step_m)
|
|
||||||
self.assertGreater(efficiency, 0.7, "专家路径效率应接近直线")
|
|
||||||
|
|
||||||
def test_executor_straightline_is_goal_only(self):
|
|
||||||
executor = LowLevelExecutor(policy=None)
|
|
||||||
self.assertEqual(executor.roll((0.0, 0.0), (500.0, 200.0)),
|
|
||||||
[(500.0, 200.0)])
|
|
||||||
|
|
||||||
def test_low_observation_source_shared(self):
|
|
||||||
obs = build_low_observation((0.0, 0.0), (300.0, 0.0), None, 0)
|
|
||||||
self.assertEqual(obs.shape, (10,))
|
|
||||||
self.assertAlmostEqual(float(obs[4]), 1.0, places=6) # cos(bearing)=1
|
|
||||||
|
|
||||||
|
|
||||||
@unittest.skipIf(numpy is None, "RL 依赖组未安装")
|
|
||||||
class CurriculumTests(unittest.TestCase):
|
|
||||||
def test_stages_are_monotonic_in_difficulty(self):
|
|
||||||
for prev, nxt in zip(STAGES, STAGES[1:]):
|
|
||||||
self.assertLessEqual(prev.sources_max, nxt.sources_max)
|
|
||||||
self.assertLessEqual(prev.channel_max, nxt.channel_max)
|
|
||||||
self.assertLessEqual(prev.bearing_error_half_deg,
|
|
||||||
nxt.bearing_error_half_deg)
|
|
||||||
|
|
||||||
def test_training_config_defaults_to_production(self):
|
|
||||||
config = training_config(CONFIG, STAGES[3])
|
|
||||||
self.assertEqual(config.field.channel_max, 20)
|
|
||||||
self.assertEqual(config.field.bearing_error_half_deg, 1.0)
|
|
||||||
self.assertEqual(config.bayes, CONFIG.bayes)
|
|
||||||
|
|
||||||
def test_fast_config_preserves_certificate_grid(self):
|
|
||||||
config = training_config(CONFIG, STAGES[3], fidelity="fast")
|
|
||||||
self.assertEqual(config.bayes.particle_spacing_m,
|
|
||||||
CONFIG.bayes.particle_spacing_m)
|
|
||||||
self.assertLess(config.bayes.particle_spacing_m * math.sqrt(2) / 2,
|
|
||||||
config.field.clear_radius_m)
|
|
||||||
self.assertLessEqual(config.bayes.score_subsample,
|
|
||||||
CONFIG.bayes.score_subsample)
|
|
||||||
|
|
||||||
def test_scene_source_count_and_channel_bounds(self):
|
|
||||||
from random import Random
|
|
||||||
rng = Random(1)
|
|
||||||
sources, channels = make_scene(rng, training_config(CONFIG, STAGES[1]),
|
|
||||||
3, STAGES[1])
|
|
||||||
self.assertEqual(len(sources), STAGES[1].sources_max)
|
|
||||||
self.assertTrue(all(1 <= c <= STAGES[1].channel_max for c in channels))
|
|
||||||
self.assertEqual(len(set(channels)), len(channels))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__": # pragma: no cover
|
|
||||||
unittest.main()
|
|
||||||
@@ -1,444 +0,0 @@
|
|||||||
"""高层训练落盘保证：checkpoint、控制文件优雅停止、信号兜底、worker 信号屏蔽。
|
|
||||||
|
|
||||||
背景：训练前端的「停止」以前直接向进程组发 SIGTERM，而旧脚本只在 ``learn()``
|
|
||||||
正常返回后才保存，导致"停一下就全丢"；在 Windows 上 ``os.killpg`` 更是直接
|
|
||||||
``AttributeError``。现在停止/暂停走**控制文件**（跨平台），信号只作兜底。这里把
|
|
||||||
保证固定成回归测试：
|
|
||||||
1. 定期 checkpoint 可续训（zip + 同名元数据）；
|
|
||||||
2. 控制文件发出 stop 后仍能保存最终模型与元数据（Windows 同样成立）；
|
|
||||||
3. POSIX 下进程组 SIGTERM 也能优雅收尾；
|
|
||||||
4. subproc worker 忽略停止信号，不抢在主进程收尾之前退出。
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import importlib.util
|
|
||||||
import json
|
|
||||||
import multiprocessing
|
|
||||||
import os
|
|
||||||
import signal
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
import threading
|
|
||||||
import time
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
from mcm26b.process_control import spawn_kwargs, terminate_process_tree, write_control
|
|
||||||
|
|
||||||
|
|
||||||
def _train_module():
|
|
||||||
"""按路径加载训练脚本（不触发 torch 导入，torch 只在函数内部惰性导入）。"""
|
|
||||||
path = Path(__file__).parents[1] / "scripts" / "train_hrl_high.py"
|
|
||||||
spec = importlib.util.spec_from_file_location("train_hrl_high_ckpt", path)
|
|
||||||
module = importlib.util.module_from_spec(spec)
|
|
||||||
assert spec.loader is not None
|
|
||||||
sys.modules[spec.name] = module
|
|
||||||
spec.loader.exec_module(module)
|
|
||||||
return module
|
|
||||||
|
|
||||||
|
|
||||||
def _require_rl():
|
|
||||||
pytest.importorskip("stable_baselines3")
|
|
||||||
pytest.importorskip("sb3_contrib")
|
|
||||||
pytest.importorskip("torch")
|
|
||||||
|
|
||||||
|
|
||||||
class _Run:
|
|
||||||
"""一次子进程训练：stdout 走文件，避免管道写满把子进程卡死。
|
|
||||||
|
|
||||||
与训练前端一致：用控制文件请求暂停/停止，进程组隔离走
|
|
||||||
:func:`mcm26b.process_control.spawn_kwargs`（POSIX ``setsid`` /
|
|
||||||
Windows ``CREATE_NEW_PROCESS_GROUP``）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
def __init__(self, run_dir: Path, *extra: str):
|
|
||||||
script = Path(__file__).parents[1] / "scripts" / "train_hrl_high.py"
|
|
||||||
self.run_dir = run_dir
|
|
||||||
self.log_path = run_dir / "train_stdout.log"
|
|
||||||
self.log_file = self.log_path.open("w", encoding="utf-8")
|
|
||||||
self.control_path = run_dir / "control.json"
|
|
||||||
write_control(self.control_path)
|
|
||||||
cmd = [sys.executable, str(script),
|
|
||||||
"--problem", "3", "--stage", "1", "--timesteps", "200000",
|
|
||||||
"--seed", "7", "--envs", "2", "--vec-env", "subproc",
|
|
||||||
"--device", "cpu", "--config-profile", "coarse-pretrain",
|
|
||||||
"--reward-profile", "time", "--eval-episodes", "1",
|
|
||||||
"--rollout-steps", "32", "--batch-size", "32", "--n-epochs", "1",
|
|
||||||
"--max-wall-s", "600", "--force", "--verbose", "0",
|
|
||||||
"--control-file", str(self.control_path),
|
|
||||||
"--model", str(run_dir / "q3.zip"), *extra]
|
|
||||||
self.proc = subprocess.Popen( # pylint: disable=consider-using-with
|
|
||||||
cmd, stdout=self.log_file, stderr=subprocess.STDOUT, text=True,
|
|
||||||
**spawn_kwargs())
|
|
||||||
|
|
||||||
def checkpoints(self) -> list[Path]:
|
|
||||||
"""历史 checkpoint（``<stem>_<步数>.zip``）。
|
|
||||||
|
|
||||||
刻意排除滚动恢复点 ``<stem>_latest.zip``：它随写随覆盖，不是"第几个
|
|
||||||
checkpoint"，混进来会让序号断言失去意义。
|
|
||||||
"""
|
|
||||||
found = (self.run_dir / "checkpoints").glob("q3_ckpt_[0-9]*.zip")
|
|
||||||
return sorted(path for path in found
|
|
||||||
if path.stem.rsplit("_", 1)[-1].isdigit())
|
|
||||||
|
|
||||||
def recovery(self) -> Path:
|
|
||||||
return self.run_dir / "checkpoints" / "q3_ckpt_latest.zip"
|
|
||||||
|
|
||||||
def resume_hint(self) -> dict:
|
|
||||||
return json.loads((self.run_dir / "resume.json").read_text(
|
|
||||||
encoding="utf-8"))
|
|
||||||
|
|
||||||
def wait_for(self, predicate, timeout_s: float) -> bool:
|
|
||||||
deadline = time.monotonic() + timeout_s
|
|
||||||
while time.monotonic() < deadline:
|
|
||||||
if predicate():
|
|
||||||
return True
|
|
||||||
if self.proc.poll() is not None:
|
|
||||||
return predicate()
|
|
||||||
time.sleep(0.4)
|
|
||||||
return predicate()
|
|
||||||
|
|
||||||
def logs(self) -> str:
|
|
||||||
try:
|
|
||||||
return self.log_path.read_text(encoding="utf-8", errors="replace")
|
|
||||||
except OSError: # pragma: no cover - 文件被并发写时的极端情况
|
|
||||||
return ""
|
|
||||||
|
|
||||||
def pause(self) -> None:
|
|
||||||
write_control(self.control_path, paused=True)
|
|
||||||
|
|
||||||
def resume(self) -> None:
|
|
||||||
write_control(self.control_path, paused=False)
|
|
||||||
|
|
||||||
def _finish(self) -> str:
|
|
||||||
self.log_file.close()
|
|
||||||
return self.logs()
|
|
||||||
|
|
||||||
def stop(self) -> str:
|
|
||||||
"""复现训练前端的停止语义：写控制文件请求优雅收尾（跨平台）。"""
|
|
||||||
write_control(self.control_path, stop=True)
|
|
||||||
forced = terminate_process_tree(self.proc, grace_s=240.0)
|
|
||||||
if forced:
|
|
||||||
self.log_file.close()
|
|
||||||
pytest.fail("控制文件停止请求后训练进程未在 240s 内退出（优雅停止失败）")
|
|
||||||
return self._finish()
|
|
||||||
|
|
||||||
def stop_by_signal(self) -> str:
|
|
||||||
"""POSIX 兜底路径：对整个进程组发 SIGTERM（Windows 不支持 killpg）。"""
|
|
||||||
if os.name == "nt": # pragma: no cover - 仅在 Windows 上走到
|
|
||||||
pytest.skip("Windows 无 os.killpg，信号路径不适用")
|
|
||||||
try:
|
|
||||||
os.killpg(os.getpgid(self.proc.pid), signal.SIGTERM)
|
|
||||||
except (ProcessLookupError, PermissionError):
|
|
||||||
self.proc.terminate()
|
|
||||||
try:
|
|
||||||
self.proc.wait(timeout=240)
|
|
||||||
except subprocess.TimeoutExpired:
|
|
||||||
self.proc.kill()
|
|
||||||
self.proc.wait(timeout=30)
|
|
||||||
self.log_file.close()
|
|
||||||
pytest.fail("SIGTERM 后训练进程未在 240s 内退出（优雅停止失败）")
|
|
||||||
return self._finish()
|
|
||||||
|
|
||||||
def hard_kill(self) -> None:
|
|
||||||
"""模拟"急停"：立即强杀整棵进程树，不给训练脚本任何收尾机会。"""
|
|
||||||
terminate_process_tree(self.proc, grace_s=0.0)
|
|
||||||
self.log_file.close()
|
|
||||||
|
|
||||||
|
|
||||||
def _run_dir(tmp_path: Path) -> Path:
|
|
||||||
run_dir = tmp_path / "run"
|
|
||||||
run_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
return run_dir
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------- 纯函数回归
|
|
||||||
def test_prune_checkpoints_keeps_newest(tmp_path):
|
|
||||||
module = _train_module()
|
|
||||||
directory = tmp_path / "checkpoints"
|
|
||||||
directory.mkdir()
|
|
||||||
for index in range(5):
|
|
||||||
for suffix in (".zip", ".json"):
|
|
||||||
target = directory / f"q3_ckpt_{index:09d}{suffix}"
|
|
||||||
target.write_bytes(b"x")
|
|
||||||
stamp = 1_700_000_000 + index
|
|
||||||
os.utime(target, (stamp, stamp))
|
|
||||||
removed = module._prune_checkpoints(directory, "q3_ckpt", 2)
|
|
||||||
assert len(removed) == 3
|
|
||||||
remaining = sorted(path.name for path in directory.iterdir())
|
|
||||||
assert remaining == ["q3_ckpt_000000003.json", "q3_ckpt_000000003.zip",
|
|
||||||
"q3_ckpt_000000004.json", "q3_ckpt_000000004.zip"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_worker_ignores_stop_signals_but_main_does_not(monkeypatch):
|
|
||||||
module = _train_module()
|
|
||||||
original = signal.getsignal(signal.SIGTERM)
|
|
||||||
|
|
||||||
class _Worker:
|
|
||||||
name = "SpawnProcess-3"
|
|
||||||
|
|
||||||
monkeypatch.setattr(multiprocessing, "current_process", _Worker)
|
|
||||||
try:
|
|
||||||
module._ignore_stop_signals_in_worker()
|
|
||||||
assert signal.getsignal(signal.SIGTERM) == signal.SIG_IGN
|
|
||||||
assert signal.getsignal(signal.SIGINT) == signal.SIG_IGN
|
|
||||||
finally:
|
|
||||||
signal.signal(signal.SIGTERM, original)
|
|
||||||
signal.signal(signal.SIGINT, signal.default_int_handler)
|
|
||||||
|
|
||||||
class _Main:
|
|
||||||
name = "MainProcess"
|
|
||||||
|
|
||||||
monkeypatch.setattr(multiprocessing, "current_process", _Main)
|
|
||||||
module._ignore_stop_signals_in_worker()
|
|
||||||
assert signal.getsignal(signal.SIGTERM) == original
|
|
||||||
|
|
||||||
|
|
||||||
def test_policy_hyperparams_records_effective_values():
|
|
||||||
"""元数据必须记实际生效的超参（resume 时来自 checkpoint，而非命令行）。"""
|
|
||||||
module = _train_module()
|
|
||||||
|
|
||||||
class _Schedule:
|
|
||||||
def __init__(self, value):
|
|
||||||
self._value = value
|
|
||||||
|
|
||||||
def current_value(self):
|
|
||||||
return self._value
|
|
||||||
|
|
||||||
class _Model:
|
|
||||||
n_steps = 256
|
|
||||||
batch_size = 256
|
|
||||||
n_epochs = 10
|
|
||||||
gamma = 1.0
|
|
||||||
gae_lambda = 0.95
|
|
||||||
learning_rate = _Schedule(3e-4)
|
|
||||||
clip_range = _Schedule(0.2)
|
|
||||||
ent_coef = 0.005
|
|
||||||
vf_coef = 0.5
|
|
||||||
max_grad_norm = 0.5
|
|
||||||
|
|
||||||
hyper = module._policy_hyperparams(_Model())
|
|
||||||
assert hyper["n_steps"] == 256 and hyper["batch_size"] == 256
|
|
||||||
assert hyper["n_epochs"] == 10
|
|
||||||
assert hyper["learning_rate"] == 3e-4
|
|
||||||
assert hyper["clip_range"] == 0.2
|
|
||||||
assert hyper["gamma"] == 1.0 and hyper["ent_coef"] == 0.005
|
|
||||||
assert module._policy_hyperparams(object())["n_steps"] == 0
|
|
||||||
|
|
||||||
|
|
||||||
def test_control_poll_state_machine(tmp_path):
|
|
||||||
"""控制文件轮询状态机：抽样读盘、暂停阻塞、stop 打断、暂停时长累计。"""
|
|
||||||
module = _train_module()
|
|
||||||
path = tmp_path / "control.json"
|
|
||||||
write_control(path, paused=True)
|
|
||||||
poll = module._ControlPoll(path, poll_every=1, poll_s=0.05)
|
|
||||||
assert poll.poll().paused is True
|
|
||||||
|
|
||||||
events = []
|
|
||||||
|
|
||||||
def _resume_later():
|
|
||||||
time.sleep(0.15)
|
|
||||||
write_control(path, paused=False)
|
|
||||||
|
|
||||||
threading.Thread(target=_resume_later, daemon=True).start()
|
|
||||||
stopped = poll.wait_while_paused(lambda: events.append("pause"),
|
|
||||||
lambda: events.append("resume"))
|
|
||||||
assert stopped is False
|
|
||||||
assert events == ["pause", "resume"]
|
|
||||||
assert poll.paused_total_s >= 0.1
|
|
||||||
|
|
||||||
# 暂停期间收到 stop：wait 必须返回 True，让调用方转入收尾
|
|
||||||
write_control(path, paused=True)
|
|
||||||
|
|
||||||
def _stop_later():
|
|
||||||
time.sleep(0.15)
|
|
||||||
write_control(path, stop=True)
|
|
||||||
|
|
||||||
threading.Thread(target=_stop_later, daemon=True).start()
|
|
||||||
assert poll.wait_while_paused(lambda: None, lambda: None) is True
|
|
||||||
assert poll.poll().stop is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_atomic_model_save_keeps_previous_snapshot_on_failure(tmp_path):
|
|
||||||
"""保存途中失败/被杀：目标路径仍是上一份完整快照，不会出现半截 zip。"""
|
|
||||||
module = _train_module()
|
|
||||||
target = tmp_path / "q3_ckpt_latest.zip"
|
|
||||||
target.write_bytes(b"OLD-COMPLETE")
|
|
||||||
|
|
||||||
class _FailingModel:
|
|
||||||
def save(self, path):
|
|
||||||
Path(path).write_bytes(b"PARTIAL")
|
|
||||||
raise RuntimeError("磁盘写满")
|
|
||||||
|
|
||||||
with pytest.raises(RuntimeError):
|
|
||||||
module._save_model_atomic(_FailingModel(), target)
|
|
||||||
assert target.read_bytes() == b"OLD-COMPLETE"
|
|
||||||
leftovers = [p.name for p in tmp_path.iterdir() if p.name != target.name]
|
|
||||||
assert leftovers and all(name.startswith(".tmp-") for name in leftovers)
|
|
||||||
|
|
||||||
|
|
||||||
def test_write_json_atomic_replaces_in_one_step(tmp_path):
|
|
||||||
module = _train_module()
|
|
||||||
path = tmp_path / "resume.json"
|
|
||||||
module._write_json_atomic(path, {"model": "a.zip", "timesteps": 1})
|
|
||||||
assert json.loads(path.read_text(encoding="utf-8"))["timesteps"] == 1
|
|
||||||
module._write_json_atomic(path, {"model": "b.zip", "timesteps": 2})
|
|
||||||
assert json.loads(path.read_text(encoding="utf-8"))["model"] == "b.zip"
|
|
||||||
assert not list(tmp_path.glob(".tmp-*"))
|
|
||||||
|
|
||||||
|
|
||||||
def test_prune_checkpoints_never_deletes_recovery_point(tmp_path):
|
|
||||||
"""滚动恢复点与临时文件不参与历史清理。"""
|
|
||||||
module = _train_module()
|
|
||||||
directory = tmp_path / "checkpoints"
|
|
||||||
directory.mkdir()
|
|
||||||
latest = directory / "q3_ckpt_latest.zip"
|
|
||||||
latest.write_bytes(b"RECOVERY")
|
|
||||||
(directory / ".tmp-q3_ckpt_000000009.zip").write_bytes(b"PARTIAL")
|
|
||||||
for index in range(5):
|
|
||||||
(directory / f"q3_ckpt_{index:09d}.zip").write_bytes(b"x")
|
|
||||||
module._prune_checkpoints(directory, "q3_ckpt", 1)
|
|
||||||
assert latest.exists() and latest.read_bytes() == b"RECOVERY"
|
|
||||||
assert (directory / ".tmp-q3_ckpt_000000009.zip").exists()
|
|
||||||
|
|
||||||
|
|
||||||
# --------------------------------------------------------------- 集成回归
|
|
||||||
def test_recovery_point_exists_before_training_and_is_loadable(tmp_path):
|
|
||||||
"""起步基线：训练刚起就有可加载、可续训的产物（--checkpoint-every 0 也成立）。"""
|
|
||||||
_require_rl()
|
|
||||||
from sb3_contrib import MaskablePPO
|
|
||||||
|
|
||||||
run = _Run(_run_dir(tmp_path), "--checkpoint-every", "0")
|
|
||||||
try:
|
|
||||||
assert run.wait_for(lambda: run.recovery().exists(), 120.0), \
|
|
||||||
"起步恢复点未落盘"
|
|
||||||
loaded = MaskablePPO.load(str(run.recovery()))
|
|
||||||
assert loaded.num_timesteps >= 0
|
|
||||||
hint = run.resume_hint()
|
|
||||||
assert Path(hint["model"]) == run.recovery()
|
|
||||||
assert hint["timesteps_semantics"] == "additional"
|
|
||||||
finally:
|
|
||||||
run.stop()
|
|
||||||
|
|
||||||
|
|
||||||
def test_hard_kill_still_leaves_resumable_recovery_point(tmp_path):
|
|
||||||
"""急停（强杀、无收尾）后仍能续训：滚动恢复点 + resume.json 都在且可加载。"""
|
|
||||||
_require_rl()
|
|
||||||
from sb3_contrib import MaskablePPO
|
|
||||||
|
|
||||||
run = _Run(_run_dir(tmp_path), "--checkpoint-every", "0",
|
|
||||||
"--recovery-every", "32")
|
|
||||||
try:
|
|
||||||
assert run.wait_for(lambda: run.recovery().exists(), 120.0), \
|
|
||||||
"起步恢复点未落盘"
|
|
||||||
first = run.recovery().stat().st_mtime_ns
|
|
||||||
assert run.wait_for(
|
|
||||||
lambda: run.recovery().stat().st_mtime_ns != first, 300.0), \
|
|
||||||
"滚动恢复点未按 --recovery-every 刷新"
|
|
||||||
assert run.proc.poll() is None, "训练已自行结束，无法验证急停"
|
|
||||||
finally:
|
|
||||||
run.hard_kill() # 急停：强杀整棵进程树，没有任何 Python 收尾
|
|
||||||
|
|
||||||
hint = run.resume_hint()
|
|
||||||
assert Path(hint["model"]).is_file(), hint
|
|
||||||
assert Path(hint["model"]) == run.recovery(), hint
|
|
||||||
loaded = MaskablePPO.load(str(run.recovery()))
|
|
||||||
assert loaded.num_timesteps >= 32, loaded.num_timesteps
|
|
||||||
metadata = json.loads(run.recovery().with_suffix(".json").read_text(
|
|
||||||
encoding="utf-8"))
|
|
||||||
assert metadata["recovery"] is True
|
|
||||||
assert metadata["checkpoint_reason"] in ("baseline", "interval")
|
|
||||||
# 急停没有 graceful 收尾，所以不会有"stopped_by_signal"的最终模型
|
|
||||||
assert not (run.run_dir / "q3.zip").exists()
|
|
||||||
|
|
||||||
|
|
||||||
def test_checkpoint_is_written_and_loadable(tmp_path):
|
|
||||||
_require_rl()
|
|
||||||
from sb3_contrib import MaskablePPO
|
|
||||||
|
|
||||||
run = _Run(_run_dir(tmp_path), "--checkpoint-every", "64")
|
|
||||||
try:
|
|
||||||
assert run.wait_for(run.checkpoints, 300.0), \
|
|
||||||
"训练未在 300s 内产出 checkpoint"
|
|
||||||
finally:
|
|
||||||
run.stop()
|
|
||||||
|
|
||||||
archives = run.checkpoints()
|
|
||||||
assert archives, "checkpoint 目录为空"
|
|
||||||
loaded = MaskablePPO.load(str(archives[-1]))
|
|
||||||
assert loaded.num_timesteps >= 64
|
|
||||||
metadata = json.loads(archives[-1].with_suffix(".json").read_text(
|
|
||||||
encoding="utf-8"))
|
|
||||||
assert metadata["checkpoint"] is True
|
|
||||||
assert metadata["checkpoint_timesteps"] == loaded.num_timesteps
|
|
||||||
assert metadata["model_sha256"]
|
|
||||||
hyper = metadata["ppo_hyperparams"]
|
|
||||||
assert (hyper["n_steps"], hyper["batch_size"], hyper["n_epochs"]) == (32, 32, 1)
|
|
||||||
|
|
||||||
|
|
||||||
def test_control_file_stop_saves_model(tmp_path):
|
|
||||||
"""控制文件 stop 是训练前端的默认停止路径，Windows 上同样必须成立。"""
|
|
||||||
_require_rl()
|
|
||||||
run = _Run(_run_dir(tmp_path), "--checkpoint-every", "64")
|
|
||||||
model_path = run.run_dir / "q3.zip"
|
|
||||||
try:
|
|
||||||
assert run.wait_for(run.checkpoints, 300.0), \
|
|
||||||
"训练未在 300s 内进入 rollout 阶段"
|
|
||||||
assert run.proc.poll() is None, "训练进程提前退出，无法验证优雅停止"
|
|
||||||
finally:
|
|
||||||
output = run.stop()
|
|
||||||
|
|
||||||
assert run.proc.returncode == 0, \
|
|
||||||
f"控制文件停止应正常退出，实际 {run.proc.returncode}\n{output}"
|
|
||||||
assert model_path.exists(), "控制文件停止后最终模型未落盘"
|
|
||||||
metadata = json.loads(model_path.with_suffix(".json").read_text(
|
|
||||||
encoding="utf-8"))
|
|
||||||
assert metadata["stopped_by_signal"] is True
|
|
||||||
assert metadata["stop_signal"] == "control_file"
|
|
||||||
assert metadata["checkpoint_files"], "元数据应记录已产出的 checkpoint"
|
|
||||||
assert not (run.run_dir / "training_failure.json").exists()
|
|
||||||
assert "saved_after_stop" in output
|
|
||||||
|
|
||||||
|
|
||||||
def test_control_file_pause_then_resume_keeps_training_alive(tmp_path):
|
|
||||||
"""暂停是协作式的：进程存活、停在 step 边界，continue 后照常收尾。"""
|
|
||||||
_require_rl()
|
|
||||||
run = _Run(_run_dir(tmp_path), "--checkpoint-every", "64")
|
|
||||||
try:
|
|
||||||
assert run.wait_for(run.checkpoints, 300.0), \
|
|
||||||
"训练未在 300s 内进入 rollout 阶段"
|
|
||||||
run.pause()
|
|
||||||
assert run.wait_for(lambda: '"event": "paused"' in run.logs(), 120.0), \
|
|
||||||
"训练脚本未在 120s 内进入暂停"
|
|
||||||
assert run.proc.poll() is None, "暂停不应结束进程"
|
|
||||||
run.resume()
|
|
||||||
assert run.wait_for(lambda: '"event": "resumed"' in run.logs(), 120.0), \
|
|
||||||
"训练脚本未在 120s 内恢复"
|
|
||||||
finally:
|
|
||||||
run.stop()
|
|
||||||
assert run.proc.returncode == 0
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.skipif(os.name == "nt", reason="Windows 没有 os.killpg，信号兜底路径不适用")
|
|
||||||
def test_sigterm_saves_model_instead_of_losing_the_run(tmp_path):
|
|
||||||
_require_rl()
|
|
||||||
run = _Run(_run_dir(tmp_path), "--checkpoint-every", "64")
|
|
||||||
model_path = run.run_dir / "q3.zip"
|
|
||||||
try:
|
|
||||||
assert run.wait_for(run.checkpoints, 300.0), \
|
|
||||||
"训练未在 300s 内进入 rollout 阶段"
|
|
||||||
assert run.proc.poll() is None, "训练进程提前退出，无法验证优雅停止"
|
|
||||||
finally:
|
|
||||||
output = run.stop_by_signal()
|
|
||||||
|
|
||||||
assert run.proc.returncode == 0, \
|
|
||||||
f"优雅停止应正常退出，实际 {run.proc.returncode}\n{output}"
|
|
||||||
assert model_path.exists(), "SIGTERM 后最终模型未落盘"
|
|
||||||
metadata = json.loads(model_path.with_suffix(".json").read_text(
|
|
||||||
encoding="utf-8"))
|
|
||||||
assert metadata["stopped_by_signal"] is True
|
|
||||||
assert metadata["stop_signal"] == "SIGTERM"
|
|
||||||
assert metadata["checkpoint_files"], "元数据应记录已产出的 checkpoint"
|
|
||||||
assert not (run.run_dir / "training_failure.json").exists()
|
|
||||||
assert "saved_after_stop" in output
|
|
||||||
@@ -1,119 +0,0 @@
|
|||||||
"""低层导航运动学与训练入口的行为回归；不验证策略收益。"""
|
|
||||||
|
|
||||||
import importlib.util
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
np = pytest.importorskip("numpy")
|
|
||||||
pytest.importorskip("gymnasium")
|
|
||||||
|
|
||||||
from mcm26b.hrl.executor import LowLevelExecutor, navigation_transition
|
|
||||||
from mcm26b.hrl.low_level_env import LowLevelEnv
|
|
||||||
|
|
||||||
|
|
||||||
def _training_script():
|
|
||||||
spec = importlib.util.spec_from_file_location(
|
|
||||||
"train_hrl_low_revision", Path(__file__).parents[1]
|
|
||||||
/ "scripts" / "train_hrl_low.py")
|
|
||||||
module = importlib.util.module_from_spec(spec)
|
|
||||||
spec.loader.exec_module(module)
|
|
||||||
return module
|
|
||||||
|
|
||||||
|
|
||||||
def test_shortened_step_preserves_selected_direction_without_snapping():
|
|
||||||
position, moved = navigation_transition((0.0, 0.0), (30.0, 40.0), 0)
|
|
||||||
assert position == pytest.approx((50.0, 0.0))
|
|
||||||
assert moved == pytest.approx(50.0)
|
|
||||||
assert position != (30.0, 40.0)
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
navigation_transition((0, 0), (300, 0), 8)
|
|
||||||
|
|
||||||
|
|
||||||
def test_near_goal_reward_and_time_charge_actual_distance():
|
|
||||||
env = LowLevelEnv()
|
|
||||||
env.reset(seed=1)
|
|
||||||
env._position, env._goal = (0.0, 0.0), (30.0, 0.0)
|
|
||||||
_, reward, terminated, truncated, info = env.step(0)
|
|
||||||
assert terminated and not truncated
|
|
||||||
assert info["path_length_m"] == pytest.approx(30.0)
|
|
||||||
assert info["step_virtual_time_s"] == pytest.approx(6.0)
|
|
||||||
assert reward == pytest.approx(env.reach_bonus - 6.0 / env.step_cost_scale)
|
|
||||||
|
|
||||||
|
|
||||||
def test_expert_reaches_all_200_fixed_seed_targets():
|
|
||||||
env = LowLevelEnv()
|
|
||||||
for seed in range(200):
|
|
||||||
env.reset(seed=seed)
|
|
||||||
for _ in range(env.max_steps):
|
|
||||||
obs, _, terminated, truncated, info = env.step(env.expert_action())
|
|
||||||
assert env.observation_space.contains(obs)
|
|
||||||
if terminated or truncated:
|
|
||||||
break
|
|
||||||
assert terminated, (seed, info)
|
|
||||||
assert info["distance_m"] <= env.reach_m
|
|
||||||
assert info["virtual_time_s"] == pytest.approx(
|
|
||||||
info["path_length_m"] / env.config.cost.speed_mps)
|
|
||||||
|
|
||||||
|
|
||||||
def test_failed_executor_reports_failure_without_free_terminal_move():
|
|
||||||
class WestPolicy:
|
|
||||||
def select(self, _observation):
|
|
||||||
return 4
|
|
||||||
|
|
||||||
executor = LowLevelExecutor(WestPolicy(), max_steps=3)
|
|
||||||
path = executor.roll((0, 0), (300, 0))
|
|
||||||
assert len(path) == 3
|
|
||||||
assert path[-1] == pytest.approx((-300, 0))
|
|
||||||
assert executor.last_roll_info == pytest.approx({
|
|
||||||
"reached": False, "truncated": True,
|
|
||||||
"final_distance_m": 600.0, "path_length_m": 300.0})
|
|
||||||
|
|
||||||
|
|
||||||
def test_env_and_executor_share_step_transitions():
|
|
||||||
class EastPolicy:
|
|
||||||
def select(self, _observation):
|
|
||||||
return 0
|
|
||||||
|
|
||||||
env = LowLevelEnv(max_steps=3)
|
|
||||||
env.reset(seed=3)
|
|
||||||
start, goal = (0.0, 0.0), (350.0, 25.0)
|
|
||||||
env._position, env._goal = start, goal
|
|
||||||
actual = []
|
|
||||||
for _ in range(3):
|
|
||||||
_, _, _, _, info = env.step(0)
|
|
||||||
actual.append(info["position"])
|
|
||||||
executor = LowLevelExecutor(EastPolicy(), max_steps=3)
|
|
||||||
assert executor.roll(start, goal) == actual
|
|
||||||
assert executor.last_roll_info["path_length_m"] == info["path_length_m"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_bc_holdout_keeps_each_episode_wholly_in_one_partition():
|
|
||||||
script = _training_script()
|
|
||||||
episode_ids = np.repeat(np.arange(20), 5)
|
|
||||||
mask = script.bc_validation_mask(episode_ids, seed=7)
|
|
||||||
assert mask.any() and not mask.all()
|
|
||||||
for episode in np.unique(episode_ids):
|
|
||||||
assert len(set(mask[episode_ids == episode])) == 1
|
|
||||||
assert np.array_equal(mask, script.bc_validation_mask(episode_ids, seed=7))
|
|
||||||
|
|
||||||
|
|
||||||
def test_bc_copy_and_maskable_ppo_learn_with_two_environments():
|
|
||||||
torch = pytest.importorskip("torch")
|
|
||||||
pytest.importorskip("sb3_contrib")
|
|
||||||
from stable_baselines3.common.vec_env import DummyVecEnv
|
|
||||||
|
|
||||||
previous_threads = torch.get_num_threads()
|
|
||||||
torch.set_num_threads(1)
|
|
||||||
script = _training_script()
|
|
||||||
env = DummyVecEnv([LowLevelEnv, LowLevelEnv])
|
|
||||||
try:
|
|
||||||
bc = script._bc_module(10, 8, script.HIDDEN)()
|
|
||||||
model = script.copy_into_ppo(bc, "cpu", env, seed=9, n_steps=8)
|
|
||||||
model.learn(total_timesteps=16)
|
|
||||||
assert model.n_envs == 2
|
|
||||||
assert model.num_timesteps == 16
|
|
||||||
assert np.all(env.env_method("action_masks"))
|
|
||||||
finally:
|
|
||||||
env.close()
|
|
||||||
torch.set_num_threads(previous_threads)
|
|
||||||
@@ -1,86 +0,0 @@
|
|||||||
"""P1 交付回归：benchmark manifest/真值隔离、纯时间奖励与 project 兼容层。
|
|
||||||
|
|
||||||
这些测试不训练神经网络，只验证评测入口和接口契约；重训练属于 smoke 手测。
|
|
||||||
"""
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import importlib.util
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
np = pytest.importorskip("numpy")
|
|
||||||
pytest.importorskip("referee")
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
|
|
||||||
|
|
||||||
def _benchmark_module():
|
|
||||||
path = Path(__file__).parents[1] / "scripts" / "benchmark_hrl.py"
|
|
||||||
spec = importlib.util.spec_from_file_location("benchmark_hrl_p1", path)
|
|
||||||
module = importlib.util.module_from_spec(spec)
|
|
||||||
assert spec.loader is not None
|
|
||||||
sys.modules[spec.name] = module
|
|
||||||
try:
|
|
||||||
spec.loader.exec_module(module)
|
|
||||||
finally:
|
|
||||||
pass
|
|
||||||
return module
|
|
||||||
|
|
||||||
|
|
||||||
def test_manifest_roundtrip_and_truth_summary():
|
|
||||||
module = _benchmark_module()
|
|
||||||
manifest = module.make_default_manifest(
|
|
||||||
problem=3, scenes=2, sources=3, directional=0,
|
|
||||||
start_seed=123, config=ProblemConfig())
|
|
||||||
scenes = module.scenes_from_manifest(manifest)
|
|
||||||
assert [scene.scene_id for scene in scenes] == ["scene-123", "scene-124"]
|
|
||||||
assert all(scene.truth_channels for scene in scenes)
|
|
||||||
assert all(1 <= channel <= 20 for scene in scenes
|
|
||||||
for channel in scene.truth_channels)
|
|
||||||
|
|
||||||
|
|
||||||
def test_benchmark_heuristic_is_valid_and_truth_isolated(tmp_path):
|
|
||||||
module = _benchmark_module()
|
|
||||||
manifest = module.make_default_manifest(
|
|
||||||
problem=3, scenes=1, sources=3, directional=0,
|
|
||||||
start_seed=777, config=ProblemConfig())
|
|
||||||
scene = module.scenes_from_manifest(manifest)[0]
|
|
||||||
run_dir = tmp_path / "run"
|
|
||||||
result = module.run_scene_variant(
|
|
||||||
scene, "hrl_heuristic", config=ProblemConfig(), hrl_model=None,
|
|
||||||
run_dir=run_dir)
|
|
||||||
row = result["row"]
|
|
||||||
assert row["valid"] is True
|
|
||||||
assert row["cleared_count"] == len(scene.truth_channels) == 3
|
|
||||||
assert row["cleared_matches_truth"] is True
|
|
||||||
assert row["no_source_declared_absent"] is True
|
|
||||||
assert (run_dir / "session.jsonl").exists()
|
|
||||||
# manifest/run summary 中不出现策略可消费的 simulator.sources 真值对象。
|
|
||||||
assert "sources" not in row
|
|
||||||
|
|
||||||
|
|
||||||
def test_time_only_reward_has_no_per_source_shaping():
|
|
||||||
from mcm26b.hrl.hierarchical_env import HRLReward
|
|
||||||
reward = HRLReward.time_only()
|
|
||||||
assert reward.clear == 0.0
|
|
||||||
assert reward.detect == 0.0
|
|
||||||
assert reward.info_weight == 0.0
|
|
||||||
assert reward.time_scale == 100.0
|
|
||||||
assert reward.failure >= 3600.0
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.skipif(importlib.util.find_spec("gymnasium") is None,
|
|
||||||
reason="RL 依赖组未安装")
|
|
||||||
def test_project_compat_layer_reexports_real_classes():
|
|
||||||
from mcm26b.hrl.hierarchical_env import HierarchicalEnv
|
|
||||||
# project 是仓库根部的兼容层：pytest 经 rootdir 把仓库根放进 sys.path，
|
|
||||||
# 而单独 lint tests/ 时 pylint 的 sys.path 不含仓库根，故此处局部豁免。
|
|
||||||
# pylint: disable=import-error
|
|
||||||
from project.env.local_env import HierarchicalEnv as CompatEnv
|
|
||||||
from project.env.simulator_client import SimulatorClient
|
|
||||||
from project.policies.high_level_ppo import HighLevelPolicy
|
|
||||||
assert CompatEnv is HierarchicalEnv
|
|
||||||
assert SimulatorClient.__module__ == "mcm26b.protocol.client"
|
|
||||||
assert HighLevelPolicy.__module__ == "mcm26b.hrl.policy"
|
|
||||||
@@ -1,88 +0,0 @@
|
|||||||
"""HRL 协议执行的分支回归；所有模拟器交互仅在本地进行。"""
|
|
||||||
|
|
||||||
from unittest.mock import patch
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
pytest.importorskip("numpy")
|
|
||||||
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.harness import LocalSimTransport
|
|
||||||
from mcm26b.hrl.constants import FALLBACK_OFFSET, N_CANDIDATES, STATION_OFFSET
|
|
||||||
from mcm26b.hrl.scheduler import Candidate
|
|
||||||
from mcm26b.planner.hrl_loop import HRLLoop
|
|
||||||
from mcm26b.protocol import SimulatorClient
|
|
||||||
from referee import LocalSimulator, Source
|
|
||||||
|
|
||||||
|
|
||||||
def make_loop(*, problem=4, high_policy=None):
|
|
||||||
simulator = LocalSimulator(
|
|
||||||
[Source(channel=20, x=45.0, y=0.0, r_eff_m=1000.0)],
|
|
||||||
robot_id="hrl-branch-test")
|
|
||||||
client = SimulatorClient("hrl-branch-test", transport=LocalSimTransport(simulator))
|
|
||||||
client.enter()
|
|
||||||
return HRLLoop(client, ProblemConfig(), problem=problem, high_policy=high_policy)
|
|
||||||
|
|
||||||
|
|
||||||
def test_fallback_failure_then_success_records_actual_time_position_and_clear():
|
|
||||||
loop = make_loop()
|
|
||||||
try:
|
|
||||||
observation = loop.client.measure((0.0, 0.0), 20)
|
|
||||||
loop._record_measure(20, (0.0, 0.0), observation)
|
|
||||||
loop.blocked_probes[20] = [(0.0, 0.0)]
|
|
||||||
candidate = Candidate(FALLBACK_OFFSET + 19, "fallback", (999.0, 999.0),
|
|
||||||
channel=20)
|
|
||||||
with patch("mcm26b.hrl.scheduler.fallback_clear_cells",
|
|
||||||
return_value=[(20.0, 0.0), (40.0, 0.0)]):
|
|
||||||
outcome = loop.hrl_step(candidate.slot, [candidate])
|
|
||||||
assert outcome.clear_success == (20,)
|
|
||||||
assert loop.cleared == [20]
|
|
||||||
assert outcome.position_reached == (40.0, 0.0)
|
|
||||||
# 40m / 5m/s + 3s失败 + 5s成功；clear不产生换频时间。
|
|
||||||
assert outcome.delta_virtual_s == pytest.approx(16.0)
|
|
||||||
assert loop.client.state.channel == 20
|
|
||||||
assert loop.fallback_log[-1]["attempts"] == 2
|
|
||||||
assert 20 not in loop.blocked_probes
|
|
||||||
assert not loop.anomalies
|
|
||||||
finally:
|
|
||||||
loop.client.exit()
|
|
||||||
|
|
||||||
|
|
||||||
def test_invalid_policy_output_is_logged_and_replaced_by_valid_action():
|
|
||||||
class InvalidPolicy:
|
|
||||||
def select(self, _observation, _mask):
|
|
||||||
return N_CANDIDATES + 1
|
|
||||||
|
|
||||||
loop = make_loop(problem=3, high_policy=InvalidPolicy())
|
|
||||||
candidate = Candidate(STATION_OFFSET, "station", (0.0, 0.0), station_index=0)
|
|
||||||
executed = []
|
|
||||||
|
|
||||||
def execute(action, candidates): # pylint: disable=unused-argument
|
|
||||||
executed.append(action)
|
|
||||||
loop.stop_reason = "test_complete"
|
|
||||||
|
|
||||||
try:
|
|
||||||
with patch.object(loop, "build_candidates", return_value=[candidate]), \
|
|
||||||
patch.object(loop, "hrl_observation", return_value=None), \
|
|
||||||
patch.object(loop, "heuristic_action", return_value=candidate.slot), \
|
|
||||||
patch.object(loop, "hrl_step", side_effect=execute):
|
|
||||||
loop._coverage_phase()
|
|
||||||
assert executed == [candidate.slot]
|
|
||||||
assert loop.anomalies[0]["kind"] == "hrl_invalid_policy_action"
|
|
||||||
finally:
|
|
||||||
loop.client.exit()
|
|
||||||
|
|
||||||
|
|
||||||
def test_opportunistic_measure_precedes_candidate_snapshot():
|
|
||||||
loop = make_loop(problem=3)
|
|
||||||
order = []
|
|
||||||
try:
|
|
||||||
assert loop.config.bayes.opportunistic_measure
|
|
||||||
with patch.object(loop, "_measure_at_stop",
|
|
||||||
side_effect=lambda: order.append("measure")), \
|
|
||||||
patch.object(loop, "build_candidates",
|
|
||||||
side_effect=lambda: order.append("snapshot") or []):
|
|
||||||
loop._coverage_phase()
|
|
||||||
assert order == ["measure", "snapshot"]
|
|
||||||
finally:
|
|
||||||
loop.client.exit()
|
|
||||||
@@ -1,101 +0,0 @@
|
|||||||
"""RL 扩展验收：特征/动作布局、训练环境闭环与模型接口守卫。
|
|
||||||
|
|
||||||
RL 依赖属于可选组 [dependency-groups] rl；未安装 numpy/gymnasium 时
|
|
||||||
全部跳过（运行闭环不依赖它们）。
|
|
||||||
"""
|
|
||||||
|
|
||||||
import unittest
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
pytest = None
|
|
||||||
try:
|
|
||||||
import pytest
|
|
||||||
numpy = pytest.importorskip("numpy")
|
|
||||||
pytest.importorskip("gymnasium")
|
|
||||||
except Exception: # pragma: no cover - 无 RL 依赖环境
|
|
||||||
numpy = None
|
|
||||||
|
|
||||||
if numpy is not None:
|
|
||||||
from mcm26b.belief import CandidateRegion
|
|
||||||
from mcm26b.config import ProblemConfig
|
|
||||||
from mcm26b.rl._constants import GLOBAL_DIM, NUM_ACTIONS, OBS_DIM, PER_ACTION_DIM
|
|
||||||
from mcm26b.rl.env import LocalizationEnv
|
|
||||||
from mcm26b.rl.features import build_observation, candidate_actions
|
|
||||||
from mcm26b.rl.policy import default_model_path
|
|
||||||
|
|
||||||
CONFIG = ProblemConfig()
|
|
||||||
STATION = (0.0, 0.0)
|
|
||||||
BEARING = 30.0
|
|
||||||
|
|
||||||
|
|
||||||
def _region():
|
|
||||||
region = CandidateRegion.initial(STATION, CONFIG)
|
|
||||||
region.clip_wedge(STATION, BEARING, 1.005)
|
|
||||||
return region
|
|
||||||
|
|
||||||
|
|
||||||
@unittest.skipIf(numpy is None, "RL 依赖组未安装")
|
|
||||||
class FeatureLayoutTests(unittest.TestCase):
|
|
||||||
def test_action_candidates_have_stable_count_and_order(self):
|
|
||||||
region = _region()
|
|
||||||
first = candidate_actions(region, STATION, CONFIG, [])
|
|
||||||
self.assertEqual(len(first), NUM_ACTIONS)
|
|
||||||
second = candidate_actions(region, STATION, CONFIG, [])
|
|
||||||
self.assertEqual(first, second)
|
|
||||||
self.assertEqual(first[0], region.mec()[0]) # 0 号候选 = MEC 圆心
|
|
||||||
|
|
||||||
def test_observation_dimension_matches_model_contract(self):
|
|
||||||
region = _region()
|
|
||||||
obs = build_observation(region, STATION, 3, CONFIG, problem=3,
|
|
||||||
blocked=[], virtual_time_s=0.0)
|
|
||||||
self.assertEqual(obs.shape, (OBS_DIM,))
|
|
||||||
self.assertEqual(OBS_DIM, GLOBAL_DIM + NUM_ACTIONS * PER_ACTION_DIM)
|
|
||||||
|
|
||||||
def test_blocked_changes_observation_but_not_layout(self):
|
|
||||||
region = _region()
|
|
||||||
clean = build_observation(region, STATION, 3, CONFIG, problem=4,
|
|
||||||
blocked=[], virtual_time_s=0.0)
|
|
||||||
blocked = build_observation(region, STATION, 3, CONFIG, problem=4,
|
|
||||||
blocked=[(100.0, 0.0)], virtual_time_s=0.0)
|
|
||||||
self.assertEqual(clean.shape, blocked.shape)
|
|
||||||
self.assertFalse(numpy.allclose(clean, blocked))
|
|
||||||
|
|
||||||
|
|
||||||
@unittest.skipIf(numpy is None, "RL 依赖组未安装")
|
|
||||||
class TrainingEnvironmentTests(unittest.TestCase):
|
|
||||||
def test_q3_episode_reset_and_step_shapes(self):
|
|
||||||
env = LocalizationEnv(problem=3)
|
|
||||||
obs, info = env.reset(seed=1)
|
|
||||||
self.assertEqual(obs.shape, (OBS_DIM,))
|
|
||||||
self.assertIn("source", info)
|
|
||||||
obs, reward, terminated, truncated, info = env.step(0)
|
|
||||||
self.assertEqual(obs.shape, (OBS_DIM,))
|
|
||||||
self.assertIsInstance(reward, float)
|
|
||||||
self.assertFalse(terminated and truncated)
|
|
||||||
|
|
||||||
def test_q4_directional_reports_blocked_or_fallback(self):
|
|
||||||
env = LocalizationEnv(problem=4, directional_probability=1.0)
|
|
||||||
_, info = env.reset(seed=7)
|
|
||||||
self.assertTrue(info["source"]["directional"])
|
|
||||||
terminated = truncated = False
|
|
||||||
for _ in range(env.max_episode_steps + 2):
|
|
||||||
_, _, terminated, truncated, info = env.step(
|
|
||||||
env.action_space.sample())
|
|
||||||
if terminated or truncated:
|
|
||||||
break
|
|
||||||
self.assertTrue(terminated or truncated)
|
|
||||||
# 定向分支：要么清除，要么走过后备清除/截断语义。
|
|
||||||
self.assertIn("fallback_attempts", info)
|
|
||||||
|
|
||||||
def test_missing_model_raises_clear_error(self):
|
|
||||||
from mcm26b.rl.policy import PointSelectionPolicy
|
|
||||||
with self.assertRaises(RuntimeError):
|
|
||||||
PointSelectionPolicy("output/rl/does-not-exist.zip")
|
|
||||||
|
|
||||||
def test_default_model_paths(self):
|
|
||||||
self.assertEqual(default_model_path(3), Path("output/rl/q3_ppo.zip"))
|
|
||||||
self.assertEqual(default_model_path(4), Path("output/rl/q4_ppo.zip"))
|
|
||||||
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
unittest.main()
|
|
||||||
@@ -1,97 +0,0 @@
|
|||||||
"""训练/部署观测和动作的一致性，以及RL改进的目标口径。"""
|
|
||||||
import importlib.util
|
|
||||||
import math
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
np = pytest.importorskip('numpy')
|
|
||||||
pytest.importorskip('gymnasium')
|
|
||||||
|
|
||||||
from mcm26b.rl.env import LocalizationEnv
|
|
||||||
from mcm26b.rl.features import build_observation, candidate_actions
|
|
||||||
|
|
||||||
|
|
||||||
def test_training_observation_matches_runtime_channel_belief():
|
|
||||||
env = LocalizationEnv(problem=3, handoff_probability=1)
|
|
||||||
obs, _ = env.reset(seed=1)
|
|
||||||
belief = env._belief.channels[env._channel]
|
|
||||||
expected = build_observation(belief.region, env._client.state.position, env._channel,
|
|
||||||
env.config, env.problem, env._blocked, belief,
|
|
||||||
virtual_time_s=env._client.state.virtual_time_s,
|
|
||||||
receiver_channel=env._client.state.channel)
|
|
||||||
np.testing.assert_array_equal(obs, expected)
|
|
||||||
assert obs[7] == np.float32(1/12)
|
|
||||||
assert tuple(obs[14:17]) == (0,1,0)
|
|
||||||
assert env._client.state.channel != env._channel
|
|
||||||
env.close()
|
|
||||||
|
|
||||||
|
|
||||||
def test_mask_rejects_measured_and_duplicate_candidates():
|
|
||||||
env = LocalizationEnv(problem=3)
|
|
||||||
env.reset(seed=17)
|
|
||||||
belief = env._belief.channels[env._channel]
|
|
||||||
region = belief.region
|
|
||||||
position = env._client.state.position
|
|
||||||
candidates = candidate_actions(region, position, env.config)
|
|
||||||
belief.measurements.append(dict(position=candidates[0], result='no_signal', svd_deg=None))
|
|
||||||
mask = env.action_masks()
|
|
||||||
assert not mask[0]
|
|
||||||
keys = [(round(p[0]*2),round(p[1]*2)) for p,v in zip(candidates,mask) if v]
|
|
||||||
assert len(keys) == len(set(keys))
|
|
||||||
assert any(mask)
|
|
||||||
assert all(math.dist(p,region.mec()[0]) <= 1000-region.mec()[1]+1e-6
|
|
||||||
for p,v in zip(candidates,mask) if v)
|
|
||||||
env.close()
|
|
||||||
|
|
||||||
|
|
||||||
def test_q4_blocked_sequence_resets_on_direction():
|
|
||||||
env = LocalizationEnv(problem=4,directional_probability=1,handoff_probability=0)
|
|
||||||
env.reset(seed=7)
|
|
||||||
for expected in ('no_signal','direction'):
|
|
||||||
region = env._belief.candidate_set(env._channel)
|
|
||||||
candidates = candidate_actions(region,env._client.state.position,env.config,env._blocked)
|
|
||||||
# 只有测试端可用真值选择一个指定分支，策略从不接触该接口。
|
|
||||||
index = next(i for i,p in enumerate(candidates)
|
|
||||||
if env._simulator._observe(p,env._channel)[0] == expected)
|
|
||||||
_, _, terminated, truncated, _ = env.step(index)
|
|
||||||
assert len(env._blocked) == (1 if expected=='no_signal' else 0)
|
|
||||||
if expected=='no_signal':
|
|
||||||
assert not terminated and not truncated
|
|
||||||
env.close()
|
|
||||||
|
|
||||||
|
|
||||||
def test_success_reward_is_constant_minus_actual_virtual_cost():
|
|
||||||
env = LocalizationEnv(problem=3,handoff_probability=0)
|
|
||||||
env.reset(seed=3)
|
|
||||||
total = 0
|
|
||||||
for _ in range(16):
|
|
||||||
legal = np.flatnonzero(env.action_masks())
|
|
||||||
action = int(0 if 0 in legal else legal[0])
|
|
||||||
_, reward, terminated, truncated, info = env.step(action)
|
|
||||||
total += reward
|
|
||||||
if terminated or truncated:
|
|
||||||
break
|
|
||||||
assert info['cleared']
|
|
||||||
assert total == pytest.approx(20-info['virtual_cost_s']/100)
|
|
||||||
assert env._client.state.exited
|
|
||||||
|
|
||||||
|
|
||||||
def test_bc_and_ppo_forward_mapping_agree():
|
|
||||||
torch = pytest.importorskip('torch')
|
|
||||||
from stable_baselines3 import PPO
|
|
||||||
spec = importlib.util.spec_from_file_location('pretrain',Path('scripts/pretrain_rl_expert.py'))
|
|
||||||
module = importlib.util.module_from_spec(spec)
|
|
||||||
spec.loader.exec_module(module)
|
|
||||||
torch.set_num_threads(1)
|
|
||||||
env = LocalizationEnv()
|
|
||||||
ppo = PPO('MlpPolicy',env,policy_kwargs={'net_arch':[128,128]},device='cpu',
|
|
||||||
n_steps=16,batch_size=8)
|
|
||||||
bc = module.BCMlp()
|
|
||||||
state = bc.net.state_dict()
|
|
||||||
ppo.policy.mlp_extractor.policy_net.load_state_dict(
|
|
||||||
{k:v for k,v in state.items() if not k.startswith('4.')})
|
|
||||||
ppo.policy.action_net.load_state_dict({'weight':state['4.weight'],'bias':state['4.bias']})
|
|
||||||
x = torch.randn(10,401)
|
|
||||||
torch.testing.assert_close(bc(x),ppo.policy.action_net(ppo.policy.mlp_extractor.policy_net(x)))
|
|
||||||
env.close()
|
|
||||||
@@ -1,957 +0,0 @@
|
|||||||
"""训练前端回归：配置契约、进度解析、HTTP API 与真实子进程生命周期。
|
|
||||||
|
|
||||||
覆盖范围与动机：
|
|
||||||
- ``/api/train/schema`` 与 ``TrainConfig`` 必须同源（字段、默认值、预设能直接
|
|
||||||
提交），否则界面会允许后端拒绝的输入；
|
|
||||||
- SB3 表格解析必须给每一行打上 ``total_timesteps`` 横轴（曾把表格行当独立
|
|
||||||
点，曲线上出现重复的幽灵点）；
|
|
||||||
- ``metric_offset``/``metric_end`` 游标契约（前端增量拉取依赖它）；
|
|
||||||
- 用一段桩训练脚本真起子进程，验证启动/轮询/暂停/继续/停止/退出码全链路。
|
|
||||||
"""
|
|
||||||
|
|
||||||
# pytest 用参数名注入 fixture：`stub_repo`/`client` 这类参数在函数体里不直接
|
|
||||||
# 引用也是正常用法（它们的作用是 setup/teardown），这里统一放行。
|
|
||||||
# pylint: disable=unused-argument
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import dataclasses
|
|
||||||
import http.client
|
|
||||||
import json
|
|
||||||
import sys
|
|
||||||
import threading
|
|
||||||
import time
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
pytest.importorskip("mcm26b")
|
|
||||||
|
|
||||||
from mcm26b import process_control as pc
|
|
||||||
from mcm26b.training_ui import demo, server as td
|
|
||||||
from mcm26b.training_ui.schema import (CONFIG_FIELDS, config_schema,
|
|
||||||
preset_payloads)
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
|
|
||||||
# 桩训练脚本：先写 Monitor CSV，再按轮次输出"progress JSON + SB3 表格"，
|
|
||||||
# 最后打印一行元数据。全部行为由参数控制，不依赖 torch/SB3。
|
|
||||||
STUB_SCRIPT = r'''
|
|
||||||
"""训练前端测试桩：模拟 SB3 verbose=1 表格 + --progress-json 输出。"""
|
|
||||||
import argparse, json, os, signal, sys, time
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
parser = argparse.ArgumentParser()
|
|
||||||
parser.add_argument("--problem", type=int, default=3)
|
|
||||||
parser.add_argument("--stage", type=int, default=1)
|
|
||||||
parser.add_argument("--timesteps", type=int, default=96)
|
|
||||||
parser.add_argument("--model", default="out/model.zip")
|
|
||||||
parser.add_argument("--rollout-steps", type=int, default=32)
|
|
||||||
# 默认 sleep 可由环境变量控制：HTTP 端到端用例无法直接追加命令行参数，
|
|
||||||
# 用小 sleep 拉长运行窗口，才能稳定地测"暂停/继续"。
|
|
||||||
parser.add_argument("--sleep", type=float,
|
|
||||||
default=float(os.environ.get("STUB_TRAIN_SLEEP", "0")))
|
|
||||||
parser.add_argument("--lines", type=int, default=4)
|
|
||||||
parser.add_argument("--table-first", action="store_true")
|
|
||||||
parser.add_argument("--progress-json", action="store_true")
|
|
||||||
parser.add_argument("--crash", action="store_true")
|
|
||||||
parser.add_argument("--ignore-sigterm", action="store_true")
|
|
||||||
parser.add_argument("--ignore-control", action="store_true")
|
|
||||||
parser.add_argument("--control-file", default=None)
|
|
||||||
args, _extra = parser.parse_known_args()
|
|
||||||
|
|
||||||
model = Path(args.model)
|
|
||||||
model.parent.mkdir(parents=True, exist_ok=True)
|
|
||||||
monitor = model.parent / "high_curve_q3_env0.monitor.csv"
|
|
||||||
with monitor.open("w", encoding="utf-8") as handle:
|
|
||||||
handle.write("#{\"t_start\": 0.0}\n")
|
|
||||||
handle.write("r,l,t\n")
|
|
||||||
for index in range(3):
|
|
||||||
handle.write("%.3f,%d,%.3f\n" % (float(index), 10 + index, index * 5.0))
|
|
||||||
|
|
||||||
if args.ignore_sigterm:
|
|
||||||
signal.signal(signal.SIGTERM, signal.SIG_IGN)
|
|
||||||
|
|
||||||
print("NORMAL-LOG-LINE")
|
|
||||||
sys.stdout.flush()
|
|
||||||
|
|
||||||
|
|
||||||
def read_control():
|
|
||||||
if not args.control_file or args.ignore_control:
|
|
||||||
return {}
|
|
||||||
try:
|
|
||||||
with open(args.control_file, encoding="utf-8") as handle:
|
|
||||||
return json.load(handle)
|
|
||||||
except Exception:
|
|
||||||
return {}
|
|
||||||
|
|
||||||
|
|
||||||
def control_pause_or_stop(step):
|
|
||||||
"""返回 True 表示应退出；暂停时阻塞到继续或停止（模拟协作式暂停）。"""
|
|
||||||
state = read_control()
|
|
||||||
if state.get("stop"):
|
|
||||||
print(json.dumps({"event": "stop_requested", "signal": "control_file"}), flush=True)
|
|
||||||
return True
|
|
||||||
if state.get("paused"):
|
|
||||||
print(json.dumps({"event": "paused", "timesteps": step}), flush=True)
|
|
||||||
while True:
|
|
||||||
time.sleep(0.05)
|
|
||||||
state = read_control()
|
|
||||||
if state.get("stop"):
|
|
||||||
print(json.dumps({"event": "stop_requested", "signal": "control_file"}),
|
|
||||||
flush=True)
|
|
||||||
return True
|
|
||||||
if not state.get("paused"):
|
|
||||||
break
|
|
||||||
print(json.dumps({"event": "resumed", "timesteps": step}), flush=True)
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
def sleep_with_control(seconds, step):
|
|
||||||
"""把长 sleep 切成 0.1s 片，保证停止/暂停请求能被及时响应。"""
|
|
||||||
deadline = time.monotonic() + max(0.0, seconds)
|
|
||||||
while True:
|
|
||||||
if control_pause_or_stop(step):
|
|
||||||
raise SystemExit(0)
|
|
||||||
remaining = deadline - time.monotonic()
|
|
||||||
if remaining <= 0:
|
|
||||||
return
|
|
||||||
time.sleep(min(0.1, remaining))
|
|
||||||
|
|
||||||
|
|
||||||
def render_table(step, fps):
|
|
||||||
return (
|
|
||||||
"-----------------------------\n"
|
|
||||||
"| rollout/ | |\n"
|
|
||||||
"| ep_len_mean | 9.67 |\n"
|
|
||||||
"| ep_rew_mean | -3.24 |\n"
|
|
||||||
"| time/ | |\n"
|
|
||||||
f"| fps | {fps:<8d} |\n"
|
|
||||||
f"| total_timesteps | {step:<8d} |\n"
|
|
||||||
"-----------------------------\n"
|
|
||||||
)
|
|
||||||
|
|
||||||
|
|
||||||
for index in range(args.lines):
|
|
||||||
step = 32 * (index + 1)
|
|
||||||
sleep_with_control(args.sleep, step)
|
|
||||||
fps = 17 + index
|
|
||||||
progress = {
|
|
||||||
"event": "progress", "timesteps": step, "n_updates": index + 1,
|
|
||||||
"ep_rew_mean": -3.2 + index, "ep_len_mean": 9.7,
|
|
||||||
"value_loss": 12.5 - index, "approx_kl": 0.01,
|
|
||||||
"entropy_loss": 1.1, "explained_variance": 0.2, "fps": float(fps),
|
|
||||||
}
|
|
||||||
payload = "PROGRESS-" + json.dumps(progress)
|
|
||||||
table = render_table(step, fps)
|
|
||||||
if args.table_first:
|
|
||||||
print(table, end="")
|
|
||||||
print(payload, flush=True)
|
|
||||||
else:
|
|
||||||
print(payload, flush=True)
|
|
||||||
print(table, end="", flush=True)
|
|
||||||
print("EPISODE-%d r=%.2f" % (index, float(index)), flush=True)
|
|
||||||
|
|
||||||
print("INFO " + json.dumps({"event": "metadata", "actual_timesteps": args.timesteps}))
|
|
||||||
if args.crash:
|
|
||||||
raise SystemExit(3)
|
|
||||||
raise SystemExit(0)
|
|
||||||
'''
|
|
||||||
|
|
||||||
|
|
||||||
def wait_until(condition, timeout=15.0, interval=0.05):
|
|
||||||
deadline = time.monotonic() + timeout
|
|
||||||
while time.monotonic() < deadline:
|
|
||||||
if condition():
|
|
||||||
return True
|
|
||||||
time.sleep(interval)
|
|
||||||
return False
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.fixture()
|
|
||||||
def stub_repo(tmp_path, monkeypatch):
|
|
||||||
"""把 TRAIN_SCRIPT 指到桩脚本，避免测试真的训练神经网络。"""
|
|
||||||
script = tmp_path / "stub_train.py"
|
|
||||||
script.write_text(STUB_SCRIPT, encoding="utf-8")
|
|
||||||
monkeypatch.setattr(td, "TRAIN_SCRIPT", script)
|
|
||||||
return script
|
|
||||||
|
|
||||||
|
|
||||||
def make_config(**overrides):
|
|
||||||
payload = {"problem": 3, "stage": 1, "timesteps": 96, "seed": 1,
|
|
||||||
"envs": 1, "vec_env": "dummy", "device": "cpu",
|
|
||||||
"config_profile": "coarse-pretrain", "reward_profile": "time",
|
|
||||||
"model": "output/hrl/ui/test/q3.zip", "eval_episodes": 0,
|
|
||||||
"rollout_steps": 32, "batch_size": 32, "n_epochs": 2,
|
|
||||||
"max_wall_s": 300.0, "verbose": 1, "force": True}
|
|
||||||
payload.update(overrides)
|
|
||||||
return td.TrainConfig.from_payload(payload)
|
|
||||||
|
|
||||||
|
|
||||||
def run_job(config, timeout=30.0):
|
|
||||||
job = td.TrainingJob(config)
|
|
||||||
job.start()
|
|
||||||
assert wait_until(lambda: job.status in ("exited", "failed", "stopped"),
|
|
||||||
timeout=timeout), f"子进程未在 {timeout}s 内结束: {job.status}"
|
|
||||||
return job
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- 配置契约
|
|
||||||
def test_schema_matches_train_config_defaults():
|
|
||||||
schema = config_schema()
|
|
||||||
assert schema["fields"] and schema["groups"] and schema["metrics"]
|
|
||||||
defaults = td.TrainConfig()
|
|
||||||
for field in schema["fields"]:
|
|
||||||
assert hasattr(defaults, field["key"]), f"字段 {field['key']} 不在 TrainConfig 中"
|
|
||||||
assert field["default"] == getattr(defaults, field["key"]), \
|
|
||||||
f"{field['key']} 默认值与后端不一致"
|
|
||||||
exposed = {field["key"] for field in schema["fields"]}
|
|
||||||
missing = {field.name for field in dataclasses.fields(defaults)} - exposed
|
|
||||||
assert not missing, f"TrainConfig 字段未暴露给界面: {sorted(missing)}"
|
|
||||||
groups = {group["key"] for group in schema["groups"]}
|
|
||||||
for field in schema["fields"]:
|
|
||||||
assert field["group"] in groups, f"{field['key']} 的分组不存在"
|
|
||||||
|
|
||||||
|
|
||||||
def test_schema_metric_catalog_covers_progress_payload():
|
|
||||||
"""进度 JSON 的每个键都要么在指标字典里，要么有明确理由不进图。"""
|
|
||||||
emitted = set(demo.demo_metrics(count=2)[0]) | {"ep_len_mean"}
|
|
||||||
catalog = {metric["key"] for metric in config_schema()["metrics"]}
|
|
||||||
known_extras = {"timesteps", "total_timesteps", "iterations", "time_elapsed"}
|
|
||||||
assert emitted <= catalog | known_extras, sorted(emitted - catalog - known_extras)
|
|
||||||
|
|
||||||
|
|
||||||
def test_presets_are_accepted_by_backend():
|
|
||||||
for key, payload in preset_payloads().items():
|
|
||||||
config = td.TrainConfig.from_payload(payload)
|
|
||||||
assert config.timesteps > 0, key
|
|
||||||
command = td.build_command(config)
|
|
||||||
assert "--progress-json" in command and "--model" in command
|
|
||||||
|
|
||||||
|
|
||||||
def test_config_rejects_out_of_range_and_unknown_combinations():
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(problem=5)
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(stage=9)
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(rollout_steps=4096) # > timesteps
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(resume=True, force=True) # 互斥
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(model="")
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(vec_env="joblib")
|
|
||||||
|
|
||||||
|
|
||||||
def test_config_accepts_paths_with_separators():
|
|
||||||
r"""回归：_SAFE_ARG_RE 曾把 \x00 写成字面反斜杠，导致所有相对路径被拒。"""
|
|
||||||
config = make_config(model="output/hrl/ui/smoke/q3.zip",
|
|
||||||
init="output/hrl/ui/q3-main/q3.zip")
|
|
||||||
assert config.model_path.is_absolute()
|
|
||||||
assert config.model_path.as_posix().endswith("output/hrl/ui/smoke/q3.zip")
|
|
||||||
assert "--init" in td.build_command(config)
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(model="bad\x01name.zip") # 控制字符仍须拒绝
|
|
||||||
|
|
||||||
|
|
||||||
def test_build_command_is_shell_free_argument_list():
|
|
||||||
config = make_config(model="output/x/q3.zip")
|
|
||||||
command = td.build_command(config)
|
|
||||||
assert all(isinstance(item, str) for item in command)
|
|
||||||
assert str(td.TRAIN_SCRIPT) in command
|
|
||||||
assert "output/x/q3.zip" in " ".join(command)
|
|
||||||
# 参数列表不经过 shell：没有把整条命令拼成一个字符串
|
|
||||||
assert not any(" --" in item for item in command)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- 进度解析
|
|
||||||
class _FakeJob:
|
|
||||||
"""借用 TrainingJob 的解析器，但不启动进程、不需要真实配置。"""
|
|
||||||
|
|
||||||
def __init__(self):
|
|
||||||
import collections
|
|
||||||
self.logs = collections.deque(maxlen=100)
|
|
||||||
self.metrics = collections.deque(maxlen=100)
|
|
||||||
self._lock = threading.RLock()
|
|
||||||
self._table_sections = []
|
|
||||||
self._table_steps = None
|
|
||||||
self._pending_table = None
|
|
||||||
self._last_table_steps = None
|
|
||||||
self._last_json_steps = None
|
|
||||||
self._last_monitor_sig = None
|
|
||||||
self.status = "running"
|
|
||||||
self.config = make_config()
|
|
||||||
|
|
||||||
_on_line = td.TrainingJob._on_line
|
|
||||||
_sync_control_events = td.TrainingJob._sync_control_events
|
|
||||||
_parse_progress_json = td.TrainingJob._parse_progress_json
|
|
||||||
_parse_sb3_table = td.TrainingJob._parse_sb3_table
|
|
||||||
_flush_table = td.TrainingJob._flush_table
|
|
||||||
_flush_pending_with_fallback = td.TrainingJob._flush_pending_with_fallback
|
|
||||||
_append_metric = td.TrainingJob._append_metric
|
|
||||||
_refresh_monitor = td.TrainingJob._refresh_monitor
|
|
||||||
|
|
||||||
|
|
||||||
def test_sb3_table_rows_get_timestep_axis():
|
|
||||||
"""表格行必须继承所属表的 total_timesteps，不能退化为点序号横轴。
|
|
||||||
|
|
||||||
刻意不打印表间分隔线，复刻"只有段头、没有横线"的真实输出变体。
|
|
||||||
"""
|
|
||||||
job = _FakeJob()
|
|
||||||
for line in ("| rollout/ | |",
|
|
||||||
"| ep_len_mean | 9.67 |",
|
|
||||||
"| ep_rew_mean | -3.24 |",
|
|
||||||
"| time/ | |",
|
|
||||||
"| fps | 17 |",
|
|
||||||
"| total_timesteps | 32 |",
|
|
||||||
"| rollout/ | |",
|
|
||||||
"| ep_rew_mean | -2.90 |",
|
|
||||||
"| time/ | |",
|
|
||||||
"| fps | 19 |",
|
|
||||||
"| total_timesteps | 64 |",
|
|
||||||
"-----------------------------"):
|
|
||||||
job._on_line(line)
|
|
||||||
job._flush_table()
|
|
||||||
assert [point["timesteps"] for point in job.metrics] == [32, 64], list(job.metrics)
|
|
||||||
# 每张表合并为一条（rollout/ 与 time/ 的字段都在同一条里），刻度取本表
|
|
||||||
assert job.metrics[0]["ep_rew_mean"] == -3.24
|
|
||||||
assert job.metrics[0]["fps"] == 17
|
|
||||||
assert job.metrics[1]["ep_rew_mean"] == -2.9
|
|
||||||
assert job.metrics[1]["fps"] == 19
|
|
||||||
assert all(point["timesteps"] != float(index) for index, point in
|
|
||||||
enumerate(job.metrics)), "横轴不得退化为点序号"
|
|
||||||
|
|
||||||
|
|
||||||
def test_sb3_table_with_separators_emits_one_point_per_table():
|
|
||||||
"""带分隔线的标准 SB3 输出：每张表恰好一条进度点，刻度取自本表。"""
|
|
||||||
job = _FakeJob()
|
|
||||||
for index, steps in enumerate((32, 64, 96)):
|
|
||||||
job._on_line("-----------------------------")
|
|
||||||
job._on_line("| rollout/ | |")
|
|
||||||
job._on_line(f"| ep_rew_mean | {-3.0 + index:.2f} |")
|
|
||||||
job._on_line("| time/ | |")
|
|
||||||
job._on_line(f"| total_timesteps | {steps} |")
|
|
||||||
job._on_line("-----------------------------")
|
|
||||||
job._flush_table()
|
|
||||||
assert [point["timesteps"] for point in job.metrics] == [32, 64, 96], list(job.metrics)
|
|
||||||
assert [point["ep_rew_mean"] for point in job.metrics] == [-3.0, -2.0, -1.0]
|
|
||||||
|
|
||||||
|
|
||||||
def test_truncated_table_falls_back_to_last_known_axis():
|
|
||||||
"""被截断的表（缺少 time/ 段）不用旧刻度冒充，挂起到进程结束才兜底提交。"""
|
|
||||||
job = _FakeJob()
|
|
||||||
for line in ("| rollout/ | |",
|
|
||||||
"| ep_rew_mean | -3.24 |",
|
|
||||||
"| time/ | |",
|
|
||||||
"| total_timesteps | 32 |",
|
|
||||||
"-----------------------------",
|
|
||||||
"| rollout/ | |",
|
|
||||||
"| ep_rew_mean | -2.90 |",
|
|
||||||
"-----------------------------"):
|
|
||||||
job._on_line(line)
|
|
||||||
job._flush_table()
|
|
||||||
assert [point["timesteps"] for point in job.metrics] == [32], list(job.metrics)
|
|
||||||
job._flush_pending_with_fallback()
|
|
||||||
assert [point["timesteps"] for point in job.metrics] == [32, 32], list(job.metrics)
|
|
||||||
assert job.metrics[-1]["ep_rew_mean"] == -2.9
|
|
||||||
|
|
||||||
|
|
||||||
def test_table_without_any_axis_is_discarded():
|
|
||||||
"""完全没有刻度来源的表宁可丢弃，也不能让曲线出现序号横轴的幽灵点。"""
|
|
||||||
job = _FakeJob()
|
|
||||||
for line in ("| rollout/ | |",
|
|
||||||
"| ep_rew_mean | -3.24 |",
|
|
||||||
"-----------------------------"):
|
|
||||||
job._on_line(line)
|
|
||||||
job._flush_table()
|
|
||||||
job._flush_pending_with_fallback()
|
|
||||||
assert not job.metrics
|
|
||||||
|
|
||||||
|
|
||||||
def test_progress_json_is_not_deduped_against_table_by_timestep():
|
|
||||||
"""同一 timesteps 的表格行与 JSON 进度行只保留一份，避免重复点。"""
|
|
||||||
job = _FakeJob()
|
|
||||||
job._on_line('{"event": "progress", "timesteps": 32, "ep_rew_mean": -1.0}')
|
|
||||||
for line in ("| rollout/ | |",
|
|
||||||
"| ep_rew_mean | -1.0 |",
|
|
||||||
"| total_timesteps | 32 |"):
|
|
||||||
job._on_line(line)
|
|
||||||
job._flush_table()
|
|
||||||
assert len(job.metrics) == 1, list(job.metrics)
|
|
||||||
|
|
||||||
|
|
||||||
def test_metadata_json_is_not_treated_as_progress():
|
|
||||||
"""收尾打印的模型元数据不带 event=progress，不能污染曲线。"""
|
|
||||||
job = _FakeJob()
|
|
||||||
job._on_line(json.dumps({"level": "high", "actual_timesteps": 96,
|
|
||||||
"fps": 999, "value_loss": 12345}))
|
|
||||||
assert not job.metrics
|
|
||||||
|
|
||||||
|
|
||||||
def test_convergence_states_are_distinguishable():
|
|
||||||
def point(step, reward, kl=0.01, ev=0.5, fps=100.0):
|
|
||||||
return {"timesteps": step, "ep_rew_mean": reward, "approx_kl": kl,
|
|
||||||
"explained_variance": ev, "fps": fps}
|
|
||||||
|
|
||||||
empty = td.summarize_convergence([], [])
|
|
||||||
assert empty["tone"] == "gray" and empty["reward_points"] == 0
|
|
||||||
|
|
||||||
rising = td.summarize_convergence(
|
|
||||||
[point(index * 100, 1.0 + index * 0.5) for index in range(1, 15)], [])
|
|
||||||
assert rising["tone"] == "ok"
|
|
||||||
assert "斜率为正" in " ".join(rising["notes"])
|
|
||||||
|
|
||||||
plateau = td.summarize_convergence(
|
|
||||||
[point(index * 100, 5.0) for index in range(1, 15)], [])
|
|
||||||
assert "平稳" in plateau["status"], plateau
|
|
||||||
assert plateau["reward_spread"] == 0.0
|
|
||||||
|
|
||||||
unstable = td.summarize_convergence(
|
|
||||||
[point(index * 100, 1.0 * index, kl=0.4) for index in range(1, 15)], [])
|
|
||||||
assert unstable["tone"] == "bad" and "KL" in unstable["status"]
|
|
||||||
|
|
||||||
unfitted = td.summarize_convergence(
|
|
||||||
[point(index * 100, 1.0 * index, ev=-0.3) for index in range(1, 15)], [])
|
|
||||||
assert unfitted["tone"] == "warn"
|
|
||||||
|
|
||||||
monitor_only = td.summarize_convergence(
|
|
||||||
[], [{"r": 1.0}, {"r": 2.0}, {"r": 3.0}])
|
|
||||||
assert monitor_only["reward_points"] == 3
|
|
||||||
|
|
||||||
missing = [dict(point(index * 100, 1.0 * index), approx_kl=None,
|
|
||||||
explained_variance=None) for index in range(1, 9)]
|
|
||||||
still_ok = td.summarize_convergence(missing, [])
|
|
||||||
assert still_ok["latest_kl"] is None and still_ok["tone"] == "ok"
|
|
||||||
|
|
||||||
|
|
||||||
def test_describe_process_progress_and_eta():
|
|
||||||
snapshot = {
|
|
||||||
"status": "running", "elapsed_s": 10.0,
|
|
||||||
"config": {"timesteps": 4000, "max_wall_s": 900.0},
|
|
||||||
"metrics": [{"timesteps": 500, "fps": 50.0},
|
|
||||||
{"timesteps": 1000, "fps": 100.0}],
|
|
||||||
"convergence": {}, "log_total": 3, "metric_total": 2,
|
|
||||||
}
|
|
||||||
process = td.describe_process(snapshot)
|
|
||||||
assert process["percent"] == 25.0
|
|
||||||
assert process["done_steps"] == 1000
|
|
||||||
assert process["rate_per_s"] == 75.0
|
|
||||||
assert process["remaining_s"] == pytest.approx(40.0, rel=1e-6)
|
|
||||||
assert process["phase"] == "update"
|
|
||||||
assert [node["key"] for node in process["timeline"]][-1] == "done"
|
|
||||||
assert process["timeline"][-1]["state"] == "pending"
|
|
||||||
|
|
||||||
# 无进度点时退化为墙钟上限估计，而不是给出 None
|
|
||||||
bare = td.describe_process({"status": "running", "elapsed_s": 30.0,
|
|
||||||
"config": {"timesteps": 1000, "max_wall_s": 100.0},
|
|
||||||
"metrics": [], "convergence": {}})
|
|
||||||
assert bare["percent"] == 0.0
|
|
||||||
assert bare["remaining_s"] == pytest.approx(70.0)
|
|
||||||
|
|
||||||
# fps 只出现在 SB3 表格行里，中间夹着若干无 fps 的 JSON 行时仍要能估速率
|
|
||||||
gapped = td.describe_process({
|
|
||||||
"status": "running", "elapsed_s": 30.0,
|
|
||||||
"config": {"timesteps": 640, "max_wall_s": 300.0},
|
|
||||||
"metrics": [{"timesteps": 100, "fps": None}, {"timesteps": 120, "fps": 18.0},
|
|
||||||
{"timesteps": 140, "fps": None}, {"timesteps": 160, "fps": None},
|
|
||||||
{"timesteps": 180, "fps": None}, {"timesteps": 200, "fps": None},
|
|
||||||
{"timesteps": 220, "fps": 20.0}],
|
|
||||||
"convergence": {}})
|
|
||||||
assert gapped["rate_per_s"] == 19.0, gapped["rate_per_s"]
|
|
||||||
assert gapped["remaining_s"] == pytest.approx(420 / 19.0, rel=1e-6)
|
|
||||||
|
|
||||||
# 完全没有 fps 时退化为墙钟上限估计
|
|
||||||
no_fps = td.describe_process(
|
|
||||||
{"status": "running", "elapsed_s": 30.0,
|
|
||||||
"config": {"timesteps": 1000, "max_wall_s": 100.0},
|
|
||||||
"metrics": [{"timesteps": 10}, {"timesteps": 20}], "convergence": {}})
|
|
||||||
assert no_fps["rate_per_s"] is None
|
|
||||||
assert no_fps["remaining_s"] == pytest.approx(70.0)
|
|
||||||
|
|
||||||
# 已结束：时间线最后节点为 active，且不再外推剩余时间
|
|
||||||
ended = td.describe_process({"status": "exited", "elapsed_s": 50.0,
|
|
||||||
"config": {"timesteps": 1000, "max_wall_s": 100.0},
|
|
||||||
"metrics": [{"timesteps": 1000, "fps": 10.0}],
|
|
||||||
"convergence": {}})
|
|
||||||
assert ended["terminal"] is True
|
|
||||||
assert ended["timeline"][-1]["state"] == "active"
|
|
||||||
assert ended["remaining_s"] is None
|
|
||||||
|
|
||||||
|
|
||||||
def test_describe_process_curriculum_transfer_uses_delta_steps():
|
|
||||||
"""--init 课程迁移：计数器从上一阶段继续，进度/ETA 必须按本次增量算。"""
|
|
||||||
snapshot = {
|
|
||||||
"status": "running", "elapsed_s": 10.0,
|
|
||||||
"config": {"timesteps": 8000, "max_wall_s": 3600.0},
|
|
||||||
"metrics": [{"timesteps": 110000, "start_timesteps": 109776, "fps": 50.0},
|
|
||||||
{"timesteps": 111776, "start_timesteps": 109776, "fps": 50.0}],
|
|
||||||
"convergence": {}, "log_total": 2, "metric_total": 2,
|
|
||||||
}
|
|
||||||
process = td.describe_process(snapshot)
|
|
||||||
assert process["done_steps"] == 111776 # 绝对计数照实显示
|
|
||||||
assert process["start_steps"] == 109776
|
|
||||||
assert process["trained_steps"] == 2000 # 本次增量
|
|
||||||
assert process["percent"] == 25.0 # 而不是被夹到 100%
|
|
||||||
assert process["remaining_s"] == pytest.approx(120.0, rel=1e-6)
|
|
||||||
|
|
||||||
|
|
||||||
def test_demo_snapshot_contract():
|
|
||||||
for scenario in demo.DEMO_SCENARIOS:
|
|
||||||
snapshot = demo.demo_snapshot(scenario=scenario)
|
|
||||||
job = td.TrainingJob(make_config())
|
|
||||||
# 示例快照必须与真实快照同键，否则前端在预览模式下会缺字段
|
|
||||||
for key in job.snapshot(log_tail=1):
|
|
||||||
assert key in snapshot, f"{scenario} 缺少快照键 {key}"
|
|
||||||
assert snapshot["process"]["percent"] < 100.0
|
|
||||||
assert snapshot["convergence"]["reward_points"] > 0
|
|
||||||
assert json.dumps(snapshot, ensure_ascii=False, allow_nan=False)
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
demo.demo_metrics(scenario="nope")
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- 子进程
|
|
||||||
def test_job_lifecycle_with_stub_script(stub_repo):
|
|
||||||
config = make_config(model="output/hrl/ui/test/live/q3.zip",
|
|
||||||
verbose=1, force=True)
|
|
||||||
job = run_job(config)
|
|
||||||
assert job.status == "exited" and job.returncode == 0
|
|
||||||
snapshot = job.snapshot(log_tail=50)
|
|
||||||
assert snapshot["status"] == "exited"
|
|
||||||
assert snapshot["process"]["percent"] == 100.0
|
|
||||||
assert snapshot["process"]["terminal"] is True
|
|
||||||
assert any("NORMAL-LOG-LINE" in line for line in snapshot["log_lines"])
|
|
||||||
assert len(snapshot["monitor"]) == 3
|
|
||||||
steps = [point.get("timesteps") for point in snapshot["metrics"]]
|
|
||||||
assert steps == [32, 64, 96, 128], steps
|
|
||||||
assert snapshot["metric_end"] == len(snapshot["metrics"]) + snapshot["metric_offset"]
|
|
||||||
assert snapshot["convergence"]["reward_points"] >= 1
|
|
||||||
|
|
||||||
|
|
||||||
def test_job_reports_nonzero_exit_code(stub_repo):
|
|
||||||
config = make_config(model="output/hrl/ui/test/crash/q3.zip")
|
|
||||||
job = td.TrainingJob(config)
|
|
||||||
job.command = list(job.command) + ["--crash"]
|
|
||||||
job.start()
|
|
||||||
assert wait_until(lambda: job.status in ("exited", "failed"), timeout=30)
|
|
||||||
assert job.status == "failed"
|
|
||||||
assert job.returncode == 3
|
|
||||||
assert job.snapshot(log_tail=5)["process"]["tone"] == "bad"
|
|
||||||
|
|
||||||
|
|
||||||
def test_job_command_carries_control_file(stub_repo):
|
|
||||||
"""控制文件路径必须随命令下发，否则暂停/停止退化成超时强杀。"""
|
|
||||||
config = make_config(model="output/hrl/ui/test/control/q3.zip")
|
|
||||||
job = td.TrainingJob(config, control_dir=Path(stub_repo).parent)
|
|
||||||
assert "--control-file" in job.command
|
|
||||||
assert str(job.control_path) in job.command
|
|
||||||
assert job.control_path.parent == Path(stub_repo).parent
|
|
||||||
|
|
||||||
|
|
||||||
def test_stop_gracefully_marks_stopped_without_forced_kill(stub_repo):
|
|
||||||
config = make_config(model="output/hrl/ui/test/stop/q3.zip")
|
|
||||||
job = td.TrainingJob(config)
|
|
||||||
job.command = list(job.command) + ["--sleep", "5", "--lines", "50"]
|
|
||||||
job.start()
|
|
||||||
assert wait_until(lambda: job.status == "running", timeout=10)
|
|
||||||
assert job.stop() is True
|
|
||||||
assert wait_until(lambda: job.status in ("stopped", "failed"), timeout=15)
|
|
||||||
assert job.status == "stopped"
|
|
||||||
# 桩脚本读控制文件后主动退出：不应走到强杀分支
|
|
||||||
assert job.forced_kill is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_pause_and_resume_via_control_file(stub_repo):
|
|
||||||
config = make_config(model="output/hrl/ui/test/pause/q3.zip")
|
|
||||||
job = td.TrainingJob(config)
|
|
||||||
job.command = list(job.command) + ["--sleep", "0.2", "--lines", "60"]
|
|
||||||
job.start()
|
|
||||||
assert wait_until(lambda: job.status == "running", timeout=10)
|
|
||||||
assert job.pause() is True
|
|
||||||
# 训练脚本回显 paused 事件；状态由 _sync_control_events 校正
|
|
||||||
assert wait_until(
|
|
||||||
lambda: any('"event": "paused"' in line for line in job.snapshot()["log_lines"]),
|
|
||||||
timeout=10), "训练脚本未回显 paused 事件"
|
|
||||||
assert job.status == "paused"
|
|
||||||
paused_metrics = len(job.snapshot()["metrics"])
|
|
||||||
time.sleep(0.6)
|
|
||||||
assert len(job.snapshot()["metrics"]) <= paused_metrics + 1, "暂停后仍在产出指标"
|
|
||||||
assert job.resume() is True
|
|
||||||
assert wait_until(
|
|
||||||
lambda: any('"event": "resumed"' in line for line in job.snapshot()["log_lines"]),
|
|
||||||
timeout=10), "训练脚本未回显 resumed 事件"
|
|
||||||
assert wait_until(lambda: job.status == "running", timeout=5)
|
|
||||||
assert job.stop() is True
|
|
||||||
assert wait_until(lambda: job.status in ("stopped", "failed"), timeout=15)
|
|
||||||
assert job.status == "stopped"
|
|
||||||
|
|
||||||
|
|
||||||
def test_stop_forced_kill_when_control_ignored(stub_repo, monkeypatch):
|
|
||||||
"""训练进程不读控制文件时必须超时强杀，并留下恢复指引。"""
|
|
||||||
monkeypatch.setattr(td, "STOP_GRACE_S", 0.5)
|
|
||||||
config = make_config(model="output/hrl/ui/test/stubborn/q3.zip")
|
|
||||||
job = td.TrainingJob(config)
|
|
||||||
job.command = list(job.command) + ["--ignore-control", "--sleep", "5", "--lines", "50"]
|
|
||||||
job.start()
|
|
||||||
assert wait_until(lambda: job.status == "running", timeout=10)
|
|
||||||
# 造一份"磁盘上已有滚动恢复点"的现场，强杀后应被记录进恢复指引
|
|
||||||
ckpt_dir = job.config.model_path.parent / "checkpoints"
|
|
||||||
ckpt_dir.mkdir(parents=True, exist_ok=True)
|
|
||||||
latest = ckpt_dir / "q3_ckpt_latest.zip"
|
|
||||||
latest.write_bytes(b"PK\x03\x04fake")
|
|
||||||
assert job.stop() is True
|
|
||||||
assert wait_until(lambda: job.status in ("stopped", "failed"), timeout=20)
|
|
||||||
assert job.forced_kill is True
|
|
||||||
assert job.snapshot(log_tail=5)["forced_kill"] is True
|
|
||||||
marker = json.loads(
|
|
||||||
(job.config.model_path.parent / "forced_stop.json").read_text(encoding="utf-8"))
|
|
||||||
assert marker["event"] == "forced_stop"
|
|
||||||
assert marker["reason"] == "grace_timeout"
|
|
||||||
assert marker["recovery"]["model"] == str(latest)
|
|
||||||
assert marker["recovery"]["can_resume"] is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_discover_recovery_prefers_resume_hint(tmp_path):
|
|
||||||
run_dir = tmp_path / "run"
|
|
||||||
(run_dir / "checkpoints").mkdir(parents=True)
|
|
||||||
latest = run_dir / "checkpoints" / "q3_ckpt_latest.zip"
|
|
||||||
latest.write_bytes(b"PK\x03\x04fake")
|
|
||||||
final = run_dir / "q3.zip"
|
|
||||||
final.write_bytes(b"PK\x03\x04fake")
|
|
||||||
(run_dir / "resume.json").write_text(json.dumps({
|
|
||||||
"model": str(latest), "timesteps": 128, "reason": "interval",
|
|
||||||
"created_unix_s": 5.0}), encoding="utf-8")
|
|
||||||
state = td.discover_recovery(run_dir, final)
|
|
||||||
assert state["can_resume"] is True
|
|
||||||
assert state["model"] == str(latest)
|
|
||||||
assert state["timesteps"] == 128
|
|
||||||
assert state["source"] == "resume.json"
|
|
||||||
|
|
||||||
|
|
||||||
def test_discover_recovery_falls_back_to_scan_on_dangling_hint(tmp_path):
|
|
||||||
run_dir = tmp_path / "run"
|
|
||||||
(run_dir / "checkpoints").mkdir(parents=True)
|
|
||||||
latest = run_dir / "checkpoints" / "q3_ckpt_latest.zip"
|
|
||||||
latest.write_bytes(b"PK\x03\x04fake")
|
|
||||||
# 悬空指针（目标文件已不在）必须回退到扫描，而不是给出一个打不开的路径
|
|
||||||
(run_dir / "resume.json").write_text(
|
|
||||||
json.dumps({"model": str(run_dir / "gone.zip")}), encoding="utf-8")
|
|
||||||
state = td.discover_recovery(run_dir, run_dir / "q3.zip")
|
|
||||||
assert state["can_resume"] is True
|
|
||||||
assert state["model"] == str(latest)
|
|
||||||
assert state["source"] == "scan"
|
|
||||||
|
|
||||||
|
|
||||||
def test_discover_recovery_with_nothing_on_disk(tmp_path):
|
|
||||||
state = td.discover_recovery(tmp_path, tmp_path / "q3.zip")
|
|
||||||
assert state["can_resume"] is False and state["model"] is None
|
|
||||||
assert state["reason"] == "none"
|
|
||||||
|
|
||||||
|
|
||||||
def test_command_carries_checkpoint_and_recovery_flags():
|
|
||||||
config = make_config(model="output/x/q3.zip", recovery_every=100,
|
|
||||||
checkpoint_every=500, checkpoint_keep=2)
|
|
||||||
joined = " ".join(td.build_command(config))
|
|
||||||
assert "--recovery-every 100" in joined
|
|
||||||
assert "--checkpoint-every 500" in joined
|
|
||||||
assert "--checkpoint-keep 2" in joined
|
|
||||||
|
|
||||||
|
|
||||||
def test_config_rejects_bad_recovery_settings():
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(recovery_every=-1)
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(checkpoint_every=-1)
|
|
||||||
with pytest.raises(ValueError):
|
|
||||||
make_config(checkpoint_keep=0)
|
|
||||||
|
|
||||||
|
|
||||||
def test_pause_resume_return_false_without_live_process(stub_repo):
|
|
||||||
config = make_config(model="output/hrl/ui/test/nolive/q3.zip")
|
|
||||||
job = td.TrainingJob(config)
|
|
||||||
assert job.pause() is False and job.resume() is False
|
|
||||||
job.start()
|
|
||||||
assert wait_until(lambda: job.status in ("exited", "failed"), timeout=30)
|
|
||||||
assert job.pause() is False and job.resume() is False
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------- 跨平台进程控制
|
|
||||||
def test_control_file_roundtrip_and_corruption(tmp_path):
|
|
||||||
path = tmp_path / "control.json"
|
|
||||||
assert pc.read_control(path) == pc.ControlState() # 缺失 = 无指令
|
|
||||||
pc.write_control(path, paused=True)
|
|
||||||
assert pc.read_control(path).paused is True
|
|
||||||
pc.write_control(path, stop=True)
|
|
||||||
state = pc.read_control(path)
|
|
||||||
assert state.stop is True and state.paused is False
|
|
||||||
path.write_text("{ 半截 JSON", encoding="utf-8")
|
|
||||||
assert pc.read_control(path) == pc.ControlState() # 损坏 = 无指令
|
|
||||||
|
|
||||||
|
|
||||||
def test_spawn_kwargs_isolates_child_process_group(monkeypatch):
|
|
||||||
monkeypatch.setattr(pc, "IS_WINDOWS", False)
|
|
||||||
assert pc.spawn_kwargs() == {"start_new_session": True}
|
|
||||||
monkeypatch.setattr(pc, "IS_WINDOWS", True)
|
|
||||||
kwargs = pc.spawn_kwargs()
|
|
||||||
# Windows 上 start_new_session 被 CPython 静默忽略，绝不能再用它
|
|
||||||
assert "start_new_session" not in kwargs
|
|
||||||
if kwargs:
|
|
||||||
assert kwargs["creationflags"] == pc._CREATE_NEW_PROCESS_GROUP
|
|
||||||
|
|
||||||
|
|
||||||
def test_hard_kill_on_windows_uses_taskkill(monkeypatch):
|
|
||||||
commands = []
|
|
||||||
|
|
||||||
class _FakeProcess:
|
|
||||||
pid = 4242
|
|
||||||
|
|
||||||
def kill(self): # pragma: no cover - Windows 分支不应走到这里
|
|
||||||
commands.append("Popen.kill")
|
|
||||||
|
|
||||||
monkeypatch.setattr(pc, "IS_WINDOWS", True)
|
|
||||||
monkeypatch.setattr(pc.subprocess, "run",
|
|
||||||
lambda argv, **kwargs: commands.append(argv))
|
|
||||||
pc._hard_kill_tree(_FakeProcess())
|
|
||||||
assert commands and commands[0][0] == "taskkill"
|
|
||||||
assert "/PID" in commands[0] and "4242" in commands[0]
|
|
||||||
|
|
||||||
|
|
||||||
def test_hard_kill_without_killpg_does_not_raise(monkeypatch):
|
|
||||||
"""回归：Windows 上 os 没有 killpg/getpgid，旧代码会抛 AttributeError。"""
|
|
||||||
killed = []
|
|
||||||
|
|
||||||
class _FakeProcess:
|
|
||||||
pid = 4242
|
|
||||||
|
|
||||||
def kill(self):
|
|
||||||
killed.append(True)
|
|
||||||
|
|
||||||
class _PosixlessOS:
|
|
||||||
"""模拟 Windows：只有 getpid 之类，没有 killpg/getpgid。"""
|
|
||||||
|
|
||||||
monkeypatch.setattr(pc, "IS_WINDOWS", False)
|
|
||||||
monkeypatch.setattr(pc, "os", _PosixlessOS())
|
|
||||||
pc._hard_kill_tree(_FakeProcess())
|
|
||||||
assert killed == [True]
|
|
||||||
|
|
||||||
|
|
||||||
def test_reader_flushes_truncated_table_on_exit(stub_repo):
|
|
||||||
"""进程被截断时，_read_loop 收尾必须把半张表按已知刻度兜底提交。"""
|
|
||||||
config = make_config(model="output/hrl/ui/test/truncated/q3.zip")
|
|
||||||
job = td.TrainingJob(config)
|
|
||||||
job.start()
|
|
||||||
# 直接注入一段"只有 rollout/ 段、没有 time/ 段"的表格，模拟中途被打断
|
|
||||||
job._on_line("| rollout/ | |")
|
|
||||||
job._on_line("| ep_rew_mean | -3.24 |")
|
|
||||||
job._on_line("| total_timesteps | 32 |")
|
|
||||||
job._on_line("| rollout/ | |")
|
|
||||||
job._on_line("| ep_rew_mean | -2.90 |")
|
|
||||||
assert wait_until(lambda: job.status in ("exited", "failed"), timeout=30)
|
|
||||||
snapshot = job.snapshot(log_tail=5)
|
|
||||||
steps = [point["timesteps"] for point in snapshot["metrics"]]
|
|
||||||
# 桩脚本自己产出的四张完整表 + 兜底提交的半张表（刻度沿用 128）
|
|
||||||
assert steps[-1] == 128, steps
|
|
||||||
assert any(point.get("ep_rew_mean") == -2.9 for point in snapshot["metrics"]), steps
|
|
||||||
|
|
||||||
|
|
||||||
def test_metric_cursor_pagination(stub_repo):
|
|
||||||
config = make_config(model="output/hrl/ui/test/cursor/q3.zip")
|
|
||||||
job = run_job(config)
|
|
||||||
full = job.snapshot(log_tail=5)["metrics"]
|
|
||||||
assert [point["timesteps"] for point in full] == [32, 64, 96, 128]
|
|
||||||
first = job.snapshot(log_tail=5, after_metric=0)
|
|
||||||
assert first["metric_offset"] == 0
|
|
||||||
assert first["metric_end"] == len(first["metrics"])
|
|
||||||
tail = job.snapshot(log_tail=5, after_metric=3)
|
|
||||||
assert tail["metric_offset"] == 3
|
|
||||||
assert len(tail["metrics"]) == 1
|
|
||||||
assert tail["metric_end"] == 4
|
|
||||||
beyond = job.snapshot(log_tail=5, after_metric=9999)
|
|
||||||
assert beyond["metrics"] == []
|
|
||||||
assert beyond["metric_offset"] == 4
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- HTTP
|
|
||||||
@pytest.fixture()
|
|
||||||
def client(stub_repo):
|
|
||||||
ui = td.TrainingUI(ui_port=0)
|
|
||||||
threading.Thread(target=ui.serve_forever, daemon=True).start()
|
|
||||||
port = ui.httpd.server_address[1]
|
|
||||||
yield ui, port
|
|
||||||
ui.shutdown()
|
|
||||||
|
|
||||||
|
|
||||||
def call(port, method, path, obj=None):
|
|
||||||
conn = http.client.HTTPConnection("127.0.0.1", port, timeout=20)
|
|
||||||
body = json.dumps(obj).encode("utf-8") if obj is not None else None
|
|
||||||
conn.request(method, path, body=body,
|
|
||||||
headers={"Content-Type": "application/json"})
|
|
||||||
response = conn.getresponse()
|
|
||||||
data = response.read()
|
|
||||||
conn.close()
|
|
||||||
try:
|
|
||||||
parsed = json.loads(data) if data else None
|
|
||||||
except ValueError:
|
|
||||||
parsed = data.decode("utf-8", "replace")
|
|
||||||
return response.status, parsed
|
|
||||||
|
|
||||||
|
|
||||||
def test_http_static_whitelist(client):
|
|
||||||
_ui, port = client
|
|
||||||
for path, needle in (("/", "<!doctype html>"), ("/app.js", "absorbMetrics"),
|
|
||||||
("/charts.js", "sparkline"), ("/style.css", ".panel")):
|
|
||||||
status, body = call(port, "GET", path)
|
|
||||||
assert status == 200, path
|
|
||||||
assert needle in body, path
|
|
||||||
status, body = call(port, "GET", "/../pyproject.toml")
|
|
||||||
assert status == 404 and "error" in body
|
|
||||||
|
|
||||||
|
|
||||||
def test_http_schema_and_preview(client):
|
|
||||||
_ui, port = client
|
|
||||||
status, schema = call(port, "GET", "/api/train/schema")
|
|
||||||
assert status == 200
|
|
||||||
assert len(schema["fields"]) == len(CONFIG_FIELDS)
|
|
||||||
assert all(key in schema for key in ("groups", "presets", "metrics"))
|
|
||||||
|
|
||||||
status, payload = call(port, "GET", "/api/train/state?preview=1&scenario=kl-spike")
|
|
||||||
assert status == 200
|
|
||||||
job = payload["job"]
|
|
||||||
assert job["preview"] is True
|
|
||||||
assert "KL" in job["convergence"]["status"]
|
|
||||||
status, body = call(port, "GET", "/api/train/state?preview=1&scenario=bogus")
|
|
||||||
assert status == 400 and "bogus" in body["error"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_http_state_query_validation(client):
|
|
||||||
_ui, port = client
|
|
||||||
status, body = call(port, "GET", "/api/train/state?log_tail=abc")
|
|
||||||
assert status == 400 and "log_tail" in body["error"]
|
|
||||||
status, body = call(port, "GET", "/api/train/state?log_tail=0")
|
|
||||||
assert status == 400
|
|
||||||
status, body = call(port, "GET", "/api/train/state?metric_limit=999999")
|
|
||||||
assert status == 400
|
|
||||||
status, body = call(port, "POST", "/api/train/start", [1, 2, 3])
|
|
||||||
assert status == 400
|
|
||||||
status, body = call(port, "POST", "/api/train/nope", {})
|
|
||||||
assert status == 404
|
|
||||||
|
|
||||||
|
|
||||||
def test_http_start_poll_stop(client):
|
|
||||||
_ui, port = client
|
|
||||||
status, body = call(port, "POST", "/api/train/start",
|
|
||||||
{"model": "output/hrl/ui/test/http/q3.zip",
|
|
||||||
"timesteps": 128, "rollout_steps": 32, "batch_size": 32})
|
|
||||||
assert status == 200
|
|
||||||
first = body["job"]
|
|
||||||
assert first["status"] in ("running", "created")
|
|
||||||
assert first["pid"]
|
|
||||||
assert "--progress-json" in first["command"]
|
|
||||||
|
|
||||||
def finished():
|
|
||||||
state = call(port, "GET", "/api/train/state?log_tail=5")[1]
|
|
||||||
return state["job"]["status"] == "exited"
|
|
||||||
|
|
||||||
assert wait_until(finished, timeout=30)
|
|
||||||
status, state = call(port, "GET", "/api/train/state?log_tail=50")
|
|
||||||
job = state["job"]
|
|
||||||
assert job["process"]["percent"] == 100.0
|
|
||||||
assert job["process"]["label"] == "正常结束"
|
|
||||||
assert len(job["monitor"]) == 3
|
|
||||||
|
|
||||||
status, body = call(port, "POST", "/api/train/start",
|
|
||||||
{"model": "output/hrl/ui/test/http2/q3.zip"})
|
|
||||||
assert status == 200
|
|
||||||
started = body["job"]
|
|
||||||
assert started["job_id"] != first["job_id"]
|
|
||||||
assert started["status"] in ("running", "created")
|
|
||||||
|
|
||||||
status, body = call(port, "POST", "/api/train/stop", {})
|
|
||||||
assert status == 200 and body["stopped"] is True
|
|
||||||
# 等训练进程真正退出后再点一次：应返回 False 而不是报错
|
|
||||||
assert wait_until(
|
|
||||||
lambda: call(port, "GET", "/api/train/state?log_tail=5")[1]["job"]["status"]
|
|
||||||
in ("stopped", "failed"), timeout=15)
|
|
||||||
status, body = call(port, "POST", "/api/train/stop", {})
|
|
||||||
assert status == 200 and body["stopped"] is False
|
|
||||||
|
|
||||||
|
|
||||||
def test_http_pause_resume_and_state(client, monkeypatch):
|
|
||||||
monkeypatch.setenv("STUB_TRAIN_SLEEP", "0.6")
|
|
||||||
_ui, port = client
|
|
||||||
status, body = call(port, "POST", "/api/train/start",
|
|
||||||
{"model": "output/hrl/ui/test/http-pause/q3.zip",
|
|
||||||
"timesteps": 4096, "rollout_steps": 32, "batch_size": 32})
|
|
||||||
assert status == 200
|
|
||||||
|
|
||||||
def status_now():
|
|
||||||
return call(port, "GET", "/api/train/state?log_tail=20")[1]["job"]["status"]
|
|
||||||
|
|
||||||
def is_running():
|
|
||||||
return status_now() == "running"
|
|
||||||
|
|
||||||
assert wait_until(is_running, timeout=10)
|
|
||||||
status, body = call(port, "POST", "/api/train/pause", {})
|
|
||||||
assert status == 200 and body["paused"] is True
|
|
||||||
assert wait_until(lambda: status_now() == "paused", timeout=10), status_now()
|
|
||||||
# 暂停中仍视为"有任务在跑"：不允许再开一局
|
|
||||||
status, body = call(port, "POST", "/api/train/start",
|
|
||||||
{"model": "output/hrl/ui/test/http-pause2/q3.zip"})
|
|
||||||
assert status == 409 and "已有训练任务" in body["error"]
|
|
||||||
status, body = call(port, "POST", "/api/train/resume", {})
|
|
||||||
assert status == 200 and body["resumed"] is True
|
|
||||||
assert wait_until(is_running, timeout=10), status_now()
|
|
||||||
status, body = call(port, "POST", "/api/train/stop", {})
|
|
||||||
assert status == 200 and body["stopped"] is True
|
|
||||||
assert wait_until(lambda: status_now() in ("stopped", "failed"), timeout=15)
|
|
||||||
|
|
||||||
|
|
||||||
def test_http_rejects_second_job_while_running(client):
|
|
||||||
_ui, port = client
|
|
||||||
status, body = call(port, "POST", "/api/train/start",
|
|
||||||
{"model": "output/hrl/ui/test/busy/q3.zip",
|
|
||||||
"timesteps": 9000, "rollout_steps": 3000,
|
|
||||||
"batch_size": 1000})
|
|
||||||
assert status == 200
|
|
||||||
status, body = call(port, "POST", "/api/train/start",
|
|
||||||
{"model": "output/hrl/ui/test/busy2/q3.zip"})
|
|
||||||
assert status == 409 and "已有训练任务" in body["error"]
|
|
||||||
call(port, "POST", "/api/train/stop", {})
|
|
||||||
|
|
||||||
|
|
||||||
def test_http_invalid_config_returns_409(client):
|
|
||||||
_ui, port = client
|
|
||||||
status, body = call(port, "POST", "/api/train/start", {"stage": 99})
|
|
||||||
assert status == 409 and "stage" in body["error"]
|
|
||||||
status, body = call(port, "POST", "/api/train/start",
|
|
||||||
{"model": "output/x.zip", "resume": True, "force": True})
|
|
||||||
assert status == 409 and "互斥" in body["error"]
|
|
||||||
|
|
||||||
|
|
||||||
def test_http_binds_loopback_only(client):
|
|
||||||
ui, _port = client
|
|
||||||
host = ui.httpd.server_address[0]
|
|
||||||
assert host == "127.0.0.1"
|
|
||||||
|
|
||||||
|
|
||||||
def test_port_conflict_reports_actionable_error(client):
|
|
||||||
_ui, port = client
|
|
||||||
with pytest.raises(OSError) as excinfo:
|
|
||||||
td.TrainingUI(ui_port=port)
|
|
||||||
assert "ui-port" in str(excinfo.value)
|
|
||||||
|
|
||||||
|
|
||||||
# ---------------------------------------------------------------- CLI
|
|
||||||
def test_cli_print_schema_and_preview(capsys):
|
|
||||||
assert td.run_training_ui(["--print-schema"]) == 0
|
|
||||||
schema = json.loads(capsys.readouterr().out)
|
|
||||||
assert schema["fields"]
|
|
||||||
assert td.run_training_ui(["--print-preview"]) == 0
|
|
||||||
payload = json.loads(capsys.readouterr().out)
|
|
||||||
assert payload["job"]["preview"] is True
|
|
||||||
|
|
||||||
|
|
||||||
def test_training_ui_module_entrypoint_exists():
|
|
||||||
text = (REPO / "scripts" / "train_hrl_ui.py").read_text(encoding="utf-8")
|
|
||||||
assert "run_training_ui" in text
|
|
||||||
assert sys.executable # 仅让 Windows 上的路径断言有明确前置条件
|
|
||||||
@@ -1,44 +0,0 @@
|
|||||||
"""训练前端 JS 逻辑回归：用 DOM 桩跑发布的 charts.js/app.js 原文。
|
|
||||||
|
|
||||||
动机：该前端的多数 bug 只在浏览器逻辑里暴露（游标重复追加、classList 取错、
|
|
||||||
字段缺失导致渲染中断），Python 侧测试无法覆盖。这里用服务端真实生成的
|
|
||||||
schema 与示例快照驱动一次完整渲染，有 node 则执行，无 node 自动跳过。
|
|
||||||
"""
|
|
||||||
|
|
||||||
from __future__ import annotations
|
|
||||||
|
|
||||||
import json
|
|
||||||
import os
|
|
||||||
import shutil
|
|
||||||
import subprocess
|
|
||||||
import sys
|
|
||||||
from pathlib import Path
|
|
||||||
|
|
||||||
import pytest
|
|
||||||
|
|
||||||
REPO = Path(__file__).resolve().parents[1]
|
|
||||||
HARNESS = REPO / "tests" / "js" / "training_ui_harness.cjs"
|
|
||||||
|
|
||||||
|
|
||||||
@pytest.mark.skipif(shutil.which("node") is None, reason="需要 node 环境运行 JS 桩测试")
|
|
||||||
def test_training_ui_frontend_logic(tmp_path):
|
|
||||||
sys.path.insert(0, str(REPO / "src"))
|
|
||||||
from mcm26b.training_ui.demo import DEMO_SCENARIOS, demo_snapshot
|
|
||||||
from mcm26b.training_ui.schema import config_schema
|
|
||||||
|
|
||||||
scenarios = {key: demo_snapshot(scenario=key) for key in DEMO_SCENARIOS}
|
|
||||||
scenarios_path = tmp_path / "scenarios.json"
|
|
||||||
schema_path = tmp_path / "schema.json"
|
|
||||||
scenarios_path.write_text(json.dumps(scenarios, ensure_ascii=False),
|
|
||||||
encoding="utf-8")
|
|
||||||
schema_path.write_text(json.dumps(config_schema(), ensure_ascii=False),
|
|
||||||
encoding="utf-8")
|
|
||||||
|
|
||||||
result = subprocess.run(
|
|
||||||
["node", str(HARNESS)], capture_output=True, text=True, timeout=120,
|
|
||||||
check=False,
|
|
||||||
env={**os.environ, "REPO": str(REPO),
|
|
||||||
"DEMO_STATE": str(scenarios_path), "SCHEMA": str(schema_path)})
|
|
||||||
assert result.returncode == 0, \
|
|
||||||
f"训练前端逻辑测试失败:\n{result.stdout}\n{result.stderr}"
|
|
||||||
assert "ALL PASS" in result.stdout
|
|
||||||
@@ -203,15 +203,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/cc/61/d01fc49b8dea277640b55a9e15960dbca9fdc8c9fde18e572d39c59f4019/charset_normalizer-3.5.1-py3-none-any.whl", hash = "sha256:6df0ec430f9a831772c23ca5a224cba36517a58a84bb32c32bb59a9fa67c47f6" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/cc/61/d01fc49b8dea277640b55a9e15960dbca9fdc8c9fde18e572d39c59f4019/charset_normalizer-3.5.1-py3-none-any.whl", hash = "sha256:6df0ec430f9a831772c23ca5a224cba36517a58a84bb32c32bb59a9fa67c47f6" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "cloudpickle"
|
|
||||||
version = "3.1.2"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/27/fb/576f067976d320f5f0114a8d9fa1215425441bb35627b1993e5afd8111e5/cloudpickle-3.1.2.tar.gz", hash = "sha256:7fda9eb655c9c230dab534f1983763de5835249750e85fbcef43aaa30a9a2414" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/88/39/799be3f2f0f38cc727ee3b4f1445fe6d5e4133064ec2e4115069418a5bb6/cloudpickle-3.1.2-py3-none-any.whl", hash = "sha256:9acb47f6afd73f60dc1df93bb801b472f05ff42fa6c84167d25cb206be1fbf4a" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "colorama"
|
name = "colorama"
|
||||||
version = "0.4.6"
|
version = "0.4.6"
|
||||||
@@ -233,81 +224,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/d4/19/0b6647bf5e331521e55d2b63bfbdc210bd9cd605189273f03614a05f702d/colorlog-6.12.0-py3-none-any.whl", hash = "sha256:30d392604e9110045a2c2aeefc27d7a017abbab63f3a8aee594eac0801df784e" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/d4/19/0b6647bf5e331521e55d2b63bfbdc210bd9cd605189273f03614a05f702d/colorlog-6.12.0-py3-none-any.whl", hash = "sha256:30d392604e9110045a2c2aeefc27d7a017abbab63f3a8aee594eac0801df784e" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "cuda-bindings"
|
|
||||||
version = "12.9.7"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "cuda-pathfinder" },
|
|
||||||
]
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/cd/2e/b1b14be5884519917f9df4d9106d3e16575c19fa847d13a3f6e9d272b5cd/cuda_bindings-12.9.7-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a318075ef3277ca2fdd7df5d4bf671388696b4f2b65e2c4483f1853517692e3b" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/82/1f/0809c53f694693d703c9efee0379875089db17ab50196845e08f6c686fd4/cuda_bindings-12.9.7-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f95debd2c54c5f087462668511bcb2b1295baa389cd6de9e768212f3cab2bbe2" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/40/f3/f9d1095f90d2a4df24cfcafe7487fd9444c6dacb94e3722be6fedd8ac26c/cuda_bindings-12.9.7-cp311-cp311-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:16043ef5b15ab88fe9954c5c2061b1d8007591b27f2c916331056de0ebc6187e" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/3a/8a/1251e1794b69865aacd5629936006b18ea0816a495de4ecea9a825556eb3/cuda_bindings-12.9.7-cp311-cp311-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c6496a88d84b1209d6651b0370c19c26319e157c22f6d018bf9a358cd8049041" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/32/45/557d4ed1fa54f0c7db8aee083229f624990d69f7d00f55477eed5c7e169a/cuda_bindings-12.9.7-cp312-cp312-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0666d3c082ef8f4b2d670950589373550e9f3bf564d635dd883f24a0b40402ff" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/91/97/e3c6e58ece26a053419ba0a18444b5443cfc64451bbf37f84e8143b8bdca/cuda_bindings-12.9.7-cp312-cp312-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4c7ef48c5e13ae90f3b2ecfb72f8e99ac43c8f4c43e67e1325b8aae331453687" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/eb/7b/f1575e41e1a17dc2f2a408b2e8e864c9324e41e3e23f6401e5efc54c152a/cuda_bindings-12.9.7-cp313-cp313-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:266379e4942051f544a8e7ea1a30ead8d7e8199b6b30fcdc8917cae2bf614e61" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/9d/dc/62d62eb4f91eb721bcf46da51b13e9872ccd8fa7e60eb8ba7b7baeac72c6/cuda_bindings-12.9.7-cp313-cp313-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:59cf4a37b0d662ba15037c9ceebe1a306ebf2c01a8235a09be13cd07094fdb74" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/f9/77/94d9b85f26add6fe9c9cb7c4ec3b96bc598f7ea5cfbd7490cc0a36adf5be/cuda_bindings-12.9.7-cp313-cp313t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2dbcd4801954eb3508f4dc2fa0d0c8eb93eb3f45326fd61be2731418c371e7a0" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/04/dd/3ec34b569e1b990b11276feba306bf8f446656cc38e8ed0f49b5facfeffa/cuda_bindings-12.9.7-cp313-cp313t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3747ea132642416786a8e31bf229032df3a7856911ae5426a7be53d032df183d" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/68/e4/075052d42872cf8162da53f14447a4b8abc004c3750e4b724ee502428da0/cuda_bindings-12.9.7-cp314-cp314-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:775960ac9e530717f3b48e165cc6f68684fa9a4141764fd923e4c1a9820acc73" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/ec/cd/3289c810a4d45e5364a3387a74b4c9b6f6f57ee96ae0e5b537cc61dec242/cuda_bindings-12.9.7-cp314-cp314-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3c47ec1a7a441d91aab32339951df7a1be53451121a12c094bba51467717a35a" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/11/43/472a6281c3d94e71687e27c657a8f60718d3579b4d94c41deea503165f8a/cuda_bindings-12.9.7-cp314-cp314t-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:00a833d399b31071fab4cf3de2929840ae462dc4848116eeff033d09219e7116" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/2b/13/10c1d0b32a9da65142d213e0733d748457fb3fd066aee4317335266f15c6/cuda_bindings-12.9.7-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:11aeafa2b33995f890086b3fb0f062075176d956e9b6a6fe1a699dddc413f6ad" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "cuda-pathfinder"
|
|
||||||
version = "1.8.1"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/9d/e6/22df83f82f9bc26cb1c42265cf14d34d4908dba2a0f261bd7b28244acb00/cuda_pathfinder-1.8.1-py3-none-any.whl", hash = "sha256:ae0137ff9e56ea97499bcbf54f5f2778ec25f3266715ac86da192a795af982a8" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "cuda-toolkit"
|
|
||||||
version = "12.8.1"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/d4/c8/7dce3a0b15b42a3b58e7d96eb22a687d3bf2c44e01d149a6874629cd9938/cuda_toolkit-12.8.1-py2.py3-none-any.whl", hash = "sha256:adc7906af4ecbf9a352f9dca5734eceb21daec281ccfcf5675e1d2f724fc2cba" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[package.optional-dependencies]
|
|
||||||
cublas = [
|
|
||||||
{ name = "nvidia-cublas-cu12" },
|
|
||||||
]
|
|
||||||
cudart = [
|
|
||||||
{ name = "nvidia-cuda-runtime-cu12" },
|
|
||||||
]
|
|
||||||
cufft = [
|
|
||||||
{ name = "nvidia-cufft-cu12" },
|
|
||||||
]
|
|
||||||
cufile = [
|
|
||||||
{ name = "nvidia-cufile-cu12" },
|
|
||||||
]
|
|
||||||
cupti = [
|
|
||||||
{ name = "nvidia-cuda-cupti-cu12" },
|
|
||||||
]
|
|
||||||
curand = [
|
|
||||||
{ name = "nvidia-curand-cu12" },
|
|
||||||
]
|
|
||||||
cusolver = [
|
|
||||||
{ name = "nvidia-cusolver-cu12" },
|
|
||||||
]
|
|
||||||
cusparse = [
|
|
||||||
{ name = "nvidia-cusparse-cu12" },
|
|
||||||
]
|
|
||||||
nvjitlink = [
|
|
||||||
{ name = "nvidia-nvjitlink-cu12" },
|
|
||||||
]
|
|
||||||
nvrtc = [
|
|
||||||
{ name = "nvidia-cuda-nvrtc-cu12" },
|
|
||||||
]
|
|
||||||
nvtx = [
|
|
||||||
{ name = "nvidia-nvtx-cu12" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "exceptiongroup"
|
name = "exceptiongroup"
|
||||||
version = "1.3.1"
|
version = "1.3.1"
|
||||||
@@ -320,33 +236,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl", hash = "sha256:a7a39a3bd276781e98394987d3a5701d0c4edffb633bb7a5144577f82c773598" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "farama-notifications"
|
|
||||||
version = "0.0.6"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/ec/91/14397890dde30adc4bee6462158933806207bc5dd10d7b4d09d5c33845cf/farama_notifications-0.0.6.tar.gz", hash = "sha256:b19acac4bb41d76e59e03394b5dd165f4761c86fa327f56307a35cbee3b60158" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/7c/f0/21f81892e4ed10f4ec3ef2e7cf8635fb76e7c0907c55d0da66be50094760/farama_notifications-0.0.6-py3-none-any.whl", hash = "sha256:f84839188efa1ce5bb361c2a84881b2dc2c0d0d7fb661ff00421820170930935" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "filelock"
|
|
||||||
version = "3.32.6"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/38/46/126b1831dca12060d4a8296bf9c4fe5c93c4f22197fa239cb0cc82042bba/filelock-3.32.6.tar.gz", hash = "sha256:a3f55a18af3652a94d8f47d6055df434f254ca1d02ef2524850c6d249ca2512c" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/cc/06/4f138f618dbea66803291274f228f01daf29f306fe8b96bc30dab765df75/filelock-3.32.6-py3-none-any.whl", hash = "sha256:3f16ecd0117feae0dfc147e8c62eb5daeccd8bd800378c3ddf416de9b4feb6b1" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "fsspec"
|
|
||||||
version = "2026.7.0"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/00/78/f34251dadb8f3921264a1d9b8946f5e542014ee2614b285261b4e40e6775/fsspec-2026.7.0.tar.gz", hash = "sha256:c803c40f4cf860b49dea58ee3e1c33cb9c790520e233537e1340049f89b82a88" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/fd/3c/6a2bf344106328fd04963664a60b9bb6496fc25df8e962fcdc1367285fb9/fsspec-2026.7.0-py3-none-any.whl", hash = "sha256:b57ddbafedfaef7018c1ecab32aa200a9d7ca26b77965f64e48b70061249d279" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "greenlet"
|
name = "greenlet"
|
||||||
version = "3.5.5"
|
version = "3.5.5"
|
||||||
@@ -417,23 +306,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/df/af/419a4e383bd600858a9b67e9b280a60fdc383ee3f2fe5b6c0c1ef04e74d1/greenlet-3.5.5-cp315-cp315t-win_arm64.whl", hash = "sha256:7f049911ee81a16a03c33d5450d8d5867d27f596ca5fb201b86f4524e874468b" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/df/af/419a4e383bd600858a9b67e9b280a60fdc383ee3f2fe5b6c0c1ef04e74d1/greenlet-3.5.5-cp315-cp315t-win_arm64.whl", hash = "sha256:7f049911ee81a16a03c33d5450d8d5867d27f596ca5fb201b86f4524e874468b" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "gymnasium"
|
|
||||||
version = "1.3.0"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "cloudpickle" },
|
|
||||||
{ name = "farama-notifications" },
|
|
||||||
{ name = "numpy", version = "2.2.6", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version < '3.11'" },
|
|
||||||
{ name = "numpy", version = "2.4.6", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version == '3.11.*'" },
|
|
||||||
{ name = "numpy", version = "2.5.3", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version >= '3.12'" },
|
|
||||||
{ name = "typing-extensions" },
|
|
||||||
]
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/4d/ff/14b6880d703dfaca204490979d3254ccd280c99550798993319902873658/gymnasium-1.3.0.tar.gz", hash = "sha256:6939e86e835d6b71b6ba6bfd360487420876deafc79bfb7bacba83a7c446bcf3" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/e9/73/fda6a25f3beeb5e49d74330b44092b9e5a547395ccd478d1103ddcbff1fc/gymnasium-1.3.0-py3-none-any.whl", hash = "sha256:6b8c159a8540dcbcb221722d7efda24d78ebbcbc3bd2ea1c2611aa2a34471fc2" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "idna"
|
name = "idna"
|
||||||
version = "3.19"
|
version = "3.19"
|
||||||
@@ -452,18 +324,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/cb/b1/3846dd7f199d53cb17f49cba7e651e9ce294d8497c8c150530ed11865bb8/iniconfig-2.3.0-py3-none-any.whl", hash = "sha256:f631c04d2c48c52b84d0d0549c99ff3859c98df65b3101406327ecc7d53fbf12" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "jinja2"
|
|
||||||
version = "3.1.6"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "markupsafe" },
|
|
||||||
]
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/df/bf/f7da0350254c0ed7c72f3e33cef02e048281fec7ecec5f032d4aac52226b/jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/62/a1/3d680cbfd5f4b8f15abc1d571870c5fc3e594bb582bc3b64ea099db13e56/jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "mako"
|
name = "mako"
|
||||||
version = "1.4.1"
|
version = "1.4.1"
|
||||||
@@ -575,13 +435,6 @@ dev = [
|
|||||||
{ name = "optuna" },
|
{ name = "optuna" },
|
||||||
{ name = "pytest" },
|
{ name = "pytest" },
|
||||||
]
|
]
|
||||||
rl = [
|
|
||||||
{ name = "gymnasium" },
|
|
||||||
{ name = "sb3-contrib" },
|
|
||||||
{ name = "stable-baselines3" },
|
|
||||||
{ name = "torch", version = "2.11.0", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "sys_platform != 'linux' and sys_platform != 'win32'" },
|
|
||||||
{ name = "torch", version = "2.11.0+cu128", source = { registry = "https://download.pytorch.org/whl/cu128" }, marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[package.metadata]
|
[package.metadata]
|
||||||
requires-dist = [{ name = "requests", specifier = ">=2.31" }]
|
requires-dist = [{ name = "requests", specifier = ">=2.31" }]
|
||||||
@@ -592,56 +445,12 @@ dev = [
|
|||||||
{ name = "optuna", specifier = ">=5.0.0" },
|
{ name = "optuna", specifier = ">=5.0.0" },
|
||||||
{ name = "pytest", specifier = ">=8" },
|
{ name = "pytest", specifier = ">=8" },
|
||||||
]
|
]
|
||||||
rl = [
|
|
||||||
{ name = "gymnasium", specifier = ">=1.1" },
|
|
||||||
{ name = "sb3-contrib", specifier = "==2.9.0" },
|
|
||||||
{ name = "stable-baselines3", specifier = ">=2.6" },
|
|
||||||
{ name = "torch", marker = "sys_platform != 'linux' and sys_platform != 'win32'", specifier = "==2.11.0" },
|
|
||||||
{ name = "torch", marker = "sys_platform == 'linux' or sys_platform == 'win32'", specifier = "==2.11.0", index = "https://download.pytorch.org/whl/cu128" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "mcm26b-referee"
|
name = "mcm26b-referee"
|
||||||
version = "1.0.0"
|
version = "1.0.0"
|
||||||
source = { editable = "referee" }
|
source = { editable = "referee" }
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "mpmath"
|
|
||||||
version = "1.3.0"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/e0/47/dd32fa426cc72114383ac549964eecb20ecfd886d1e5ccf5340b55b02f57/mpmath-1.3.0.tar.gz", hash = "sha256:7a28eb2a9774d00c7bc92411c19a89209d5da7c4c9a9e227be8330a23a25b91f" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl", hash = "sha256:a0b2b9fe80bbcd81a6647ff13108738cfb482d481d826cc0e02f5b35e5c88d2c" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "networkx"
|
|
||||||
version = "3.4.2"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
resolution-markers = [
|
|
||||||
"(python_full_version < '3.11' and sys_platform == 'linux') or (python_full_version < '3.11' and sys_platform == 'win32')",
|
|
||||||
"python_full_version < '3.11' and sys_platform != 'linux' and sys_platform != 'win32'",
|
|
||||||
]
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/fd/1d/06475e1cd5264c0b870ea2cc6fdb3e37177c1e565c43f56ff17a10e3937f/networkx-3.4.2.tar.gz", hash = "sha256:307c3669428c5362aab27c8a1260aa8f47c4e91d3891f48be0141738d8d053e1" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/b9/54/dd730b32ea14ea797530a4479b2ed46a6fb250f682a9cfb997e968bf0261/networkx-3.4.2-py3-none-any.whl", hash = "sha256:df5d4365b724cf81b8c6a7312509d0c22386097011ad1abe274afd5e9d3bbc5f" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "networkx"
|
|
||||||
version = "3.6.1"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
resolution-markers = [
|
|
||||||
"(python_full_version >= '3.12' and sys_platform == 'linux') or (python_full_version >= '3.12' and sys_platform == 'win32')",
|
|
||||||
"python_full_version >= '3.12' and sys_platform != 'linux' and sys_platform != 'win32'",
|
|
||||||
"(python_full_version == '3.11.*' and sys_platform == 'linux') or (python_full_version == '3.11.*' and sys_platform == 'win32')",
|
|
||||||
"python_full_version == '3.11.*' and sys_platform != 'linux' and sys_platform != 'win32'",
|
|
||||||
]
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/6a/51/63fe664f3908c97be9d2e4f1158eb633317598cfa6e1fc14af5383f17512/networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "numpy"
|
name = "numpy"
|
||||||
version = "2.2.6"
|
version = "2.2.6"
|
||||||
@@ -868,155 +677,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/fb/0b/b12a2df5d1b774bd9007a6fdff9381145b6223d37f11afc9c37ab0efd9a1/numpy-2.5.3-cp315-cp315t-win_arm64.whl", hash = "sha256:befa1ae5bd6030b3f512b43ff3fa5290bbed6b84411a44244b14adf835f5b89d" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/fb/0b/b12a2df5d1b774bd9007a6fdff9381145b6223d37f11afc9c37ab0efd9a1/numpy-2.5.3-cp315-cp315t-win_arm64.whl", hash = "sha256:befa1ae5bd6030b3f512b43ff3fa5290bbed6b84411a44244b14adf835f5b89d" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cublas-cu12"
|
|
||||||
version = "12.8.4.1"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/29/99/db44d685f0e257ff0e213ade1964fc459b4a690a73293220e98feb3307cf/nvidia_cublas_cu12-12.8.4.1-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:b86f6dd8935884615a0683b663891d43781b819ac4f2ba2b0c9604676af346d0" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/dc/61/e24b560ab2e2eaeb3c839129175fb330dfcfc29e5203196e5541a4c44682/nvidia_cublas_cu12-12.8.4.1-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:8ac4e771d5a348c551b2a426eda6193c19aa630236b418086020df5ba9667142" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cuda-cupti-cu12"
|
|
||||||
version = "12.8.90"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/d5/1f/b3bd73445e5cb342727fd24fe1f7b748f690b460acadc27ea22f904502c8/nvidia_cuda_cupti_cu12-12.8.90-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:4412396548808ddfed3f17a467b104ba7751e6b58678a4b840675c56d21cf7ed" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/f8/02/2adcaa145158bf1a8295d83591d22e4103dbfd821bcaf6f3f53151ca4ffa/nvidia_cuda_cupti_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:ea0cb07ebda26bb9b29ba82cda34849e73c166c18162d3913575b0c9db9a6182" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cuda-nvrtc-cu12"
|
|
||||||
version = "12.8.93"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/05/6b/32f747947df2da6994e999492ab306a903659555dddc0fbdeb9d71f75e52/nvidia_cuda_nvrtc_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:a7756528852ef889772a84c6cd89d41dfa74667e24cca16bb31f8f061e3e9994" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/eb/d1/e50d0acaab360482034b84b6e27ee83c6738f7d32182b987f9c7a4e32962/nvidia_cuda_nvrtc_cu12-12.8.93-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:fc1fec1e1637854b4c0a65fb9a8346b51dd9ee69e61ebaccc82058441f15bce8" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cuda-runtime-cu12"
|
|
||||||
version = "12.8.90"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/7c/75/f865a3b236e4647605ea34cc450900854ba123834a5f1598e160b9530c3a/nvidia_cuda_runtime_cu12-12.8.90-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:52bf7bbee900262ffefe5e9d5a2a69a30d97e2bc5bb6cc866688caa976966e3d" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/0d/9b/a997b638fcd068ad6e4d53b8551a7d30fe8b404d6f1804abf1df69838932/nvidia_cuda_runtime_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:adade8dcbd0edf427b7204d480d6066d33902cab2a4707dcfc48a2d0fd44ab90" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cudnn-cu12"
|
|
||||||
version = "9.19.0.56"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "nvidia-cublas-cu12" },
|
|
||||||
]
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/09/b8/277c51962ee46fa3e5b203ac5f76107c650f781d6891e681e28e6f3e9fe6/nvidia_cudnn_cu12-9.19.0.56-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:08caaf27fe556aca82a3ee3b5aa49a77e7de0cfcb7ff4e5c29da426387a8267e" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/c5/41/65225d42fba06fb3dd3972485ea258e7dd07a40d6e01c95da6766ad87354/nvidia_cudnn_cu12-9.19.0.56-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:ac6ad90a075bb33a94f2b4cf4622eac13dd4dc65cf6dd9c7572a318516a36625" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cufft-cu12"
|
|
||||||
version = "11.3.3.83"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "nvidia-nvjitlink-cu12" },
|
|
||||||
]
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/60/bc/7771846d3a0272026c416fbb7e5f4c1f146d6d80704534d0b187dd6f4800/nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:848ef7224d6305cdb2a4df928759dca7b1201874787083b6e7550dd6765ce69a" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/1f/13/ee4e00f30e676b66ae65b4f08cb5bcbb8392c03f54f2d5413ea99a5d1c80/nvidia_cufft_cu12-11.3.3.83-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:4d2dd21ec0b88cf61b62e6b43564355e5222e4a3fb394cac0db101f2dd0d4f74" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cufile-cu12"
|
|
||||||
version = "1.13.1.3"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/bb/fe/1bcba1dfbfb8d01be8d93f07bfc502c93fa23afa6fd5ab3fc7c1df71038a/nvidia_cufile_cu12-1.13.1.3-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1d069003be650e131b21c932ec3d8969c1715379251f8d23a1860554b1cb24fc" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/1e/f5/5607710447a6fe9fd9b3283956fceeee8a06cda1d2f56ce31371f595db2a/nvidia_cufile_cu12-1.13.1.3-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:4beb6d4cce47c1a0f1013d72e02b0994730359e17801d395bdcbf20cfb3bb00a" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-curand-cu12"
|
|
||||||
version = "10.3.9.90"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/45/5e/92aa15eca622a388b80fbf8375d4760738df6285b1e92c43d37390a33a9a/nvidia_curand_cu12-10.3.9.90-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:dfab99248034673b779bc6decafdc3404a8a6f502462201f2f31f11354204acd" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/fb/aa/6584b56dc84ebe9cf93226a5cde4d99080c8e90ab40f0c27bda7a0f29aa1/nvidia_curand_cu12-10.3.9.90-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:b32331d4f4df5d6eefa0554c565b626c7216f87a06a4f56fab27c3b68a830ec9" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cusolver-cu12"
|
|
||||||
version = "11.7.3.90"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "nvidia-cublas-cu12" },
|
|
||||||
{ name = "nvidia-cusparse-cu12" },
|
|
||||||
{ name = "nvidia-nvjitlink-cu12" },
|
|
||||||
]
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/c8/32/f7cd6ce8a7690544d084ea21c26e910a97e077c9b7f07bf5de623ee19981/nvidia_cusolver_cu12-11.7.3.90-py3-none-manylinux_2_27_aarch64.whl", hash = "sha256:db9ed69dbef9715071232caa9b69c52ac7de3a95773c2db65bdba85916e4e5c0" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/85/48/9a13d2975803e8cf2777d5ed57b87a0b6ca2cc795f9a4f59796a910bfb80/nvidia_cusolver_cu12-11.7.3.90-py3-none-manylinux_2_27_x86_64.whl", hash = "sha256:4376c11ad263152bd50ea295c05370360776f8c3427b30991df774f9fb26c450" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cusparse-cu12"
|
|
||||||
version = "12.5.8.93"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "nvidia-nvjitlink-cu12" },
|
|
||||||
]
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/bc/f7/cd777c4109681367721b00a106f491e0d0d15cfa1fd59672ce580ce42a97/nvidia_cusparse_cu12-12.5.8.93-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:9b6c161cb130be1a07a27ea6923df8141f3c295852f4b260c65f18f3e0a091dc" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/c2/f5/e1854cb2f2bcd4280c44736c93550cc300ff4b8c95ebe370d0aa7d2b473d/nvidia_cusparse_cu12-12.5.8.93-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1ec05d76bbbd8b61b06a80e1eaf8cf4959c3d4ce8e711b65ebd0443bb0ebb13b" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-cusparselt-cu12"
|
|
||||||
version = "0.7.1"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/73/b9/598f6ff36faaece4b3c50d26f50e38661499ff34346f00e057760b35cc9d/nvidia_cusparselt_cu12-0.7.1-py3-none-manylinux2014_aarch64.whl", hash = "sha256:8878dce784d0fac90131b6817b607e803c36e629ba34dc5b433471382196b6a5" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/56/79/12978b96bd44274fe38b5dde5cfb660b1d114f70a65ef962bcbbed99b549/nvidia_cusparselt_cu12-0.7.1-py3-none-manylinux2014_x86_64.whl", hash = "sha256:f1bb701d6b930d5a7cea44c19ceb973311500847f81b634d802b7b539dc55623" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-nccl-cu12"
|
|
||||||
version = "2.28.9"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/08/c4/120d2dfd92dff2c776d68f361ff8705fdea2ca64e20b612fab0fd3f581ac/nvidia_nccl_cu12-2.28.9-py3-none-manylinux_2_18_aarch64.whl", hash = "sha256:50a36e01c4a090b9f9c47d92cec54964de6b9fcb3362d0e19b8ffc6323c21b60" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/4a/4e/44dbb46b3d1b0ec61afda8e84837870f2f9ace33c564317d59b70bc19d3e/nvidia_nccl_cu12-2.28.9-py3-none-manylinux_2_18_x86_64.whl", hash = "sha256:485776daa8447da5da39681af455aa3b2c2586ddcf4af8772495e7c532c7e5ab" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-nvjitlink-cu12"
|
|
||||||
version = "12.8.93"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/f6/74/86a07f1d0f42998ca31312f998bd3b9a7eff7f52378f4f270c8679c77fb9/nvidia_nvjitlink_cu12-12.8.93-py3-none-manylinux2010_x86_64.manylinux_2_12_x86_64.whl", hash = "sha256:81ff63371a7ebd6e6451970684f916be2eab07321b73c9d244dc2b4da7f73b88" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/2a/a2/8cee5da30d13430e87bf99bb33455d2724d0a4a9cb5d7926d80ccb96d008/nvidia_nvjitlink_cu12-12.8.93-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:adccd7161ace7261e01bb91e44e88da350895c270d23f744f0820c818b7229e7" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-nvshmem-cu12"
|
|
||||||
version = "3.4.5"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/1d/6a/03aa43cc9bd3ad91553a88b5f6fb25ed6a3752ae86ce2180221962bc2aa5/nvidia_nvshmem_cu12-3.4.5-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:0b48363fc6964dede448029434c6abed6c5e37f823cb43c3bcde7ecfc0457e15" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/b5/09/6ea3ea725f82e1e76684f0708bbedd871fc96da89945adeba65c3835a64c/nvidia_nvshmem_cu12-3.4.5-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:042f2500f24c021db8a06c5eec2539027d57460e1c1a762055a6554f72c369bd" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "nvidia-nvtx-cu12"
|
|
||||||
version = "12.8.90"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/10/c0/1b303feea90d296f6176f32a2a70b5ef230f9bdeb3a72bddb0dc922dc137/nvidia_nvtx_cu12-12.8.90-py3-none-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:d7ad891da111ebafbf7e015d34879f7112832fc239ff0d7d776b6cb685274615" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/a2/eb/86626c1bbc2edb86323022371c39aa48df6fd8b0a1647bc274577f72e90b/nvidia_nvtx_cu12-12.8.90-py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:5b17e2001cc0d751a5bc2c6ec6d26ad95913324a4adb86788c944f8ce9ba441f" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "optuna"
|
name = "optuna"
|
||||||
version = "5.0.0"
|
version = "5.0.0"
|
||||||
@@ -1161,27 +821,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl", hash = "sha256:2a0d60c172f83ac6ab31e4554906c0f3b3588d37b5cb939b1c061f4907e278e0" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "sb3-contrib"
|
|
||||||
version = "2.9.0"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "stable-baselines3" },
|
|
||||||
]
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/8f/8e/476222d88ba09dce0feebf6fb30b55e4dd0c80be5b19dc7b40d12d70752a/sb3_contrib-2.9.0.tar.gz", hash = "sha256:6e839c669552ecb3deb616a42ea98ed5e6c599eb5ac5f86232415b87a3873699" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/c3/06/add3f1d6250f58ca493830ddb2118975aa3c4433bc07662d5ac5089eb691/sb3_contrib-2.9.0-py3-none-any.whl", hash = "sha256:b492b2be792f8f8214ff6b984e94812dc877f809ae82302cbe00626442ae316a" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "setuptools"
|
|
||||||
version = "81.0.0"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/0d/1c/73e719955c59b8e424d015ab450f51c0af856ae46ea2da83eba51cc88de1/setuptools-81.0.0.tar.gz", hash = "sha256:487b53915f52501f0a79ccfd0c02c165ffe06631443a886740b91af4b7a5845a" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/e1/e3/c164c88b2e5ce7b24d667b9bd83589cf4f3520d97cad01534cd3c4f55fdb/setuptools-81.0.0-py3-none-any.whl", hash = "sha256:fdd925d5c5d9f62e4b74b30d6dd7828ce236fd6ed998a08d81de62ce5a6310d6" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "sqlalchemy"
|
name = "sqlalchemy"
|
||||||
version = "2.0.52"
|
version = "2.0.52"
|
||||||
@@ -1231,36 +870,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/b3/3f/3582293d1e185e71d19d7c731c3e2ee20ba21981c4a1115c0806c1f62120/sqlalchemy-2.0.52-py3-none-any.whl", hash = "sha256:3b81b8363a919ce53453591cdb93702e6bd54ade6c4fa2f468fc053baee5ed89" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/b3/3f/3582293d1e185e71d19d7c731c3e2ee20ba21981c4a1115c0806c1f62120/sqlalchemy-2.0.52-py3-none-any.whl", hash = "sha256:3b81b8363a919ce53453591cdb93702e6bd54ade6c4fa2f468fc053baee5ed89" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "stable-baselines3"
|
|
||||||
version = "2.9.0"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "cloudpickle" },
|
|
||||||
{ name = "gymnasium" },
|
|
||||||
{ name = "numpy", version = "2.2.6", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version < '3.11'" },
|
|
||||||
{ name = "numpy", version = "2.4.6", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version == '3.11.*'" },
|
|
||||||
{ name = "numpy", version = "2.5.3", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version >= '3.12'" },
|
|
||||||
{ name = "torch", version = "2.11.0", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "sys_platform != 'linux' and sys_platform != 'win32'" },
|
|
||||||
{ name = "torch", version = "2.11.0+cu128", source = { registry = "https://download.pytorch.org/whl/cu128" }, marker = "sys_platform == 'linux' or sys_platform == 'win32'" },
|
|
||||||
]
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/89/f8/9c1901a42e55b21b9e9559a720cb33dffdeb4c4215b1dfd10d63224b19c0/stable_baselines3-2.9.0.tar.gz", hash = "sha256:92b46c6099a0e8f99163ff09e26729e4d0a68b33dc8598626ca13ade3c0b3a61" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/a6/70/d18c8278223d4928378c4af822b66d3c87e19e73bcb011e05e2f0b43f4e5/stable_baselines3-2.9.0-py3-none-any.whl", hash = "sha256:95f39a473dce081d1abe31acaf7cee446dcf223dcd74093d6ea460bc37e2e748" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "sympy"
|
|
||||||
version = "1.14.0"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
dependencies = [
|
|
||||||
{ name = "mpmath" },
|
|
||||||
]
|
|
||||||
sdist = { url = "https://mirrors.aliyun.com/pypi/packages/83/d3/803453b36afefb7c2bb238361cd4ae6125a569b4db67cd9e79846ba2d68c/sympy-1.14.0.tar.gz", hash = "sha256:d3d3fe8df1e5a0b42f0e7bdf50541697dbe7d23746e894990c030e2b05e72517" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl", hash = "sha256:e091cc3e99d2141a0ba2847328f5479b05d94a6635cb96148ccb3f34671bd8f5" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "tomli"
|
name = "tomli"
|
||||||
version = "2.4.1"
|
version = "2.4.1"
|
||||||
@@ -1315,85 +924,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/7b/61/cceae43728b7de99d9b847560c262873a1f6c98202171fd5ed62640b494b/tomli-2.4.1-py3-none-any.whl", hash = "sha256:0d85819802132122da43cb86656f8d1f8c6587d54ae7dcaf30e90533028b49fe" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/7b/61/cceae43728b7de99d9b847560c262873a1f6c98202171fd5ed62640b494b/tomli-2.4.1-py3-none-any.whl", hash = "sha256:0d85819802132122da43cb86656f8d1f8c6587d54ae7dcaf30e90533028b49fe" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "torch"
|
|
||||||
version = "2.11.0"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
resolution-markers = [
|
|
||||||
"python_full_version >= '3.12' and sys_platform != 'linux' and sys_platform != 'win32'",
|
|
||||||
"python_full_version == '3.11.*' and sys_platform != 'linux' and sys_platform != 'win32'",
|
|
||||||
"python_full_version < '3.11' and sys_platform != 'linux' and sys_platform != 'win32'",
|
|
||||||
]
|
|
||||||
dependencies = [
|
|
||||||
{ name = "filelock" },
|
|
||||||
{ name = "fsspec" },
|
|
||||||
{ name = "jinja2" },
|
|
||||||
{ name = "networkx", version = "3.4.2", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version < '3.11'" },
|
|
||||||
{ name = "networkx", version = "3.6.1", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version >= '3.11'" },
|
|
||||||
{ name = "setuptools" },
|
|
||||||
{ name = "sympy" },
|
|
||||||
{ name = "typing-extensions" },
|
|
||||||
]
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/ac/f2/c1690994afe461aae2d0cac62251e6802a703dec0a6c549c02ecd0de92a9/torch-2.11.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:2c0d7fcfbc0c4e8bb5ebc3907cbc0c6a0da1b8f82b1fc6e14e914fa0b9baf74e" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/ae/0d/98b410492609e34a155fa8b121b55c7dca229f39636851c3a9ec20edea21/torch-2.11.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:7b6a60d48062809f58595509c524b88e6ddec3ebe25833d6462eeab81e5f2ce4" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/6f/8b/69e3008d78e5cee2b30183340cc425081b78afc5eff3d080daab0adda9aa/torch-2.11.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:4b5866312ee6e52ea625cd211dcb97d6a2cdc1131a5f15cc0d87eec948f6dd34" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/87/89/5ea6722763acee56b045435fb84258db7375c48165ec8be7880ab2b281c5/torch-2.11.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:1e6debd97ccd3205bbb37eb806a9d8219e1139d15419982c09e23ef7d4369d18" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/db/38/8ac78069621b8c2b4979c2f96dc8409ef5e9c4189f6aac629189a78677ca/torch-2.11.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:8b394322f49af4362d4f80e424bcaca7efcd049619af03a4cf4501520bdf0fb4" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/26/0d/8603382f61abd0db35841148ddc1ffd607bf3100b11c6e1dab6d2fc44e72/torch-2.11.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:01018087326984a33b64e04c8cb5c2795f9120e0d775ada1f6638840227b04d7" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/bf/46/4419098ed6d801750f26567b478fc185c3432e11e2cad712bc6b4c2ab0d0/torch-2.11.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:8245477871c3700d4370352ffec94b103cfcb737229445cf9946cddb7b2ca7cd" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "torch"
|
|
||||||
version = "2.11.0+cu128"
|
|
||||||
source = { registry = "https://download.pytorch.org/whl/cu128" }
|
|
||||||
resolution-markers = [
|
|
||||||
"(python_full_version >= '3.12' and sys_platform == 'linux') or (python_full_version >= '3.12' and sys_platform == 'win32')",
|
|
||||||
"(python_full_version == '3.11.*' and sys_platform == 'linux') or (python_full_version == '3.11.*' and sys_platform == 'win32')",
|
|
||||||
"(python_full_version < '3.11' and sys_platform == 'linux') or (python_full_version < '3.11' and sys_platform == 'win32')",
|
|
||||||
]
|
|
||||||
dependencies = [
|
|
||||||
{ name = "cuda-bindings", marker = "sys_platform != 'win32'" },
|
|
||||||
{ name = "cuda-toolkit", extra = ["cublas", "cudart", "cufft", "cufile", "cupti", "curand", "cusolver", "cusparse", "nvjitlink", "nvrtc", "nvtx"], marker = "sys_platform != 'win32'" },
|
|
||||||
{ name = "filelock" },
|
|
||||||
{ name = "fsspec" },
|
|
||||||
{ name = "jinja2" },
|
|
||||||
{ name = "networkx", version = "3.4.2", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version < '3.11'" },
|
|
||||||
{ name = "networkx", version = "3.6.1", source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }, marker = "python_full_version >= '3.11'" },
|
|
||||||
{ name = "nvidia-cudnn-cu12", marker = "sys_platform != 'win32'" },
|
|
||||||
{ name = "nvidia-cusparselt-cu12", marker = "sys_platform != 'win32'" },
|
|
||||||
{ name = "nvidia-nccl-cu12", marker = "sys_platform != 'win32'" },
|
|
||||||
{ name = "nvidia-nvshmem-cu12", marker = "sys_platform != 'win32'" },
|
|
||||||
{ name = "setuptools" },
|
|
||||||
{ name = "sympy" },
|
|
||||||
{ name = "triton", marker = "sys_platform != 'win32'" },
|
|
||||||
{ name = "typing-extensions" },
|
|
||||||
]
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp310-cp310-manylinux_2_28_aarch64.whl", hash = "sha256:5ac6e34681d5a0e527edb741b38254899cd03087a7dd7e841791a4ee0a5e7011", upload-time = "2026-04-27T17:32:32Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp310-cp310-manylinux_2_28_x86_64.whl", hash = "sha256:72d53f3176a69cc20710c4ecb95f7dc4c6ba10c4e4eda45b8396ee79ee40f75a", upload-time = "2026-04-27T17:33:09Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp310-cp310-win_amd64.whl", hash = "sha256:7c792fe95ad5edaf622cf9e4f5573f5aecf2bc0654c7e866eda6134088f95d72", upload-time = "2026-04-27T17:34:55Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp311-cp311-manylinux_2_28_aarch64.whl", hash = "sha256:d76f08e212285bd84c4c5a3472417f8eb4ee72e4067a604f7508dbfa2119771f", upload-time = "2026-04-27T17:36:45Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp311-cp311-manylinux_2_28_x86_64.whl", hash = "sha256:c9a7ca4c74fae10a58e6175b4b2cea953f9322bb6562bbf339ad6a05f52190ad", upload-time = "2026-04-27T17:37:32Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp311-cp311-win_amd64.whl", hash = "sha256:90ef0c2454e5296a9fb021ddd42252e4ce1abe2c0a4988a173ef90a6cded0bf5", upload-time = "2026-04-27T17:39:29Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp312-cp312-manylinux_2_28_aarch64.whl", hash = "sha256:9c8f38efee365cb9d334de8a83ce52fc7e5fc9e5a7b0853285efa1b69e00b0f2", upload-time = "2026-04-27T17:41:30Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp312-cp312-manylinux_2_28_x86_64.whl", hash = "sha256:d252cf975fb18c94a85336323ad425f473df56dab35a44b00399bd70c7a3b997", upload-time = "2026-04-27T17:42:06Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp312-cp312-win_amd64.whl", hash = "sha256:7c78215c3af4f62e63f2b2e360f1722fc719b0853c7ac22666483d9810613a4c", upload-time = "2026-04-27T17:43:49Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp313-cp313-manylinux_2_28_aarch64.whl", hash = "sha256:7db3580106bba044da5b8950f3fb8fe5f31999eaab3f6a3aa2ac5d202c3684d2", upload-time = "2026-04-27T17:45:35Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp313-cp313-manylinux_2_28_x86_64.whl", hash = "sha256:db964b33c55035a72ab3e2162287af8f1cc276039c65d015740cc88c26dcedf7", upload-time = "2026-04-27T17:46:18Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp313-cp313-win_amd64.whl", hash = "sha256:6f367e62fd81b75cdf23ca4b75ced834d2db2cf98d1588ac935bde345de9de23", upload-time = "2026-04-27T17:48:09Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp313-cp313t-manylinux_2_28_aarch64.whl", hash = "sha256:cd1cf1005c5fe419194ee294b7b584ba5ad0f2fb1778b3fe5a7b9c3f4617ddbc", upload-time = "2026-04-27T17:50:01Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp313-cp313t-manylinux_2_28_x86_64.whl", hash = "sha256:74b628dbc71603977b09f4e140792c6e997081a35ef3421555f3f6e201b81210", upload-time = "2026-04-27T17:50:42Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp313-cp313t-win_amd64.whl", hash = "sha256:c2a5984deba8e001d166bf9cb83b8351f63a28b009e1a2fa0e4bbf08c90b259b", upload-time = "2026-04-27T17:52:32Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp314-cp314-manylinux_2_28_aarch64.whl", hash = "sha256:baa52f7b8a53cab16587b10f1c27d1000ca033f97236878b685b75d5a1b92408", upload-time = "2026-04-27T17:54:24Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp314-cp314-manylinux_2_28_x86_64.whl", hash = "sha256:d389a850677f0d24dafae1573644034428d8d3b9c80b51d55ba62fed7e6c8777", upload-time = "2026-04-27T17:55:03Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp314-cp314-win_amd64.whl", hash = "sha256:d6c21797ff75271b4fbdd905e2d703be4ecea5ea5bbdde4d1c201e9c71bc411d", upload-time = "2026-04-27T17:56:46Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp314-cp314t-manylinux_2_28_aarch64.whl", hash = "sha256:06849e9311dbb0617c97557d9c26c99a9e1c4f2ac9cb8e9b6d9b420d522acb91", upload-time = "2026-04-27T17:58:48Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp314-cp314t-manylinux_2_28_x86_64.whl", hash = "sha256:169a9987e1f84f0c5eee07544b3a34827a163ac9180e23abf0c3548f1335762c", upload-time = "2026-04-27T17:59:26Z" },
|
|
||||||
{ url = "https://download-r2.pytorch.org/whl/cu128/torch-2.11.0%2Bcu128-cp314-cp314t-win_amd64.whl", hash = "sha256:d86c125d720c2c368c53bd1a4ef062916d91fa965c10448c74c78b5d039faf2d", upload-time = "2026-04-27T18:01:14Z" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "tqdm"
|
name = "tqdm"
|
||||||
version = "4.70.0"
|
version = "4.70.0"
|
||||||
@@ -1406,27 +936,6 @@ wheels = [
|
|||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/f9/1c/01bfd571a64e7f270e6bab5e33777debe0edc56759233ce84f27dec92d14/tqdm-4.70.0-py3-none-any.whl", hash = "sha256:7f585706bfddbdebf89daac705b2dfcc16890130727d3197ca62c732b4310953" },
|
{ url = "https://mirrors.aliyun.com/pypi/packages/f9/1c/01bfd571a64e7f270e6bab5e33777debe0edc56759233ce84f27dec92d14/tqdm-4.70.0-py3-none-any.whl", hash = "sha256:7f585706bfddbdebf89daac705b2dfcc16890130727d3197ca62c732b4310953" },
|
||||||
]
|
]
|
||||||
|
|
||||||
[[package]]
|
|
||||||
name = "triton"
|
|
||||||
version = "3.6.0"
|
|
||||||
source = { registry = "https://mirrors.aliyun.com/pypi/simple/" }
|
|
||||||
wheels = [
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/44/ba/b1b04f4b291a3205d95ebd24465de0e5bf010a2df27a4e58a9b5f039d8f2/triton-3.6.0-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6c723cfb12f6842a0ae94ac307dba7e7a44741d720a40cf0e270ed4a4e3be781" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/8c/f7/f1c9d3424ab199ac53c2da567b859bcddbb9c9e7154805119f8bd95ec36f/triton-3.6.0-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a6550fae429e0667e397e5de64b332d1e5695b73650ee75a6146e2e902770bea" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/0f/2c/96f92f3c60387e14cc45aed49487f3486f89ea27106c1b1376913c62abe4/triton-3.6.0-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:49df5ef37379c0c2b5c0012286f80174fcf0e073e5ade1ca9a86c36814553651" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/e0/12/b05ba554d2c623bffa59922b94b0775673de251f468a9609bc9e45de95e9/triton-3.6.0-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e8e323d608e3a9bfcc2d9efcc90ceefb764a82b99dea12a86d643c72539ad5d3" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/17/5d/08201db32823bdf77a0e2b9039540080b2e5c23a20706ddba942924ebcd6/triton-3.6.0-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:374f52c11a711fd062b4bfbb201fd9ac0a5febd28a96fb41b4a0f51dde3157f4" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/ab/a8/cdf8b3e4c98132f965f88c2313a4b493266832ad47fb52f23d14d4f86bb5/triton-3.6.0-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:74caf5e34b66d9f3a429af689c1c7128daba1d8208df60e81106b115c00d6fca" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/3c/12/34d71b350e89a204c2c7777a9bba0dcf2f19a5bfdd70b57c4dbc5ffd7154/triton-3.6.0-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:448e02fe6dc898e9e5aa89cf0ee5c371e99df5aa5e8ad976a80b93334f3494fd" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/f9/0b/37d991d8c130ce81a8728ae3c25b6e60935838e9be1b58791f5997b24a54/triton-3.6.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:10c7f76c6e72d2ef08df639e3d0d30729112f47a56b0c81672edc05ee5116ac9" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/ce/4e/41b0c8033b503fd3cfcd12392cdd256945026a91ff02452bef40ec34bee7/triton-3.6.0-cp313-cp313t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1722e172d34e32abc3eb7711d0025bb69d7959ebea84e3b7f7a341cd7ed694d6" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/35/f8/9c66bfc55361ec6d0e4040a0337fb5924ceb23de4648b8a81ae9d33b2b38/triton-3.6.0-cp313-cp313t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d002e07d7180fd65e622134fbd980c9a3d4211fb85224b56a0a0efbd422ab72f" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/49/55/5ecf0dcaa0f2fbbd4420f7ef227ee3cb172e91e5fede9d0ecaddc43363b4/triton-3.6.0-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ef5523241e7d1abca00f1d240949eebdd7c673b005edbbce0aca95b8191f1d43" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/df/3d/9e7eee57b37c80cec63322c0231bb6da3cfe535a91d7a4d64896fcb89357/triton-3.6.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a17a5d5985f0ac494ed8a8e54568f092f7057ef60e1b0fa09d3fd1512064e803" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/48/db/56ee649cab5eaff4757541325aca81f52d02d4a7cd3506776cad2451e060/triton-3.6.0-cp314-cp314t-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b3a97e8ed304dfa9bd23bb41ca04cdf6b2e617d5e782a8653d616037a5d537d" },
|
|
||||||
{ url = "https://mirrors.aliyun.com/pypi/packages/f6/56/6113c23ff46c00aae423333eb58b3e60bdfe9179d542781955a5e1514cb3/triton-3.6.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:46bd1c1af4b6704e554cad2eeb3b0a6513a980d470ccfa63189737340c7746a7" },
|
|
||||||
]
|
|
||||||
|
|
||||||
[[package]]
|
[[package]]
|
||||||
name = "typing-extensions"
|
name = "typing-extensions"
|
||||||
version = "4.16.0"
|
version = "4.16.0"
|
||||||
|
|||||||
Reference in New Issue
Block a user