Skip to content

Commit 325159b

Browse files
committed
feat(deepxctl): add --rm flag to run command for one-shot execution
- cmd/run.go: Add --rm flag that FLUSHDBs Redis then calls ExecShutdown after successful dx execution, providing a one-shot cleanup workflow - cmd/shutdown.go: Export ExecShutdown() so run.go can reuse shutdown logic - cmd/boot.go, cmd/common.go: New boot subcommand with PID file tracking and shared print/helper utilities - internal/process/manager.go: Add log file redirection, Detach support and SetWorkDir/SetLogDir for multi-process orchestration - main.go: Update CLI to support boot/run/shutdown subcommands - Update .claude/CLAUDE.md and doc/deepxctl/CLAUDE.md with --rm workflow
1 parent 0ec09b4 commit 325159b

16 files changed

Lines changed: 1454 additions & 1080 deletions

File tree

.claude/CLAUDE.md

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -114,13 +114,22 @@ deepxctl 将生命周期拆分为三个独立命令,**所有组件间通信严
114114

115115
```bash
116116
deepxctl boot # 构建 + 启动 op-metal、heap-metal、VM,写入 PID 文件 /tmp/deepx-boot.json
117-
deepxctl run a.dx # 检测 boot 状态 → 加载 dx → 创建 vthread → 等待结果 (可多次执行)
117+
deepxctl run a.dx # 检测 boot 状态 → loader 加载 dx → 自动检测 /func/main → 等待结果 (可多次执行)
118+
# --rm: 执行后自动 FLUSHDB + shutdown (一键清理)
119+
# --entry <name>: 手动指定入口函数 (即使文件无顶层调用也会执行)
118120
deepxctl shutdown # 有序退出: plats → VM → 心跳验证 → 清理
119121
```
120122

123+
**dxlang 执行语义 (v2)**
124+
- **纯定义文件** (只有 `def` 块,无顶层调用): `deepxctl run` 仅加载函数定义到 `/src/func/*`,不执行任何代码。VM 的 `/func/main` 监视器保持等待。
125+
- **包含顶层调用的文件** (在 `def { }` 块外部有 `funcName(args) -> outputs`): loader 自动写入 `/func/main`,VM 检测后创建 vthread 并执行。deepxctl 轮询等待结果。
126+
- **手动指定入口**: `--entry <funcName>` 绕过顶层调用检测,直接写入 `/func/main`
127+
- 关键 Redis key: `/func/main` — 入口协议 (`{"entry":"funcName","reads":[...],"writes":[...]}` → VM 认领后改为 `{"vtid":"...","status":"executing"}` → 最终 `{"vtid":"...","status":"done"}`)
128+
121129
**通信规则**
122130
- 业务队列: `cmd:op-metal:0`, `cmd:heap-metal:0`, `notify:vm`
123131
- 系统队列 (`sys:` 前缀): `sys:cmd:op-metal:0`, `sys:cmd:heap-metal:0`, `sys:cmd:vm:0`
132+
- 入口协议: `/func/main` (loader → VM → deepxctl 三方协作)
124133
- 心跳上报: `/sys/heartbeat/op-metal:0`, `/sys/heartbeat/heap-metal:0`, `/sys/heartbeat/vm:0`
125134
- 各组件每 2s 上报 `{"ts":...,"status":"running","pid":...}`
126135
- 退出时上报 `{"status":"stopped"}` — shutdown 以此验证退出完成

.claude/commands/CLAUDE.md

Lines changed: 15 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@
2626
| 命令 | 说明 |
2727
|------|------|
2828
| `/boot` | 构建并启动所有服务 (deepxctl boot) |
29-
| `/run <file.dx>` | 运行 .dx 文件 (deepxctl run) |
29+
| `/run <file.dx>` | 加载 .dx 文件 (如有顶层调用则自动执行) → deepxctl run |
30+
| `/run --entry <f> <file.dx>` | 手动指定入口函数执行 |
3031
| `/shutdown` | 停止所有 booted 服务 (deepxctl shutdown) |
3132
| `/status` | 查看所有服务状态 → `make status` |
3233
| `/pipeline` | 完整联调流水线 → `make pipeline` |
@@ -46,8 +47,18 @@
4647
# 1. 构建并启动所有服务 (一次性)
4748
deepxctl boot
4849

49-
# 2. 运行 .dx 文件 (可多次重复)
50-
deepxctl run example/dxlang/native/arith/add.dx
50+
# 2a. 加载纯定义文件 (只定义函数,不执行)
51+
deepxctl run example/dxlang/call/add_test.dx
52+
# → "Loaded 1 function(s) into KV Space."
53+
# → 没有顶层调用,VM 的 /func/main 监视器保持等待
54+
55+
# 2b. 加载含顶层调用的文件 (自动执行)
56+
deepxctl run my_file_with_call.dx
57+
# → loader 检测到顶层调用 → 写入 /func/main → VM 自动创建 vthread → 执行
58+
59+
# 2c. 手动指定入口函数
60+
deepxctl run --entry native_arith example/dxlang/native/arith/add.dx
61+
# → 绕过顶层调用检测,直接以 native_arith 为入口执行
5162

5263
# 3. 停止所有服务
5364
deepxctl shutdown
@@ -94,6 +105,7 @@ plats (op-metal / heap-metal / io-metal / VM) 由 deepxctl 通过 subprocess 管
94105
| `cmd:heap-metal:0` | 业务 | 堆内存管理指令 |
95106
| `cmd:io-metal:0` | 业务 | I/O 指令 (print/save/load) |
96107
| `notify:vm` | 业务 | VThread 调度通知 |
108+
| `/func/main` | **入口** | loader→VM→deepxctl 三方协作: `{"entry":"f","reads":[...],"writes":[...]}``{"vtid":"...","status":"executing"}``{"vtid":"...","status":"done"}` |
97109
| `sys:cmd:op-metal:0` | 系统 | op-metal shutdown |
98110
| `sys:cmd:heap-metal:0` | 系统 | heap-metal shutdown |
99111
| `sys:cmd:io-metal:0` | 系统 | io-metal shutdown |

doc/deepxctl/CLAUDE.md

Lines changed: 48 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -148,46 +148,56 @@ deepxctl **绝对不能**读写的 key:
148148

149149
---
150150

151-
## 5. `deepxctl run` 执行流程图
151+
## 5. 命令架构
152152

153+
deepxctl 将生命周期拆分为三个独立命令:
154+
155+
```
156+
deepxctl boot → 构建 + 启动 op-metal、heap-metal、VM,写入 PID 文件
157+
deepxctl run a.dx → 检测 boot 状态 → 加载 dx → 创建 vthread → 轮询等待结果
158+
deepxctl shutdown → 有序退出: plats → VM → 心跳验证 → 清理 PID 文件
153159
```
154-
deepxctl run xxx.dx
160+
161+
### `deepxctl run` 执行流程
162+
163+
```
164+
deepxctl run xxx.dx [--rm]
155165
156166
│ (deepxctl 负责的部分)
157167
158-
├─ connect Redis ───────────── PING
159-
├─ FLUSHDB ─────────────────── 重置 KV 空间
160-
├─ build (if needed) ───────── exec build.sh (子进程)
161-
├─ start op-plat ───────────── exec binary (子进程)
162-
├─ start heap-plat ─────────── exec binary (子进程)
163-
├─ start VM ────────────────── exec binary (子进程)
164-
├─ wait ready ──────────────── GET /sys/* (检查 status=running)
165-
├─ loader ──────────────────── exec loader binary (子进程加载 .dx)
166-
167-
│ (VM + op-plat + heap-plat 负责的部分 — deepxctl 不参与)
168-
169-
├─ create vthread ──────────── SET /vthread/<vtid> (deepxctl 只做初始创建)
170-
├─ wake VM ─────────────────── LPUSH notify:vm
171-
│ │
172-
│ ▼ (VM 接手)
173-
│ VM 拾取 → CALL 翻译 → dispatch → op/heap
174-
│ │
175-
│ ▼ (op/heap 完成)
176-
│ VM PC++ ... 循环直到 done/error
168+
├─ [1/3] Check services ─────── 检查 boot PID 文件 + Redis 服务就绪
169+
├─ [2/3] Load dx ────────────── exec loader 二进制 (子进程加载 .dx 到 /src/func/)
170+
├─ [3/3] Execute ──────────────
171+
│ ├─ create vthread ───────── SET /vthread/<vtid> (初始状态)
172+
│ ├─ wake VM ──────────────── LPUSH notify:vm
173+
│ │ │
174+
│ │ ▼ (VM 接手 — deepxctl 不参与)
175+
│ │ VM 拾取 → CALL 翻译 → dispatch → op/heap → PC++
176+
│ │
177+
│ └─ poll status ──────────── GET /vthread/<vtid> (轮询 status)
178+
│ ├─ done → print result ✓
179+
│ └─ error → print error ✗
177180
178-
│ (deepxctl 恢复监控)
179-
180-
├─ poll status ─────────────── GET /vthread/<vtid> (轮询 status)
181-
├─ ┌ done → print result ✓
182-
│ └ error → print error ✗
183-
├─ kill all subprocesses ──── SIGTERM → SIGKILL
184-
└─ exit
181+
└─ [--rm] Cleanup (可选)
182+
├─ FLUSHDB ───────────────── 重置 Redis KV 空间
183+
└─ ExecShutdown ──────────── 复用 shutdown 逻辑: plats → VM → 清理
185184
```
186185

187186
**关键分界线**:deepxctl 在 `notify:vm` 之后就不再参与执行——后续所有步骤(CALL 翻译、
188187
指令 dispatch、op 执行、done 通知)都是 VM/op-plat/heap-plat 之间通过 Redis 的协作。
189188
deepxctl 只是**旁观**:轮询 `/vthread/<vtid>` 的 status 字段,直到 `done``error`
190189

190+
### `--rm` 一键清理
191+
192+
`deepxctl run a.dx --rm` 在 dx 代码执行成功后自动:
193+
1. **FLUSHDB** — 重置 Redis KV 空间
194+
2. **shutdown** — 复用 `deepxctl shutdown` 的完整退出逻辑(Redis sys:shutdown 命令 → 心跳验证 → 清理 PID 文件 → OS 信号兜底)
195+
196+
等价于手动执行:
197+
```bash
198+
deepxctl run a.dx && make reset-redis && deepxctl shutdown
199+
```
200+
191201
---
192202

193203
## 6. 不允许的"快捷方式"
@@ -243,29 +253,28 @@ VM 执行到 `[0,0]` 时:
243253

244254
---
245255

246-
## 9. MVP 范围
247-
248-
只实现 `deepxctl run xxx.dx`,文件清单:
256+
## 9. 当前文件清单
249257

250258
```
251259
tool/deepxctl/
252260
├── main.go
253261
├── go.mod
254262
├── cmd/
255-
│ └── run.go ← run 子命令入口
263+
│ ├── boot.go ← boot 子命令 (构建 + 启动服务)
264+
│ ├── run.go ← run 子命令 (加载 dx + 创建 vthread + 轮询)
265+
│ ├── shutdown.go ← shutdown 子命令 (有序退出服务)
266+
│ └── common.go ← 共享打印/辅助函数
256267
├── internal/
257-
│ ├── redis/redis.go ← 连接 + FLUSHDB + 状态检查
258-
│ ├── build/builder.go ← exec build.sh
259-
│ ├── process/manager.go ← 子进程生命周期
260-
│ ├── loader/loader.go ← exec loader 二进制
268+
│ ├── redis/redis.go ← 连接 + FLUSHDB + 状态检查 + vthread 管理
269+
│ ├── builder/builder.go ← exec build.sh
270+
│ ├── process/manager.go ← 子进程生命周期
261271
│ └── executor/executor.go ← vthread 创建 + 轮询
262-
└── tensor/ ← 已有,不动
272+
└── tensor/ ← tensor 文件操作 (print/save/load)
263273
```
264274

265275
不做的:
266-
- `build` / `start` / `stop` / `status` / `clean` / `install` 子命令
267276
- YAML 配置文件解析
268-
- JSON 输出格式
277+
- JSON 输出格式(结构化)
269278
- 多平台自动检测(先只支持 metal)
270279
- 守护进程模式
271280
- 远程 Redis TLS

executor/heap-metal/build.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,4 +6,6 @@ mkdir -p "$BUILD_DIR"
66
cd "$BUILD_DIR"
77
cmake "$DIR"
88
cmake --build . -j$(sysctl -n hw.ncpu 2>/dev/null || nproc)
9+
# Copy runtime dependencies (rpath)
10+
cp -f common-metal/libdeepx_common_metal.a "$BUILD_DIR/" 2>/dev/null || true
911
echo "Built: $BUILD_DIR/deepx-heap-metal"

executor/heap-metal/src/main.cpp

Lines changed: 56 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -16,9 +16,12 @@
1616
using namespace deepx::heap;
1717
using json = nlohmann::json;
1818

19-
static const char *HEAP_QUEUE = "cmd:heap-metal:0";
20-
static const char *INSTANCE_KEY = "/sys/heap-plat/heap-metal:0";
21-
static const int BLOCK_TIMEOUT_SEC = 5;
19+
static const char *HEAP_QUEUE = "cmd:heap-metal:0";
20+
static const char *SYS_QUEUE = "sys:cmd:heap-metal:0";
21+
static const char *INSTANCE_KEY = "/sys/heap-plat/heap-metal:0";
22+
static const char *HEARTBEAT_KEY = "/sys/heartbeat/heap-metal:0";
23+
static const int BLOCK_TIMEOUT_SEC = 5;
24+
static const int HEARTBEAT_INTERVAL_SEC = 2;
2225

2326
// ── Redis helpers ──
2427

@@ -50,6 +53,15 @@ static bool redis_set(redisContext *c, const std::string &key, const std::string
5053
return ok;
5154
}
5255

56+
static void update_heartbeat(redisContext *c, const std::string &status) {
57+
json hb;
58+
hb["ts"] = std::chrono::duration_cast<std::chrono::seconds>(
59+
std::chrono::system_clock::now().time_since_epoch()).count();
60+
hb["status"] = status;
61+
hb["pid"] = getpid();
62+
redis_set(c, HEARTBEAT_KEY, hb.dump());
63+
}
64+
5365
static void register_instance(redisContext *c) {
5466
json reg;
5567
reg["program"] = "heap-metal";
@@ -248,11 +260,17 @@ int main(int argc, char **argv) {
248260
FileRegistry reg(registry_path);
249261
LifecycleManager mgr(&reg);
250262

251-
std::cout << "[heap] listening on " << HEAP_QUEUE << "\n";
263+
std::cout << "[heap] listening on " << HEAP_QUEUE << " + " << SYS_QUEUE << "\n";
264+
std::cout << "[heap] heartbeat → " << HEARTBEAT_KEY << " (every " << HEARTBEAT_INTERVAL_SEC << "s)\n";
265+
266+
// 初始心跳
267+
update_heartbeat(redis, "running");
252268

253-
// 消费循环
254-
while (true) {
255-
redisReply *r = redis_cmd(redis, "BLPOP %s %d", HEAP_QUEUE, BLOCK_TIMEOUT_SEC);
269+
// 消费循环 (同时监听业务队列和系统命令队列)
270+
std::atomic<bool> running{true};
271+
auto last_heartbeat = std::chrono::steady_clock::now();
272+
while (running) {
273+
redisReply *r = redis_cmd(redis, "BLPOP %s %s %d", HEAP_QUEUE, SYS_QUEUE, BLOCK_TIMEOUT_SEC);
256274
if (!r) {
257275
// Redis 断连 → 无限重连(不自退,heap-plat 由元程控制退出)
258276
std::cerr << "[heap] Redis disconnected, reconnecting...\n";
@@ -266,9 +284,18 @@ int main(int argc, char **argv) {
266284
}
267285
}
268286
register_instance(redis);
287+
last_heartbeat = std::chrono::steady_clock::now();
288+
update_heartbeat(redis, "running");
269289
continue;
270290
}
271291

292+
// ── 心跳上报 ──
293+
auto now = std::chrono::steady_clock::now();
294+
if (std::chrono::duration_cast<std::chrono::seconds>(now - last_heartbeat).count() >= HEARTBEAT_INTERVAL_SEC) {
295+
update_heartbeat(redis, "running");
296+
last_heartbeat = now;
297+
}
298+
272299
if (r->type == REDIS_REPLY_NIL) {
273300
// BLPOP timeout — no tasks
274301
REDIS_FREE(r);
@@ -280,9 +307,28 @@ int main(int argc, char **argv) {
280307
continue;
281308
}
282309

310+
std::string queue_name(r->element[0]->str);
283311
std::string payload(r->element[1]->str);
284312
REDIS_FREE(r);
285313

314+
// ── 系统命令处理 ──
315+
if (queue_name == SYS_QUEUE) {
316+
try {
317+
json sys_cmd = json::parse(payload);
318+
std::string cmd = sys_cmd.value("cmd", "");
319+
if (cmd == "shutdown") {
320+
std::cout << "[heap] received sys shutdown command, exiting...\n";
321+
running = false;
322+
} else {
323+
std::cerr << "[heap] unknown sys command: " << cmd << "\n";
324+
}
325+
} catch (const std::exception &e) {
326+
std::cerr << "[heap] sys cmd JSON parse error: " << e.what() << "\n";
327+
}
328+
continue;
329+
}
330+
331+
// ── 业务命令处理 ──
286332
// 解析 JSON
287333
json task;
288334
try {
@@ -318,7 +364,10 @@ int main(int argc, char **argv) {
318364
}
319365

320366
mgr.shutdown();
367+
// 上报 stopped 心跳,然后注销
321368
if (redis) {
369+
update_heartbeat(redis, "stopped");
370+
std::cout << "[heap] final heartbeat: stopped\n";
322371
redis_cmd(redis, "DEL %s", INSTANCE_KEY);
323372
redisFree(redis);
324373
}

executor/op-metal/build.sh

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,5 +6,7 @@ mkdir -p "$BUILD_DIR"
66
cd "$BUILD_DIR"
77
cmake "$DIR"
88
cmake --build . -j$(sysctl -n hw.ncpu 2>/dev/null || nproc)
9+
# Copy runtime dependencies (rpath: @rpath/libdeepx_metal.dylib)
10+
cp -f libdeepx_metal.dylib default.metallib "$BUILD_DIR/" 2>/dev/null || true
911
echo "Built: $BUILD_DIR/deepx-op-metal"
1012
echo "Test: $BUILD_DIR/test/shm/test_cross_process"

executor/vm/cmd/loader/main.go

Lines changed: 32 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,7 @@ package main
1818

1919
import (
2020
"context"
21+
"encoding/json"
2122
"fmt"
2223
"log"
2324
"os"
@@ -57,20 +58,45 @@ func main() {
5758

5859
log.Printf("found %d .dx file(s)", len(files))
5960
loaded := 0
61+
entryCreated := false
6062
for _, f := range files {
61-
fn, err := testutil.LoadDxFile(f)
63+
df, err := testutil.ParseDxFile(f)
6264
if err != nil {
6365
log.Printf("SKIP %s: %v", f, err)
6466
continue
6567
}
66-
if err := fn.RegisterFunc(ctx, rdb); err != nil {
67-
log.Printf("FAIL %s: %v", f, err)
68-
continue
68+
69+
// Register all function definitions
70+
for i := range df.Funcs {
71+
fn := &df.Funcs[i]
72+
if err := fn.RegisterFunc(ctx, rdb); err != nil {
73+
log.Printf("FAIL %s: %v", f, err)
74+
continue
75+
}
76+
loaded++
77+
log.Printf("OK %-50s → /src/func/%-30s (%d body lines)", f, fn.Name, len(fn.Body))
78+
}
79+
80+
// If file has top-level calls, write /func/main to trigger VM execution
81+
if len(df.TopLevelCalls) > 0 {
82+
tc := df.TopLevelCalls[0] // first top-level call is the entry point
83+
entryData, _ := json.Marshal(map[string]interface{}{
84+
"entry": tc.FuncName,
85+
"reads": tc.Args,
86+
"writes": tc.Outputs,
87+
})
88+
if err := rdb.Set(ctx, "/func/main", entryData, 0).Err(); err != nil {
89+
log.Printf("FAIL %s: write /func/main: %v", f, err)
90+
continue
91+
}
92+
entryCreated = true
93+
log.Printf("ENTRY /func/main → %s (reads=%v writes=%v)", tc.FuncName, tc.Args, tc.Outputs)
6994
}
70-
loaded++
71-
log.Printf("OK %-50s → /src/func/%-30s (%d body lines)", f, fn.Name, len(fn.Body))
7295
}
7396
log.Printf("loaded %d/%d functions into Redis", loaded, len(files))
97+
if entryCreated {
98+
log.Printf("ENTRY /func/main set — VM will auto-execute")
99+
}
74100
}
75101

76102
func printUsage() {

0 commit comments

Comments
 (0)