Init test#160
Draft
ArabellaJi wants to merge 70 commits into
Draft
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Repository UI Review profile: ASSERTIVE Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…te prompt for single action
Allows CLI override without ++ prefix. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
vLLM 0.8+ enables chunked prefill by default with max_num_batched_tokens smaller than max_model_len=12288, causing a ValueError at startup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
entropy_from_logits materializes full logits tensor: micro_batch x seq_len x vocab_size x 2 bytes With micro_batch=4 and max_model_len=12288 this exceeds 80GB. micro_batch=1 reduces the allocation to ~3.6 GB. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Dumps training rollout JSONL to \/traj/\/ each step. Each file is named by step number (1.jsonl, 2.jsonl, ...) and contains input prompt, model output, score, and reward breakdown per trajectory. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
backward() activation memory scales with micro_batch x seq_len. At micro_batch=4 and seq_len=4096 on 3B model, backward OOMs. micro_batch=2 keeps activation memory within the ~79GB H100 budget. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the PPO update pipeline, batch.batch has no 'prompts' key. Use 'input_ids' (full sequence = prompt + response concatenated) and 'responses' (model output only) which are always present. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ruth Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
min_solution_steps=[1,3] puzzles are too easy: box/target often visible in local 3x3 window, no memory needed to solve them. [4,8] forces multi-step exploration where agent must navigate to unseen box/target, creating real memory pressure. search_depth=5->20 allows harder configurations to be generated. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
reward_std_per_sample is episode-level [num_episodes] but mask is turn-level [num_turns] when has_episode_ids=True. Indexing the episode tensor with the turn mask causes IndexError. Fix: expand reward_std from episode-level to turn-level using episode_ids before applying the mask. Same fix applied to the no-filter early-return path (rollout_filter_value >= 1). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
With enable_chunked_prefill=False, vLLM defaults max_num_batched_tokens to 8192, which is smaller than max_model_len=12288 and causes a startup ValueError. Must set both params explicitly. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- envs.yaml: rewrite in-context example (Position 2,2→2,1); row 0 now correctly shown as '? ? ? ? ?' when not in view; action/PLAN consistent; memory retention demo uses left-border discovery + Turn-1 cell retention - agent_trainer.py: override _dump_generations to include episode_id field in rollout JSONL, enabling cross-turn MAP quality analysis - pomdp_*_v2.slurm: new job scripts for v2 runs with fixed example Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Updated SYSTEM_PROMPT with corrected in-context example (Position (2,2) with row 0 = '?????' explicitly shown, matching the fixed envs.yaml) - Added TurnRecord dataclass tracking position, MAP row count, player position correctness, and row-0 correctness per turn - Added analyze_map() to extract and validate MAP blocks from responses - Updated print_summary() with MAP quality statistics across all turns Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…tions numpy.int64 is not JSON serializable; convert episode_ids, scores, and reward_extra_infos_dict values with a _to_py() helper. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Scans training trajectory JSONL files and reports per-step MAP quality: - % turns with valid 5-row MAP - % with player at correct (r,c) - % with row 0 = '?????' when r>=2 - Per-episode turn-level breakdown via --episode_detail Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
output field = full sequence (system + history + model response); re.search picked up the system-prompt example's <think> block. Use extract_last_think() to get the model's actual response. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
vLLM 0.8.2 V1 engine uses fork-based multiprocessing which conflicts with CUDA initialization. Fall back to V0 engine. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--max_context_window 1 matches mem1 training conditions (system + prev turn + current obs), avoiding context overflow that caused 75 parse failures/ep in the full-context 14B eval. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Matches training condition custom_envs.POMDPSokoban.env_config.min_solution_steps=[4,8] so 14B zero-shot success rate is comparable to 3B RL results. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
v2_resume: continues from step ~100 checkpoint, extends to 250 steps v3: fresh independent run to check if step-65 KL spike is systematic Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Tests whether larger model (7B) + RL training achieves both good MAP quality and task success, bridging the gap between: - 3B RL: success~35% but P@(r,c)%=6-16% - 14B zero-shot: P@(r,c)%=31% but success=24% Key changes vs 3B: micro_batch=2 (was 4), gpu_mem_util=0.75 (was 0.80) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.