训练问题

[JiangWu0826]

/Trinity-RFT/examples/dapo_math
开发者您好，我在测试dapo这个示例的时候，遇到了评估这个path to AIME2024数据集一直报错的问题，要么识别不到路径，要么就是数据集没识别到
ERROR 11-05 10:20:36 [launcher.py:133] FileNotFoundError: Couldn't find any data file at /data/Trinity-RFT/aime_2024_problems.parquet.

要么就是字符串的问题，想问一下您这边是跑通的嘛，数据集AIME2024怎么给的呢，我把eval_tasksets:那部分删掉是可以正常运行的

project: Trinity-RFT-example
name: dapo
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
model:
model_path: ${oc.env:TRINITY_MODEL_PATH,/data/LLM/llm_rl/Qwen2.5-1.5B}
max_response_tokens: 20480
max_model_len: 21504
algorithm:
algorithm_type: grpo
repeat_times: 16
policy_loss_fn_args:
clip_range_low: 0.2
clip_range_high: 0.28
cluster:
node_num: 1
gpu_per_node: 8
buffer:
total_epochs: 1
batch_size: 32
explorer_input:
taskset:
name: dapo-math
storage_type: file
path: /data/Trinity-RFT/open-r1/DAPO-Math-17k-Processed
subset_name: all
format:
prompt_key: 'prompt'
response_key: 'solution'
system_prompt: 'Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.'
rollout_args:
temperature: 1.0
logprobs: 0
workflow_args:
use_base: true
reward_fn_args:
enable_overlong_penalty: true
penalty_factor: 1.0
max_response_length: 20480
cache_length: 4096
eval_tasksets:
- name: AIME2024
storage_type: file
path: ${oc.env:TRINITY_TASKSET_PATH, /data/Trinity-RFT/aime_2024} # e.g. path to AIME2024
repeat_times: 32
format:
prompt_key: 'question'
response_key: 'answer'
system_prompt: 'Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.'
rollout_args:
temperature: 1.0
top_p: 0.7
default_workflow_type: 'math_boxed_workflow'
default_reward_fn_type: 'math_dapo_reward'
trainer_input:
experience_buffer:
name: math_buffer
storage_type: queue
explorer:
eval_interval: 10
runner_per_model: 8
rollout_model:
engine_num: 4
tensor_parallel_size: 1
enable_prefix_caching: false
enforce_eager: true
dtype: bfloat16
seed: 42
synchronizer:
sync_method: 'nccl'
sync_interval: 16
sync_timeout: 1200
trainer:
save_interval: 100
trainer_config:
actor_rollout_ref:
model:
use_remove_padding: true
actor:
use_dynamic_bsz: true
ppo_max_token_len_per_gpu: 22000
ulysses_sequence_parallel_size: 1
optim:
lr: 1e-6
lr_warmup_steps: 20
ref:
log_prob_use_dynamic_bsz: ${trainer.trainer_config.actor_rollout_ref.actor.use_dynamic_bsz}
log_prob_max_token_len_per_gpu: ${trainer.trainer_config.actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
ulysses_sequence_parallel_size: ${trainer.trainer_config.actor_rollout_ref.actor.ulysses_sequence_parallel_size} # sp size

[hiyuchang]

您好，eval_task 读取的方式类似于 huggingface 中 datasets.load_dataset() 的读取方式。

假设下载的AIME-24路径如下

AIME-2024/
└── train.parquet

那么path应该写成父目录xxxxx/AIME-24即可，不需要写至train.parquet。

你可以利用下面的代码来检查能否正确读取数据，如果可以正常输出，则Trinity可以正常读取此路径：

path = "/path/to/AIME-2024"
from datasets import load_dataset
dataset = load_dataset(path, split="train")
print(dataset[0])
print(len(dataset))

[JiangWu0826]

感谢您的回复，但是我感觉在DAPO-Math-17k-Processed数据集的训练曲线不太对劲，rollout/accuracy/mean和其他测试脚本在gsm8k的差距很多。
训练脚本如下：
project: Trinity-RFT-example
name: dapo
checkpoint_root_dir: ${oc.env:TRINITY_CHECKPOINT_ROOT_DIR,./checkpoints}
model:
model_path: ${oc.env:TRINITY_MODEL_PATH,/data/LLM/llm_rl/Qwen2.5-1.5B}
max_response_tokens: 20480
max_model_len: 21504
algorithm:
algorithm_type: grpo
repeat_times: 16
policy_loss_fn_args:
clip_range_low: 0.2
clip_range_high: 0.28
cluster:
node_num: 1
gpu_per_node: 8
buffer:
total_epochs: 1
batch_size: 32
explorer_input:
taskset:
name: dapo-math
storage_type: file
path: /data/Trinity-RFT/open-r1/DAPO-Math-17k-Processed/all
split: 'train'
format:
prompt_key: 'prompt'
response_key: 'solution'
system_prompt: 'Solve the following math problem step by step. The last line of your response should be of the form Answer: $Answer (without quotes) where $Answer is the answer to the problem.'
rollout_args:
temperature: 1.0
logprobs: 0
workflow_args:
use_base: true
reward_fn_args:
enable_overlong_penalty: true
penalty_factor: 1.0
max_response_length: 20480
cache_length: 4096
default_workflow_type: 'math_boxed_workflow'
default_reward_fn_type: 'math_dapo_reward'
trainer_input:
experience_buffer:
name: math_buffer
storage_type: queue
explorer:
eval_interval: 10
runner_per_model: 8
rollout_model:
engine_num: 4
tensor_parallel_size: 1
enable_prefix_caching: false
enforce_eager: true
dtype: bfloat16
seed: 42
synchronizer:
sync_method: 'nccl'
sync_interval: 16
sync_timeout: 1200
trainer:
save_interval: 100
trainer_config:
actor_rollout_ref:
model:
use_remove_padding: true
actor:
use_dynamic_bsz: true
ppo_max_token_len_per_gpu: 22000
ulysses_sequence_parallel_size: 1
optim:
lr: 1e-6
lr_warmup_steps: 20
ref:
log_prob_use_dynamic_bsz: ${trainer.trainer_config.actor_rollout_ref.actor.use_dynamic_bsz}
log_prob_max_token_len_per_gpu: ${trainer.trainer_config.actor_rollout_ref.actor.ppo_max_token_len_per_gpu}
ulysses_sequence_parallel_size: ${trainer.trainer_config.actor_rollout_ref.actor.ulysses_sequence_parallel_size} # sp size

[garyzhang99]

@JiangWu0826 抱歉，这里我们的config参数配置(learning_rate, batch_size, sync_interval等)其实是对应更强模型的，而不是当时example config里面填写的qwen2.5-1.5B-instruct。这套参数对1.5B模型不合适导致导致训练不稳定。

用更难的dapo数据训练的话，最好是可以将对应base模型换成强一些的模型（比如我们当时这套config使用的是qwen3-8b-base），对应reward曲线会稳定很多。如果就是想用较小size模型的话，也最好将sync_interval从16稍微调小一些在4左右，同时适当调大learning rate。

后面我们也会改进调整这个example的config和readme。

[JiangWu0826]

@JiangWu0826 抱歉，这里我们的config参数配置(learning_rate, batch_size, sync_interval等)其实是对应更强模型的，而不是当时example config里面填写的qwen2.5-1.5B-instruct。这套参数对1.5B模型不合适导致导致训练不稳定。

用更难的dapo数据训练的话，最好是可以将对应base模型换成强一些的模型（比如我们当时这套config使用的是qwen3-8b-base），对应reward曲线会稳定很多。如果就是想用较小size模型的话，也最好将sync_interval从16稍微调小一些在4左右，同时适当调大learning rate。

后面我们也会改进调整这个example的config和readme。

感谢您的回复，已经了解了，其实我们现在框架下dapo这个算法，只是在grpo的基础上改了clip数值吧，还没有涉及原论文的其他trick