Question: What is the linking rule between intermediate rewards and the triplet?

[kyk666123]

Hello, I encountered some issues when adding intermediate rewards to the SQL agent in the official example:

(1) I would like to know if the intermediate rewards emitted by emit_reward and the final reward returned by rollout are by default linked to the closest triplet with a None reward, when no link is specified. In the SQL agent example, in the check_query function, I analyze the check result according to certain rules and then add emit_reward (I only added emit_reward in the check function!). When I print the trajectory of a sample triplet, I noticed that the rewrite closest to the final_reward (which has a value of 1) also received a reward of 1, even though I didn’t add a reward to the rewrite. Why is this happening?

(2) Still in the SQL agent example: I only added emit_reward in check_query (with a value of 0.2, -0.2, or -0.1). I also modified the get_train_data_batch function in daemon.py of VERL, changing the reward_list. Originally, only the final reward was added, but I changed it to add the sum of the final reward and the intermediate rewards in the trace. However, I found that in the training/reward curve in wandb (with the value of np.mean(finished_id_to_final_reward.values())), negative numbers appear, even though I haven’t changed anything related to the final reward, which should only be 0 or 1. Therefore, negative numbers shouldn’t appear. Why is this happening?

I made two main changes to the code.

1. Extracted the intermediate rewards from the trace and saved them
2. Summed the intermediate and final rewards to form the actual reward of the triplet, then appended it to the reward_list.

step_reward = float(trace.get("step_reward", 0.0))
total_reward = final_reward + step_reward
reward_list.append(total_reward)

Thank you for helping with my confusion!

[ultmaster]

(1) I would like to know if the intermediate rewards emitted by emit_reward and the final reward returned by rollout are by default linked to the closest triplet with a None reward, when no link is specified. In the SQL agent example, in the check_query function, I analyze the check result according to certain rules and then add emit_reward (I only added emit_reward in the check function!). When I print the trajectory of a sample triplet, I noticed that the rewrite closest to the final_reward (which has a value of 1) also received a reward of 1, even though I didn’t add a reward to the rewrite. Why is this happening?

Did you return the reward from the rollout function? If you return it, the runner will automatically emit that reward if you haven't called emit_reward.

(2) Still in the SQL agent example: I only added emit_reward in check_query (with a value of 0.2, -0.2, or -0.1). I also modified the get_train_data_batch function in daemon.py of VERL, changing the reward_list. Originally, only the final reward was added, but I changed it to add the sum of the final reward and the intermediate rewards in the trace. However, I found that in the training/reward curve in wandb (with the value of np.mean(finished_id_to_final_reward.values())), negative numbers appear, even though I haven’t changed anything related to the final reward, which should only be 0 or 1. Therefore, negative numbers shouldn’t appear. Why is this happening?

Did you rewrite the final reward with the sum of intermediate and final reward. In that case, if your intermediate reward can be negative, I don't see why the final reward must be non-negative.

[kyk666123]

(1) I would like to know if the intermediate rewards emitted by emit_reward and the final reward returned by rollout are by default linked to the closest triplet with a None reward, when no link is specified. In the SQL agent example, in the check_query function, I analyze the check result according to certain rules and then add emit_reward (I only added emit_reward in the check function!). When I print the trajectory of a sample triplet, I noticed that the rewrite closest to the final_reward (which has a value of 1) also received a reward of 1, even though I didn’t add a reward to the rewrite. Why is this happening?

Did you return the reward from the rollout function? If you return it, the runner will automatically emit that reward if you haven't called emit_reward.

(2) Still in the SQL agent example: I only added emit_reward in check_query (with a value of 0.2, -0.2, or -0.1). I also modified the get_train_data_batch function in daemon.py of VERL, changing the reward_list. Originally, only the final reward was added, but I changed it to add the sum of the final reward and the intermediate rewards in the trace. However, I found that in the training/reward curve in wandb (with the value of np.mean(finished_id_to_final_reward.values())), negative numbers appear, even though I haven’t changed anything related to the final reward, which should only be 0 or 1. Therefore, negative numbers shouldn’t appear. Why is this happening?

Did you rewrite the final reward with the sum of intermediate and final reward. In that case, if your intermediate reward can be negative, I don't see why the final reward must be non-negative.

Thank you very much. In the SQL-agent example the default rollout() already returns the final reward and I didn’t change that; I only added an emit_reward inside check_query(), which is why I’m puzzled that the rewrite-query triple in the trajectory also carries a reward—normally it should be None. I’ll look into it again myself.
By the way, will you be providing any examples of how to add intermediate rewards for algorithms in the future?

[dalek-who]

There is a great need for an official and complete tutorial on intermediate rewards (especially for the Spider example), as there are currently many issues about this. #390 #392 #387 #282 #31

[dalek-who]

Based on your practice, can the reward for a langchain node (e.g., check_query) during training be summarized as follows?

1. If a "step_reward" is submitted via emit_reward in check_query, and rollout() returns None, then the check_query's reward = step_reward.
2. If a "step_reward" is submitted via emit_reward in check_query, and rollout() returns a final_reward, then the check_query’s reward = step_reward + final_reward.
3. If check_query does not call emit_reward, and rollout() returns a final_reward, then the check_query’s reward = final_reward.
4. If the trainer’s adapter excludes check_query (e.g., trainer = agl.Trainer(..., adapter={"agent_match": r'^(?!.*check_query).*$'})), then regardless of what rollout() returns, check_query will not receive any reward and will not be trained.

[kyk666123]

Based on your practice, can the reward for a langchain node (e.g., check_query) during training be summarized as follows?

1. If a "step_reward" is submitted via emit_reward in check_query, and rollout() returns None, then the check_query's reward = step_reward.
2. If a "step_reward" is submitted via emit_reward in check_query, and rollout() returns a final_reward, then the check_query’s reward = step_reward + final_reward.
3. If check_query does not call emit_reward, and rollout() returns a final_reward, then the check_query’s reward = final_reward.
4. If the trainer’s adapter excludes check_query (e.g., trainer = agl.Trainer(..., adapter={"agent_match": r'^(?!.*check_query).*$'})), then regardless of what rollout() returns, check_query will not receive any reward and will not be trained.

In the SQL agent, I haven’t tested the case where rollout returns None.
I only inserted emit_reward inside functions like write_query and check_query.
After turning the traces into triplets, I saw that the reward emitted inside a function is indeed attached to the triplet created by that exact function call (but it seems to be only the emitted reward—no final reward is added; I still need to verify this).
If emit_reward is never called, the reward field in the triplet is always None, and only the final_reward ends up being 0 or 1.
So my open question is: when emit_reward is called without supplying a link, does it automatically attach the reward to the closest upstream LLM-call span?
I originally wanted to provide an explicit link (the official docs show you can use the gen_ai.response_id of the LLM call when using the OpenAI SDK), but LangChain’s LLM invocations don’t expose any response_id or similar attribute, so I haven’t found a reliable way to specify the link argument when I call emit_reward.

[ultmaster]

There is a great need for an official and complete tutorial on intermediate rewards (especially for the Spider example), as there are currently many issues about this. #390 #392 #387 #282 #31

I think #367 is on this, but it's very complicated. They even published a paper for this.

[ultmaster]

If the trainer’s adapter excludes check_query (e.g., trainer = agl.Trainer(..., adapter={"agent_match": r'^(?!.check_query).$'})), then regardless of what rollout() returns, check_query will not receive any reward and will not be trained.

I think that's not correct. emit_reward() call does not automatically associate to a specific LLM call. When the check query call is filtered, the reward will just attach to the closest previous query, which might be a write_query call or rewrite_query call. There is a span link feature trying to address this, but it's only on the latest main branch.

So my open question is: when emit_reward is called without supplying a link, does it automatically attach the reward to the closest upstream LLM-call span?

It depends on the adapter implementation. By default, it selects a policy called RewardMatchPolicy.FIRST_OCCURRENCE, which does exactly what you said.

I originally wanted to provide an explicit link (the official docs show you can use the gen_ai.response_id of the LLM call when using the OpenAI SDK), but LangChain’s LLM invocations don’t expose any response_id or similar attribute, so I haven’t found a reliable way to specify the link argument when I call emit_reward.

In that case, you will need to manually annotate a code block as an operation and link the reward to that operation. Read more here. I think #367 has also implemented an example for how to do this.