Re-producing Tau-Bench Issue

[steventan0110]

I am wondering if it's possible to get eval script for communicating with the served api. I am not able to reproduces the results from the paper. I served the model (8b version) with vllm, but when I evaluate the model, I have the following issue:

I tried 2 approaches:

• In the first approach, I follow the official multi-turn script (https://github.com/SalesforceAIResearch/xLAM/blob/main/xLAM/client/xLAM.py) to generate results. However, I found that the model does not follow the format at all:
  sometimes it directly output strings
  sometimes it directly output actions.
  in very few cases it output the thought and actions in the json as required by the FORMAT_INSTRUCTION.

This result in very bad score. I feel like something is wrong with the prompt structuring, but I made sure that the prompt is created using the build_prompt function shared from the official multi-turn implementation. @jianguoz I am wondering if you have solution for this? if this is the script used for evaluating tau-bench? Can you also provide some feedback on what might be the reason for bad format following?

• In the second approach, I used my own structured messages, and I check if the tool_calls and msg_content exist and process them accordingly. some example code below:

response = self.xlam_client.chat.completions.create(
    model_name = "/opt/tiger/voice_agent/ckpt/hf_models/Llama-xLAM-2-8b-fc-r",
    messages=self.messages,
    tools=tools,
    tool_choice="auto"
)
assist_message = response.choices[0].message
msg_content = assist_message.content
msg_tool_calls = assist_message.tool_calls
if msg_content is not None:
    # reply to user
    response_content = msg_content
    self.messages.append({"role": "assistant", "content": response_content})
else:
    assert len(msg_tool_calls) > 0
    # print(msg_tool_calls)
    function = msg_tool_calls[0].function
    cur_fc = {
        "name": function.name,
        "arguments": function.arguments
    }
    function_str = json.dumps(cur_fc)
    self.messages.append({"role": "assistant", "content": function_str})
    response_content = cur_fc

This resolve the previous parsing error, however, the performance of the model after evaluating is far below the reported result (~20-30%).

@jianguoz , can you share the official eval script so that consistent re-production of the baseline is possible? Thanks!

[jianguoz]

Hi @steventan0110, thank you for your question. Could you please refer to our ReadME for detailed instructions on interacting with xLAM models? There are two recommended methods: the more efficient approach is ⚡📈 Using vLLM for Inference, or alternatively, you can 🤗 Use Transformers for Inference.

Refer: #27

[steventan0110]

Hi @jianguoz ,

Thanks for the reference, I did check the issue and README carefully and I have served the model with vLLM and can run inference without issue. My question is about

• How prompt should be constructed in multi-turn tool-use application
• How did you parse the model output when it does not follow instruction

The reference you pointed at is for single-turn, which is not very helpful for re-producing tau-bench results. I also saw #27 but since the original poster closes the issue, I cannot comment there to ask for eval script.

As I mentioned, I tried two approaches so far with no luck. The first version is to follow your official multi-turn example at https://github.com/SalesforceAIResearch/xLAM/blob/main/xLAM/client/xLAM.py . However, xlam-8b rarely follows the instruction format:

{"thought": "the thought process, or an empty string", "tool_calls": [{"name": "api_name1", "arguments": {"argument1": "value1", "argument2": "value2"}}]})

and result in lots of parsing error.

Then I tried re-using my eval code from tau-bench with xlam model but the performance is much worse than reported results, so there must be something wrong in my prompt construction or parsing. To provide you with more details, my full code is shown below:

XLAM_FORMAT_INSTRUCTION = """You have access to a set of tools. When using tools, make calls in a single JSON array: 

[{"name": "tool_call_name", "arguments": {"arg1": "value1", "arg2": "value2"}}, ... (additional parallel tool calls as needed)]

If no tool is suitable, state that explicitly. If the user's input lacks required parameters, ask for clarification. Do not interpret or respond until tool results are returned. Once they are available, process them or make additional calls if needed. For tasks that don't require tools, such as casual conversation or general advice, respond directly in plain text. The available tools are:"""


client = openai.OpenAI(
    base_url="http://localhost:8001/v1",  # Default vLLM server URL
    api_key="empty"  # Can be any string
)
system_prompt =  RETAIL_WIKI # RETAIL WIKI from Tau-Bench repo
system_prompt += "\n" + XLAM_FORMAT_INSTRUCTION + "\n\n"
messages = [
    {"role": "system", "content": system_prompt},
]

def run_eval(env, task, max_steps=30):
    user_request = env.reset(task) # reset env for current task, and obtain first user msg (e.g., request to update some order)
    msg_type = 'user'
    msg_content = user_request
    done = False
    num_steps = 0
    while not done and num_steps < max_steps:
        messages.append({"role": msg_type, "content": msg_content})
        ### Update prompt for the XLAM agent to generate new message
        agent_response = _generate_next_step(msg_type, msg_content, messages)
        
        #### ENV STEP ####
        # if agent_response is a tool call, it will parse function name and arguments to execute the tool
        # if agent_response is a message, it will be used to prompt the simulated user for response
        env_response, done, info = env.step(agent_response)

        # extract new message to append to conversation trajectory
        msg_type = info["response_type"] # msg_type is user or tool
        msg_content = env_response
        num_steps += 1
        if done:
            break
    # compute reward following tau-bench
    # ....



def _generate_next_step(msg_type, msg_content, messages):
    # tools_info is a list of tool's information following OPENAI function call format
    # this is already prepared from tau-bench
    tools = []
    for tool_info in tools_infos:
        tools.append(tool_info)

    response = client.chat.completions.create(
        model = "/{my model path}/Llama-xLAM-2-8b-fc-r",
        messages=messages,
        tools=tools,
        tool_choice="auto"
    )
    assist_message = response.choices[0].message
    msg_content = assist_message.content
    msg_tool_calls = assist_message.tool_calls

    if len(msg_tool_calls) > 0:
        # agent is calling a tool in its response
        function = msg_tool_calls[0].function
        cur_fc = {
            "name": function.name,
            "arguments": function.arguments
        }
        function_str = json.dumps(cur_fc)
        messages.append({"role": "assistant", "content": function_str})
        response_content = cur_fc
    else:
        response_content = msg_content
        messages.append({"role": "assistant", "content": response_content})
    return response_content

I can also provide trajectory after running this code but most of tasks have failed. Since the long-horizon multi-turn task is sensitive to prompt construction and response parsing, can you provide more details on how to properly run the code for tau-bench? I think the potential issues are:

• System message construction: I used the tokenizer's chat template to craft XLAM_FORMAT_INSTRUCTION but you might used some other system prompt for tau-bench.
• Response parsing: for the responded message (assist_message = response.choices[0].message), maybe function is also parsed from the content rather than tool_calls?
• Message appending: msg_role might affect prompt construction
• Other details that I might be unaware of

Thanks a lot for your time and help! Please let me know what to modify to re-produce your result for tau-bench.

[steventan0110]

note that for XLAM_FORMAT_INSTRUCTION, I also tried your ReACT-style instruction (https://github.com/SalesforceAIResearch/xLAM/blob/main/xLAM/client/config.py#L25) and the model does not follow the format well, so I think the system message needs to be carefully adjusted and the response parsing needs to be more sophisticated for the model to be usable

FORMAT_INSTRUCTION = """
    The output should be of the JSON format, which specifies a list of generated function calls. The example format is as follows, please make sure the parameter type is correct. If no function call is needed, please make 
    tool_calls an empty list "[]".
    ```
    {"thought": "the thought process, or an empty string", "tool_calls": [{"name": "api_name1", "arguments": {"argument1": "value1", "argument2": "value2"}}]}"""
    ```

[jianguoz]

Hi @steventan0110, I assume you want to evaluate Several models like xLAM, Mistral, Qwen, etc. using vLLM with multi-turn tool-use prompts, please follow these steps:

Step 1: Host your model with vLLM. vLLM already has chat templates configured (similar to tokenizer.tokenize) for each model, so it will automatically construct multi-turn prompts for you as long as you provide the messages and tools. You could check the vLLM output log to learn more about the constructed prompt format details.

Step 2: Below is a code for tau_bench/agents/tool_calling_agent.py to test hosted vLLM model directly. There’s no need to write extra, redundant code.

# Copyright Sierra

import json
from litellm import completion
from typing import List, Optional, Dict, Any
import time
import random

from tau_bench.agents.base import Agent
from tau_bench.envs.base import Env
from tau_bench.types import SolveResult, Action, RESPOND_ACTION_NAME


class ToolCallingAgent(Agent):
    def __init__(
        self,
        tools_info: List[Dict[str, Any]],
        wiki: str,
        model: str,
        provider: str,
        port_id: int,
        temperature: float = 0.0,
    ):
        self.tools_info = tools_info
        self.wiki = wiki
        self.model = model
        self.provider = provider
        self.port_id = port_id
        self.temperature = temperature

    def solve(
        self, env: Env, task_index: Optional[int] = None, max_num_steps: int = 30
    ) -> SolveResult:
        total_cost = 0.0
        env_reset_res = env.reset(task_index=task_index)
        obs = env_reset_res.observation
        info = env_reset_res.info.model_dump()
        reward = 0.0
        messages: List[Dict[str, Any]] = [
            {"role": "system", "content": self.wiki},
            {"role": "user", "content": obs},
        ]
        for _ in range(max_num_steps):
            if self.provider == "hosted_vllm":
                res = completion(
                    messages=messages,
                    model=self.model,
                    tools=self.tools_info,
                    tool_choice="auto",
                    temperature=self.temperature,
                    custom_llm_provider=self.provider,
                    api_base=f"http://localhost:{self.port_id}/v1"
                )
                total_cost += 0.0
            else:
                res = completion(
                    messages=messages,
                    model=self.model,
                    custom_llm_provider=self.provider,
                    tools=self.tools_info,
                    temperature=self.temperature,
                )
                total_cost += res._hidden_params["response_cost"]
                # Handle case where response_cost might be None
                response_cost = res._hidden_params.get("response_cost", 0.0)
                if response_cost is not None:
                    total_cost += response_cost
            
            next_message = res.choices[0].message.model_dump()
            
    
            action = message_to_action(next_message)
            env_response = env.step(action)
            reward = env_response.reward
            info = {**info, **env_response.info.model_dump()}
            if action.name != RESPOND_ACTION_NAME:
                next_message["tool_calls"] = next_message["tool_calls"][:1]
                messages.extend(
                    [
                        next_message,
                        {
                            "role": "tool",
                            "tool_call_id": next_message["tool_calls"][0]["id"],
                            "name": next_message["tool_calls"][0]["function"]["name"],
                            "content": env_response.observation,
                        },
                    ]
                )
            else:
                messages.extend(
                    [
                        next_message,
                        {"role": "user", "content": env_response.observation},
                    ]
                )
            if env_response.done:
                break
        return SolveResult(
            reward=reward,
            info=info,
            messages=messages,
            total_cost=total_cost,
        )


def message_to_action(
    message: Dict[str, Any],
) -> Action:
    if "tool_calls" in message and message["tool_calls"] is not None and len(message["tool_calls"]) > 0 and message["tool_calls"][0]["function"] is not None:
        tool_call = message["tool_calls"][0]
        return Action(
            name=tool_call["function"]["name"],
            kwargs=json.loads(tool_call["function"]["arguments"]),
        )
    else:
        return Action(name=RESPOND_ACTION_NAME, kwargs={"content": message["content"]})

Step 3: Evaluate your models using the same procedure as you would for evaluating OpenAI models.

[steventan0110]

Hi @jianguoz , thanks a lot for your detailed response and code release! I will try it out soon and let you know if I have more issues.

From the current code, I see that you use WIKI as the only system message. This will not give instruction on the model's message formatting. In tau-bench, a few different agents are tested and I mostly re-use the ReACT agent (https://github.com/sierra-research/tau-bench/blob/main/tau_bench/agents/chat_react_agent.py#L96).

Just to confirm about this, for xLAM model testing on tau-bench, did you use the default formatting instruction (which vllm will automatically add if tools info are provided, and I believe it's not including thinking block right?), or did you use something else?

[steventan0110]

Model	pass@1	pass@2	pass@3	pass@4	pass^1	pass^2	pass^3	pass^4
AIRLINE Domain
GPT4.1	48	58	70	70	48	34	26	24
xLAM-2-fc-8b	32	52	54	64	32	24	18	16
Qwen3-8b	30	46	50	62	30	20	18	18
RETAIL Domain
GPT4.1	58.3	65.2	68.7	73	58.3	53	49.6	46.1
xLAM-2-fc-8b	41.7	53.9	60	60.9	41.7	30.4	25.2	22.6
Qwen3-8b	40	57.4	67.8	71.3	40	27.8	22.6	18.3

btw this is my new benchmark result with your provided code. It indeed resolve my previous parsing issue and the trajectory looks pretty okay to me. On AIRLINE domain, it's close to what's reported in the paper. On RETAIL, it seems very different compared to the reported number (pass@1 41 v.s. 57). Do you have suggestions for further check?
Generated trajectories are publicly available here: https://drive.google.com/file/d/1YDHjxhCrhDTo1orCY16_1yXwxc94rmvZ/view?usp=sharing