Seeking clarification: significantly lower ToolBench workflow scores on Colab vs reported ACBench results

[jakobahlberg]

Hi ACBench authors,
we are two students from the University of Copenhagen working on a bachelor project. We’re trying to reproduce your ToolBench workflow results, but our scores are much lower than the numbers reported in your paper: "Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression".

We’d really appreciate guidance on what we may be doing wrong.

What we ran:

• Environment: Google Colab

We used Qwen2.5-1.5B-Instruct - where you used Qwen2.5-1.5B, but we would expect instruct to be even better at the task.

NOTE: We had a quite a few dependency issues, which is why we installed and uninstalled packages.

`!git clone https://github.com/pprp/ACBench.git
%cd /content/ACBench
!ls

!pip install -U pip
!pip install -r requirements.txt
%cd /content/ACBench
!pip install -e .
!pip -q uninstall -y tensorflow google-ai-generativelanguage grpcio-status || true

%cd /content

!git clone https://github.com/zjunlp/WorfBench.git

!mkdir -p /content/ACBench/data

!cp -r /content/WorfBench/gold_traj /content/ACBench/data/gold_traj

!find /content/ACBench/data/gold_traj -maxdepth 2 -type f -name "graph_eval.json" | head -n 20

!pip uninstall -y flashinfer flashinfer-python nvidia-cutlass-dsl cutlass || true

%env VLLM_ATTENTION_BACKEND=TORCH_SDPA
%env VLLM_USE_V1=0

!python -m acbench.node_eval
--task gen_workflow
--task_type toolbench
--model_name Qwen/Qwen2.5-1.5B-Instruct
--gold_path /content/ACBench/data/gold_traj/toolbench/graph_eval.json
--pred_path /content/ACBench/outputs/qwen15_fp16_toolbench.json
--few_shot
--temperature 0
--top_p 1
--max_tokens 1024
--dtype float16
--tensor_parallel_size 1`

Output we get:
We see repeated parser warnings: "edge_workflow is empty"

And final metrics:
Average Precision: 0.2861009847851952
Average Recall: 0.3667502088554719
Average F1_score: 0.3166924206397889

From your reported results (table. 10), we expected something closer to:
Precision ≈ 0.77
Recall ≈ 0.74
F1 ≈ 0.73

Thanks a lot for your help, we’d really appreciate any clarification.

All the best,
Lasse and Jakob

[pprp]

Thanks for your detailed setup. The main issues are likely:
Use the exact same model - we used Qwen/Qwen2.5-1.5B (base), not the instruct variant. The instruct version uses a chat template and produces different output format, which explains the low scores.
"edge_workflow is empty" - this indicates the model output isn't being parsed into valid tool calls. The base model requires the non-chat prompt format. Make sure you're using our original prompt templates (in acbench/prompts/).
Quick fix:

• Switch to --model_name Qwen/Qwen2.5-1.5B
• Do NOT modify prompt files
• Use a clean environment (no uninstall steps)
  If you still get ~0.3 F1, please share:
• Exact model HF name
• First few entries of your pred_path JSON
• Any additional logs

Feel free to follow up

[jakobahlberg]

Hi again,

Thank you for getting back to us earlier.

We tried running ACBench on Google Colab, which provides a Tesla T4 GPU (compute capability 7.5). When running the demo setup with vLLM, the workflow generation initially crashed during model initialization, with errors originating from the FlashInfer attention backend.

Initial Attempt

We first tried to resolve this by setting the following environment variables before running the benchmark:
import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0"
os.environ["VLLM_ATTENTION_BACKEND"] = "XFORMERS"
os.environ["VLLM_USE_V1"] = "0"

However, this still resulted in errors during initialization.

We were only able to get the benchmark to start running after uninstalling FlashInfer:

pip uninstall -y flashinfer flashinfer-python

After doing this, the model successfully loaded and the evaluation began.

We ran the benchmark as close to the demo setup as possible:

!python -m acbench.node_eval
--task gen_workflow
--task_type toolbench
--model_name Qwen/Qwen2.5-1.5B
--gold_path /content/ACBench/data/gold_traj/toolbench/graph_eval.json
--pred_path /content/ACBench/outputs/qwen15_fp16_toolbench.json
--few_shot
--temperature 2
--quantization 3
--tensor_parallel_size 1

The benchmark begins running after removing FlashInfer, but the generated outputs appear very noisy. We suspect this may primarily be due to the temperature value of 2, which is what is used in the demo.

For testing purposes we terminated the run early because the intermediate outputs appeared incorrect.

Below you can see an image of the output we get - before I terminated it.

Maybe you can help us clarify what we should try next - to get closer to your results?

Thank you again!