Skip to content

[FEAT] Add MMAR/ MMAU-PRO for reasoning task support#16

Open
nhhoang96 wants to merge 13 commits intomainfrom
feat/add_mmar
Open

[FEAT] Add MMAR/ MMAU-PRO for reasoning task support#16
nhhoang96 wants to merge 13 commits intomainfrom
feat/add_mmar

Conversation

@nhhoang96
Copy link
Copy Markdown
Collaborator

📌 Description

  • Adding support for deep reasoning tasks, including MMAR and MMAU-PRO.
  • Reusing llm-judge-binary for the MCQ evaluation for these two datasets.

🔗 Related Issue(s)

🛠️ Type of Change

  • Bug fix (non-breaking change that fixes an issue)
  • New feature (non-breaking change that adds functionality including new tasks)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation update
  • Refactor / Code cleanup
  • Maintenance / Chore / Task
  • Other (please describe):

✅ How Has This Been Tested?

  • Unit tests
  • Integration tests
  • Manual testing

Test Results / Screenshots (if applicable):

Sample run_config:

task_metric: 
  - ['mmar', 'llm_judge_binary']
  - ['mmau-pro', 'llm_judge_binary']
aggregate:
  - ['llm_judge_binary', ['mmau-pro']]
  - ['llm_judge_binary', ['mmar']]

Sample Experimental Log:
2025-09-20_23-40-59_544502_default.log

📸 Screenshots / Demos

N/A

📋 Checklist

  • Code follows project style guidelines
  • Tests have been added/updated (if applicable)
  • Documentation has been updated (if applicable)
  • Linked relevant issue(s)
  • Self-reviewed my code

🙌 Additional Notes

N/A

@nhhoang96 nhhoang96 self-assigned this Sep 21, 2025
@nhhoang96 nhhoang96 added documentation Improvements or additions to documentation enhancement New feature or request labels Sep 21, 2025
@nhhoang96 nhhoang96 requested review from jonggunp and khyatimahajan and removed request for jash-mehta-3300 April 19, 2026 18:18
Comment thread requirements.txt Outdated
Comment thread utils/util.py Outdated
Comment thread utils/util.py Outdated
Comment thread utils/util.py
Comment thread utils/data_utils.py Outdated
Comment thread preprocessors/mmar_preprocessor.py Outdated
Comment thread preprocessors/mmar_preprocessor.py Outdated
choices_column_name = task_config.get('choice_column', None)
category_column_name = task_config.get('category_column', '')
sample_instruction_column_name = task_config.get('instruction_column', None)
user_query_column_name = task_config.get('textual_input_column', None)
Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This doesn't seem to be used anywhere.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

user_query_column_name is not used anywhere, so I removed it. The other variables are used in the current preprocessing logic. Thanks!

Comment thread preprocessors/mmar_preprocessor.py Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants