Skip to content

[TTS] Update multi-language eval report format #15915

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
rlangman wants to merge 1 commit into
base: main
Choose a base branch
from
Open

[TTS] Update multi-language eval report format #15915

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions examples/tts/magpietts_inference.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -216,7 +216,9 @@ def run_inference_and_evaluation(
violin_plot_metrics.remove('utmosv2')

# Build full checkpoint identifier (include MoE info if present)
full_checkpoint_name = f"{checkpoint_name}_{moe_info}{inference_config.build_identifier()}_SV_{eval_config.sv_model}_{eval_config.language}"
full_checkpoint_name = (
f"{checkpoint_name}_{moe_info}{inference_config.build_identifier()}_SV_{eval_config.sv_model}"
)
Comment on lines +219 to +221

@coderabbitai coderabbitai Bot Jul 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Preserve language in aggregate CSV identity.

full_checkpoint_name no longer contains the dataset language, but append_metrics_to_csv still writes it at Lines 349 and 376 while the CSV schema has no language column. Multilingual runs can therefore produce indistinguishable aggregate rows for the same checkpoint and dataset. Add a language column or pass a language-qualified checkpoint identifier to the aggregate CSV writer.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/tts/magpietts_inference.py` around lines 219 - 221, Update the
checkpoint identity flow around full_checkpoint_name and append_metrics_to_csv
so aggregate CSV rows preserve dataset language. Either add a language column to
the aggregate CSV schema and populate it at both writer call sites, or include
the language in the checkpoint identifier passed to the writer; ensure
multilingual runs remain distinguishable for the same checkpoint and dataset.


# Tracking metrics across datasets
ssim_per_dataset = []
Expand Down Expand Up @@ -261,7 +263,7 @@ def run_inference_and_evaluation(
}

# Setup output directories
eval_dir = os.path.join(out_dir, f"{full_checkpoint_name}_{dataset}")
eval_dir = os.path.join(out_dir, f"{full_checkpoint_name}_{language}_{dataset}")
audio_dir = os.path.join(eval_dir, "audio")
os.makedirs(eval_dir, exist_ok=True)

Expand Down Expand Up @@ -341,7 +343,7 @@ def run_inference_and_evaluation(

sorted_filewise = sorted(filewise_metrics, key=lambda x: x.get('cer', 0), reverse=True)
with open(os.path.join(eval_dir, f"{dataset}_filewise_metrics_{repeat_idx}.json"), "w") as f:
json.dump(sorted_filewise, f, indent=4)
json.dump(sorted_filewise, f, indent=4, ensure_ascii=False)

@coderabbitai coderabbitai Bot Jul 16, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | ⚡ Quick win

Open the JSON file with an explicit UTF-8 encoding.

ensure_ascii=False writes Unicode characters directly, but the surrounding open(..., "w") uses the platform locale. On non-UTF-8 environments this can raise UnicodeEncodeError or corrupt the intended raw text.

Proposed fix 
-            with open(os.path.join(eval_dir, f"{dataset}_filewise_metrics_{repeat_idx}.json"), "w") as f:
+            with open(
+                os.path.join(eval_dir, f"{dataset}_filewise_metrics_{repeat_idx}.json"),
+                "w",
+                encoding="utf-8",
+            ) as f:
                 json.dump(sorted_filewise, f, indent=4, ensure_ascii=False)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
json.dump(sorted_filewise, f, indent=4, ensure_ascii=False)
with open(
os.path.join(eval_dir, f"{dataset}_filewise_metrics_{repeat_idx}.json"),
"w",
encoding="utf-8",
) as f:
json.dump(sorted_filewise, f, indent=4, ensure_ascii=False)
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@examples/tts/magpietts_inference.py` at line 346, Update the JSON output file
opening in the code surrounding json.dump to explicitly use UTF-8 encoding,
while preserving ensure_ascii=False and the existing serialization behavior.


# Append to per-run CSV
append_metrics_to_csv(per_run_csv, full_checkpoint_name, dataset, metrics)
Expand Down
4 changes: 2 additions & 2 deletions scripts/tts_comparison_report/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,8 +187,8 @@ The default expiration time is one year.
- The expiration time is also included as a suffix in the uploaded artifacts
directory name, using the format `%Y-%m-%dT%H-%M-%SZ`, so uploaded reports
can be filtered and deleted later if needed.
- Both generated reports include a clickable Jira link derived from `--task_id`.
If no task ID is specified, the link points to the Jira project page.
- Both generated reports include a clickable POR link derived from `--task_id`.
If no task ID is specified, the link points to the POR project page.

@artem-gorodetskii artem-gorodetskii Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggestion: remove this sentence, since every comparison report should now be linked to a POR ticket, so a missing task ID is no longer a valid case


## Maintenance

Expand Down
35 changes: 25 additions & 10 deletions scripts/tts_comparison_report/reporting/constants.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,30 @@
_ROOT: Path = Path(__file__).parent.parent

# Benchmark names supported by the comparison report pipeline.
SUPPORTED_BENCHMARK_NAMES: list[str] = [
"libritts_seen",
"libritts_test_clean",
"riva_hard_digits",
"riva_hard_letters",
"riva_hard_money",
"riva_hard_short",
"vctk",
]
BENCHMARK_META = {
'libritts': 'en',
'riva_en': 'en',
'riva_en_hard_sentences': 'en',
'riva_en_short_sentences': 'en',
'riva_en_qa': 'en',
'riva_en_qa_longform': 'en',
'King_ASR_sa_diacritics': 'ar',
'King_ASR_sa_no_diacritics': 'ar',
'King_ASR_uae_diacritics': 'ar',
'King_ASR_uae_no_diacritics': 'ar',
'cmltts_de': 'de',
'cmltts_es': 'es',
'cmltts_fr': 'fr',
'AI4bharat': 'hi',
'cmltts_it': 'it',
'jvs_jsut': 'ja',
'F5I9N7A1': 'ko',
'cmltts_pt': 'pt',
'vivos': 'vi',
'mscenespeech': 'zh',
}

SUPPORTED_BENCHMARK_NAMES = BENCHMARK_META.keys()

@artem-gorodetskii artem-gorodetskii Jul 16, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The type was list[str], but it became KeysView; it should be converted to a list


# Default width of tqdm progress bars in terminal columns.
TQDM_NCOLS: int = 80
Expand All @@ -52,4 +67,4 @@
DUMMY_TASK_ID: str = "NEMOTTS-0000"

@artem-gorodetskii artem-gorodetskii Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we decided to move from Jira to POR, DUMMY_TASK_ID is no longer valid. Linking a report to Jira used to be optional, so the task ID was optional as well. However, all development work must now be linked to a POR ticket, which means every comparison report should also be linked to a POR ticket. This suggests that we no longer need DUMMY_TASK_ID, and that --task_id should become a required argument in the report-generation CLI. This will also require updates to dependent functionality, such as the TaskInfo structure.


# URL prefix used to construct clickable Jira ticket links in reports.
JIRA_TICKET_URL_PREFIX: str = "https://jirasw.nvidia.com/browse"
TICKET_URL_PREFIX: str = "https://modelpor.ideas.aha.io/ideas"
4 changes: 2 additions & 2 deletions scripts/tts_comparison_report/reporting/helpers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
from datetime import UTC, datetime, timedelta
from pathlib import Path

from scripts.tts_comparison_report.reporting.constants import DUMMY_TASK_ID, JIRA_TICKET_URL_PREFIX
from scripts.tts_comparison_report.reporting.constants import DUMMY_TASK_ID, TICKET_URL_PREFIX
from scripts.tts_comparison_report.reporting.models import ExpirationInfo, TaskInfo


Expand Down Expand Up @@ -46,7 +46,7 @@ def make_task_info(task_id: str) -> TaskInfo:
Task information with the original task ID, derived Jira ID, and Jira URL.
"""
jira_id = task_id if task_id != DUMMY_TASK_ID else task_id.split("-")[0]
jira_url = f"{JIRA_TICKET_URL_PREFIX}/{jira_id}"
jira_url = f"{TICKET_URL_PREFIX}/{jira_id}"

return TaskInfo(
task_id=task_id,
Expand Down
9 changes: 7 additions & 2 deletions scripts/tts_comparison_report/reporting/orchestrator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@
prepare_eval_artifacts,
)
from scripts.tts_comparison_report.reporting.constants import (
BENCHMARK_META,
S3_AUDIO_DIR,
S3_IMAGES_DIR,
S3_LINK_EXPIRES_IN,
Expand Down Expand Up @@ -268,7 +269,9 @@ def _render_audio_report(
pair_blocks=pair_blocks,
)
benchmark_blocks.append(block)
benchmark_section_info.append((benchmark_name, benchmark_name))
benchmark_language = BENCHMARK_META[benchmark_name]
name_info = f"{benchmark_name} ({benchmark_language})"
benchmark_section_info.append((benchmark_name, name_info))

report = self.renderer.render(
name=TemplateName.audio_report,
Expand Down Expand Up @@ -370,7 +373,9 @@ def _render_eval_report(
image_block=image_block,
)
benchmark_blocks.append(block)
benchmark_section_info.append((benchmark_name, benchmark_name))
benchmark_language = BENCHMARK_META[benchmark_name]
name_info = f"{benchmark_name} ({benchmark_language})"
benchmark_section_info.append((benchmark_name, name_info))

report = self.renderer.render(
name=TemplateName.eval_report,
Expand Down
2 changes: 1 addition & 1 deletion scripts/tts_comparison_report/templates/audio_report.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -281,7 +281,7 @@
<h2>Links ↗</h2>
<ul>
<li class="link-comment">
<a href="{{ jira_url }}" target="_blank" rel="noopener noreferrer">Jira {{ jira_id }}</a>
<a href="{{ jira_url }}" target="_blank" rel="noopener noreferrer">POR {{ jira_id }}</a>

@artem-gorodetskii artem-gorodetskii Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please rename jira_url to task_url and jira_id to task_id for this template to match the updated terminology

</li>
</ul>
</aside>
Expand Down
2 changes: 1 addition & 1 deletion scripts/tts_comparison_report/templates/eval_report.jinja
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,7 +384,7 @@
<h2>Links ↗</h2>
<ul>
<li class="link-comment">
<a href="{{ jira_url }}" target="_blank" rel="noopener noreferrer">Jira {{ jira_id }}</a>
<a href="{{ jira_url }}" target="_blank" rel="noopener noreferrer">POR {{ jira_id }}</a>

@artem-gorodetskii artem-gorodetskii Jul 16, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same: please rename jira_url to task_url and jira_id to task_id for this template to match the updated terminology

</li>
{% if audio_report_url %}
<li class="link-comment">
Expand Down
Loading