Add DTW, nDTW, and SDTW trajectory metrics#12
Open
nalinraut wants to merge 2 commits into
Open
Conversation
AmeyaWagh
reviewed
Feb 22, 2026
| from torchmetrics import Metric | ||
|
|
||
|
|
||
| def _compute_dtw(predicted: Tensor, reference: Tensor) -> Tensor: |
Owner
There was a problem hiding this comment.
Does this need to be an independenct function? can this be part of the metric class?
Author
There was a problem hiding this comment.
All three classes (DTWDistance, NormalizedDTW, SuccessWeightedDTW) use it. Making them part of one would require other two to call DTWDistance._compute_dtw(...) without really depending on the class. Including it in all three would violate DRY. I can make a base class with this method as static or leave it as is in the module.
AmeyaWagh
reviewed
Feb 22, 2026
Add Dynamic Time Warping based metrics for evaluating trajectories that may have different lengths or temporal alignment. These metrics are particularly useful for evaluating VLA models and policies using action chunking (e.g., ACT, Diffusion Policy). New metrics: - DTWDistance: Raw DTW distance using dynamic programming (lower=better) - NormalizedDTW: Mapped to [0,1] using exp(-DTW/(|R|*d)) (higher=better) - SuccessWeightedDTW: nDTW weighted by task success (SDTW = nDTW * Success) Key features: - Support for trajectories of different lengths (core advantage over MSE/ATE) - Tolerates temporal misalignment (hesitation, speed differences) - Optional custom normalization factor - Full torchmetrics.Metric compatibility with distributed training support - Comprehensive test suite and example usage Reference: Ilharco et al., "General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping," arXiv:1907.05446, NeurIPS ViGIL Workshop, 2019.
nalinraut
force-pushed
the
feature/dtw-metrics
branch
from
cc8933e to
88aa716
Compare
Welcome to Codecov 🎉Once you merge this PR into your default branch, you're all set! Codecov will compare coverage reports and display results in all future pull requests. Thanks for integrating Codecov - We've got you covered ☂️ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add Dynamic Time Warping based metrics for evaluating trajectories that may have different lengths or temporal alignment. These metrics are particularly useful for evaluating VLA models and policies using action chunking (e.g., ACT, Diffusion Policy).
New metrics:
Key features:
Reference: Ilharco et al., "General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping," arXiv:1907.05446, NeurIPS ViGIL Workshop, 2019.