UI should pop up a warning when merge fails due to offset too large

[cheneyuwu]

When user confirms a merge point by clicking on the yellow triangle, internally we check if the distance to the merging vertex is too large to merge. However, if a merge attempt is rejected due to offset too large, we silently drop back to branch state without popping up a warning saying merge attempt rejected in the UI, so the user wouldn't know why merge is not successful.