Skip to content

feat: EvaluatorRuleArgs允许自由拓展 #431

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
shijinpjlab merged 2 commits into from
Jun 12, 2026
+12 −7
Merged

feat: EvaluatorRuleArgs允许自由拓展 #431

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
a7cc6af
feat: EvaluatorRuleArgs允许自由拓展
shijinpjlab Jun 12, 2026
f86c7e1
feat: md更新
shijinpjlab Jun 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 1 addition & 2 deletions dingo/config/input_args.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -114,13 +114,12 @@ class ExecutorArgs(BaseModel):


class EvaluatorRuleArgs(BaseModel):
model_config = {"extra": "forbid"}
model_config = {"extra": "allow"}

threshold: Optional[float] = None
pattern: Optional[str] = None
key_list: Optional[List[str]] = None
refer_path: Optional[List[str]] = None
parameters: Optional[dict] = None


class EmbeddingConfigArgs(BaseModel):
Expand Down
5 changes: 2 additions & 3 deletions dingo/model/rule/rule_common.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2691,7 +2691,7 @@ class RuleDictConsistency(BaseRule):
}

_required_fields = [RequiredField.METADATA, RequiredField.CONTEXT]
dynamic_config = EvaluatorRuleArgs(parameters={"ignore_order": True})
dynamic_config = EvaluatorRuleArgs(ignore_order=True)

@classmethod
def _normalize_value(cls, value, ignore_order: bool):
Expand Down Expand Up @@ -2719,8 +2719,7 @@ def eval(cls, input_data: Data) -> EvalDetail:
res = EvalDetail(metric=cls.__name__)
left_dict = getattr(input_data, "metadata", None)
right_dict = getattr(input_data, "context", None)
parameters = cls.dynamic_config.parameters or {}
ignore_order = parameters.get("ignore_order", True)
ignore_order = getattr(cls.dynamic_config, "ignore_order", True)

@gemini-code-assist gemini-code-assist Bot Jun 12, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

When retrieving the dynamic configuration ignore_order using getattr, if the attribute is explicitly configured as None (e.g., "ignore_order": null in JSON), getattr will return None instead of the default value True. Since _normalize_value expects a boolean, this can lead to unexpected behavior where ignore_order is treated as False (falsy) but passed recursively as None.

To ensure robust defensive programming and guarantee that ignore_order is always a proper boolean (defaulting to True unless explicitly set to False), we can use is not False comparison.

Suggested change
ignore_order = getattr(cls.dynamic_config, "ignore_order", True)
ignore_order = getattr(cls.dynamic_config, "ignore_order", True) is not False


if not isinstance(left_dict, dict) or not isinstance(right_dict, dict):
res.status = True
Expand Down
3 changes: 2 additions & 1 deletion docs/config.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -110,14 +110,15 @@ HuggingFace 特定配置：

#### EvaluatorRuleArgs 配置 (evaluator.rule_config.[rule_name])

规则配置：
规则配置（支持额外字段，规则可按需读取自定义参数）：

| Parameter | Type | Default | Required | Description |
|-----------|------|---------|----------|-------------|
| threshold | float | null | No | 规则决策阈值 |
| pattern | str | null | No | 匹配模式字符串 |
| key_list | list | null | No | 匹配关键词列表 |
| refer_path | list | null | No | 参考文件路径或小模型路径 |
| *其他字段* | any | - | No | 规则自定义扩展字段（如 `ignore_order`） |

#### EvaluatorLLMArgs 配置 (evaluator.llm_config.[llm_name])

Expand Down
8 changes: 7 additions & 1 deletion docs/technical/technical_all.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -325,7 +325,7 @@ class RuleColonEnd(BaseRule):

+ metric_type: 函数 rule_register 执行时赋值
+ group: 函数 rule_register 执行时赋值
+ dynamic_config: 开放的自定义接口
+ dynamic_config: 开放的自定义接口（支持 `EvaluatorRuleArgs` 额外字段扩展，不需要 `parameters` 包裹）

其次，所有的规则都需要执行注册操作，即 Model.rule_register 函数，并指明 metric_type 与 group。

Expand All @@ -342,6 +342,12 @@ class RuleColonEnd(BaseRule):
dynamic_config = EvaluatorRuleArgs()
```

例如需要自定义参数时，可直接写为：

```python
dynamic_config = EvaluatorRuleArgs(ignore_order=True)
```

最后，实现 eval 类函数，需要注意接收变量与返回值的类型

```python
Expand Down
Loading