[LFX Term 2 2026] Proposal for Example Restoration and CI Example Validation Framework

[ken6078]

What type of PR is this?

/kind bug
/kind documentation
/kind design

What this PR does / why we need it:

This PR adds an Example Restoration CI pipeline that automatically scans and validates all examples, classifies detected issues, and publishes health reports for maintainers. It also includes targeted restoration work for examples/llm_simple_qa, including portable paths, documented dependencies, dataset/JSONL validation, model configuration, device fallback, and metric robustness.

Many examples may become invalid over time due to code, dependency, or documentation changes. Without automated validation, these issues are difficult to track at scale. The proposed CI provides continuous visibility into example status, helps maintain a reliable example ecosystem, and uses llm_simple_qa as the first restored reference example for validating future LLM-related examples.

Which issue(s) this PR fixes:
Fixes #230

[gemini-code-assist]

Code Review

This pull request introduces a comprehensive proposal for the KubeEdge Ianvs Example Classification CI Validation Framework, which aims to automate example classification, validation, and the restoration of the llm_simple_qa example. The review feedback correctly identifies minor path and spelling inconsistencies in the proposal metadata and documentation, specifically recommending the correction of examples/llm_qa_example/requestment.txt to examples/llm_simple_qa/requirements.txt.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[MooreZheng]

Good to see the new proposal. As discussion,

1. User flow can be added, e.g., the use of github action and robot
2. Architecture design, e.g., modified and submitted modules
3. Directory organization and design, e.g., proposals and CI
4. Demo

[MooreZheng]

Good to see the updates, especally the demo, directories and user flow

1. More thoughts on User flow

• Developer use local CI, Github CI -> robot/CI check PR -> CI for docs, CI for example, CI for core code, take a look at doc guidance to learn how to check results and modification. Need to check out whether all user cases are covered
• Maintainer use Github CI -> integration test. But how to triger CI and where to check the report?
• Priority: Global CI > local CI > Robot.

2. Some Methods

• As for CI for docs, take a look at baoyue's work #105
• Add Document guidance. Might consider adding a statement in KubeEdge robot in the future
• For integration CI test, might use unit test for developer small scale dataset.

3. Architecture design

• Show relationship between new modules and existing modules
• New module: CI in .github/workflow, take "workflow" as a module. might rename as ci-workflow
• Directory: ianvs/tool -> ianvs/resource/tool

[MooreZheng]

1. Put example status table in the example directory
2. Add A-B-C example and justify the reason why we compare BC instead AB.
3. Add definition for Tier 0, Tier 1... etc
4. T2 and T3 output is not the same,and avoid link to the same "classiy failure"
5. Merged pull request triggers T3, instead of CICD triggers T3. In this case, T2 failure report can also update T3 failure report
6. UC numbering and naming update
7. Update module name like Event Trigger

[MooreZheng]

This version is very close to the final one

1. add example status into examples/README.md
2. event trigger and example validator under CICD instead of workflow
3. put resources/tools into example validator
4. add other functions in validator into validator/tools
5. alpha-beta-gramma rename

[ken6078]

Dear @MooreZheng,

Thank you for your review. I have updated the proposal based on each of your comments, as summarized below:

1. Add example status information to examples/README.md

   Addressed. I added an example status matrix to examples/README.md, including the latest validation time and status of each example.

   I also added status_directions.md to the documentation section to explain the meaning of each status.

2. Place Event Trigger and Example Validator under CI/CD instead of Workflow

   Addressed in the updated software architecture diagram.

3. Move resources/tools into Example Validator

   Addressed. The validation-related tooling is now grouped under the Example Validator component instead of being represented as a general-purpose repository-level resources/tools module.

   The proposal now uses the following dedicated location:

   .github/tools/example_validator/
   

   This keeps the CI-specific validation implementation close to the GitHub Actions integration and avoids mixing it with Ianvs runtime resources.

4. Place the other validator functions under validator/tools

   Addressed by reorganizing the proposed validator structure. I placed the supporting modules under example_validator/core/ instead of introducing another tools/ directory, because the parent path already contains a tools directory.

   .github/tools/example_validator/
   ├── main.py
   ├── static_validator.py
   ├── dependency_validator.py
   ├── smoke_test_validator.py
   ├── core/
   │   ├── validation_branch_wrapper.py
   │   ├── inventory.py
   │   ├── regression_detector.py
   │   └── report_generator.py
   └── data/
       └── example_inventory.yaml
   

   This structure makes the ownership and responsibilities of the validation utilities clearer while keeping all related implementation within the Example Validator module.

5. Rename Alpha, Beta, and Gamma

   Addressed. I replaced the placeholder labels with names that more clearly describe the corresponding events:

   Alpha → Previous Tier 3
   Beta  → Tier 2 Merge
   Gamma → Original Scheduled Next Tier 3
   Delta → Rescheduled Next Tier 3
   

In addition, I squashed the previous commits into two focused commits to make the revision history easier to review:

1. Reorganize the proposed file structure.
2. Add and update the proposal content.

The detailed commit history and corresponding changes can be found here:

https://github.com/ken6078/ianvs/commits/temp/week5-changes/

Thank you again for the detailed feedback.

[MooreZheng]

Most of the changes look fine to me. Just a little bit weird to see the directory

└── .github/
 ├── tools/
 │   └── example_validator/
 │       ├── main.py
 │       ├── static_validator.py
 │       ├── dependency_validator.py
 │       ├── smoke_test_validator.py
 │       ├── core/
 │       │   ├── validation_branch_wrapper.py
 │       │   ├── inventory.py
 │       │   ├── regression_detector.py
 │       │   └── report_generator.py
 │       └── data/
 │           └── example_inventory.yaml
 │
 └── workflows/
     └── example_validator.yml

1. The directory needs to be consistent with the software architecture diagram
   For example, the tools directory should be put into validators, and we would have a CICD directory on top of validators instead. We can also get rid of the workflow files now. Then it would be better if changed as

└── .github/
   ├── CICD/
   │   └── event_trigger/
   │       ├── example_validation_trigger.yml #need to check with the difference against example_validator.yml
   │       ├── fossa and other trigger files need to be added?
   │   └── general_validator/
   │       ├── fossa and other validator files need to be added?
   │   └── example_validator/
   │       ├── main.py #what is it for? 
   │       ├── example_validator.yml #remove the workflow folder so that put it here. need to check whether proper
   │       ├── static_validator.py
   │       ├── dependency_validator.py
   │       ├── smoke_test_validator.py
   │       ├── tools/
   │       │   ├── validation_branch_wrapper.py
   │       │   ├── inventory.py
   │       │   ├── regression_detector.py
   │       │   └── report_generator.py
   │       └── data/
   │           └── example_inventory.yaml

2. Check the new folder and files
   For example, check the files in the new event_trigger folder and general_validator folder.
   Besides, what is the main.py for? It does not look proper for a formal submission. Could we change it to another name, and maybe a readme is needed for the example_validator folder?

[ken6078]

Dear @MooreZheng,

The previous main.py is the entry point of the example validation process. In an earlier version, this file was named validate_examples.py. To make its responsibility clearer and distinguish it from the individual validator modules, it has now been renamed to validation_runner.py.

And, I have updated the directory structure and naming to align with the revised software architecture.

I have added the reason that explains why using .github/workflows folder, but not .github/CICD

Although the software architecture presents GitHub Actions as the CI/CD execution layer, the repository does not introduce a separate .github/CICD/ directory because GitHub Actions only discovers workflow definitions under .github/workflows/ (document). Each validation concern therefore remains represented by a single workflow file, such as CodeQL, FOSSA, Pylint, or example validation. The General Validators are third-party integrations, so they require only their corresponding YAML workflow files and do not introduce additional implementation files in the repository. In contrast, the Example Validator contains project-specific reusable logic, which is implemented under .github/tools/example_validator/ and invoked by the example-validation workflow.

Other changes and rationale

Change	Update	Rationale
Rename the entry point	main.py → validation_runner.py	validation_runner.py clearly indicates that the file is the execution entry point for the validation workflow, rather than a generic main program.
Rename the branch management module	validation_branch_wrapper.py → validation_branch_manager.py	The module manages the full validation-branch lifecycle, including upstream checks, fetching, merge-base discovery, temporary branch creation, rebasing, and cleanup. manager therefore describes its responsibility more accurately than wrapper.
Rename the inventory module	inventory.py → inventory_loader.py	The module primarily loads and provides example inventory metadata, so inventory_loader.py communicates its purpose more clearly.
Rename core/ to services/	.github/tools/example_validator/core/ → .github/tools/example_validator/services/	These modules provide reusable supporting services, including branch management, inventory loading, regression detection, and report generation. services/ is more specific and less likely to be confused with Ianvs core modules.

The resulting directory structure is:

.github/
└── tools/
    └── example_validator/
        ├── validation_runner.py
        ├── static_validator.py
        ├── dependency_validator.py
        ├── smoke_test_validator.py
        ├── services/
        │   ├── validation_branch_manager.py
        │   ├── inventory_loader.py
        │   ├── regression_detector.py
        │   └── report_generator.py
        └── data/
            └── example_inventory.yaml

The detailed commit history and corresponding changes can be found here:
ken6078@465b642

[hsj576]

/lgtm

[kubeedge-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hsj576, ken6078
To complete the pull request process, please assign moorezheng after the PR has been reviewed.
You can assign the PR to them by writing /assign @moorezheng in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment