Showing aumovio changes then close - #17
Open
hajnalmt wants to merge 13 commits into
Open
Conversation
This adds AGENTS.md to provide AI coding agents with essential information about the Volcano codebase including build commands, testing, code style guidelines, project conventions, and comprehensive agent-scheduler documentation covering its architecture, plugin/action interfaces, sharding model, and the agent node daemon. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Add AGENTS.md with comprehensive developer guide for AI coding agents working on the Volcano codebase, covering build commands, test commands, code style, import conventions, and agent/batch scheduler architecture. Update .gitignore with local development tool paths. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Add design document covering motivation, architecture, detailed design, tool comparison (Tilt vs Skaffold, DevSpace, Telepresence), and agentic workflow safety considerations for the Kind + ctlptl + Tilt local development workflow. Update prepare-for-development.md and development.md with Tilt-based quick start and full workflow reference. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Add Tilt-based local development environment using Kind (via ctlptl) with a local container registry. Provides single-command setup (make dev-up) with live-reload for scheduler, controller-manager, and webhook-manager components. Key features: - ctlptl-managed Kind cluster with local Docker registry - Tilt live-update with in-container Go rebuilds via entrypoint.sh - Multi-architecture support (amd64/arm64) for tool downloads, Docker builds, and in-container rebuilds - Integrated monitoring (Prometheus, Grafana) with port-forwards - On-demand unit and e2e test triggers via Tilt resources - All dev tooling self-contained in _output/bin Includes Helm chart fix: use dedicated admission_init_image_name variable for admission init Job image. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Avoid misleading cache logs when schedulerPodName is empty in single scheduler setups. In that mode the empty scheduler name made normal cache checks look suspicious and also made the consistent hash ownership check unnecessary. Skip the hash-circle check when schedulerPodName is empty so cache responsibility checks stay quiet and easier to interpret. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Keep the Kubernetes scheduler predicate snapshot consistent when reserve plugins fail during rollback. The old path returned before AddPodInfo, which could leave the snapshot missing pods after statement discard or recovery flows. Skip follow-up device allocation when reserve fails, but still update the snapshot so later predicate checks see the same node state as Volcano. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Add statement support for group eviction, last-operation tracking, discard reasons, pipeline eviction state, and merging node-local statements into a caller statement. This lets scheduler actions isolate speculative evictions and pipelines per node, then transfer operation ownership only after a scheduling attempt succeeds. The merge helper is required by the reclaim flow to avoid discard/recover cycles that can corrupt predicate snapshot state. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Introduce Aumovio-specific victim ordering helpers so preempt and reclaim can order victims by queue/job semantics before falling back to task ordering. This prevents creation timestamp or incidental queue traversal order from dominating same-priority victim selection, and adds coverage for preemptor-found and preemptor-missing ordering paths. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Add parent-aware reclaim behavior for hierarchical queues so sibling leaf queues do not reclaim cyclically when parentBasedReclaimEnabled is set and there is no relevant parent-level deserved signal. Keep cross-parent reclaim semantics intact, document the parent-based scenarios, and include scalar-resource handling needed by hierarchical capacity checks. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Switch normal preemption to the Aumovio victim priority queue so preempt uses the same queue-aware victim ordering as reclaim. The generic QueueOrderFn and preemptorTasks regression coverage already exists in the release-1.14 base, so this commit only carries the Aumovio-specific victim selection delta. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
Apply queue-aware victim selection across reclaim and track reclaim-earned credits so jobs can pipeline only when reclaimed or idle resources make the chosen node valid. Evaluate victims across candidate nodes, isolate failed node attempts, share committed reclaim credits across jobs, and support direct pipeline paths for cases like grouped evictions and MPI launchers without committing spurious victims. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
I will need to fix the capacity plugin tests on the long run, because they are failing now. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 2ac64a4609
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Comment on lines
+784
to
+786
| lv := l.(*api.TaskInfo) | ||
| rv := r.(*api.TaskInfo) | ||
| pv := preemptor.(*api.TaskInfo) |
There was a problem hiding this comment.
When hierarchical capacity is enabled, this callback is still reached through the dry-run preemption path: SelectVictimsOnNode calls BuildVictimsPriorityQueue, and that helper invokes VictimQueueOrderFn with *api.QueueInfo arguments when victims are from different queues. Casting those arguments to *api.TaskInfo here will panic during normal preemption whenever capacity's victim queue order function is registered and cross-queue victims are compared; either keep this comparator accepting queues or update the remaining BuildVictimsPriorityQueue call path as well.
Useful? React with 👍 / 👎.
| var idleCandidateNodes []*api.NodeInfo | ||
| for _, nodes := range predicateNodesByShard { | ||
| for _, node := range nodes { | ||
| if !task.InitResreq.LessEqual(node.Idle, api.Zero) { |
There was a problem hiding this comment.
This idle-resource shortcut checks node.Idle, but pipelined tasks are recorded in NodeInfo.Pipelined and are not subtracted from Idle; only FutureIdle() subtracts them. If the reclaim action pipelines one pending task using idle capacity, a later task in the same job can still see the original idle resources here and be pipelined to the same node, over-reserving resources in configurations where predicates do not reject it. Use future idle resources for this fit check so prior pipelines in the current statement are honored.
Useful? React with 👍 / 👎.
Move eviction credits from reclaim's action-local state into framework.Session so all actions in a scheduling cycle share them. Add idle and credit fast paths to preempt's normalPreempt. Session changes: - Add EvictionCreditsByNode, TotalEvictionCredits, CreditedEvictions to Session; initialize in openSession. - Add eviction_credits.go with shared helpers (AccumulateCreditsFromVictim, MergeCredits, ConsumeCredits, HasSufficientCredits, CreditNodeNames, CloneCreditsByNode, CloneCreditedSet). Reclaim changes: - Replace the three action-local credit vars with clones of the session fields; merge back into session on stmt.Commit so subsequent actions (including preempt) can consume the credits. Preempt changes: - Idle fast path (pipelineWithIdleResources): bypasses Allocatable when the task requests no dimension tracked by the queue's Deserved set. Fixes MPI Launcher preemption on GPU-only queues where the Launcher requests only CPU/memory. - Credit fast path (pipelineWithEvictionCredits): pipelines onto a node whose session-level eviction credits cover the full request, without new victim discovery. - Credit accumulation: after a successful eviction+pipeline in normalPreempt, records freed resources into the session credit pool so later preemptor tasks benefit from the credit fast path. Signed-off-by: Hajnal Máté <hajnalmt@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.