Skip to content

feat: add cron orchestration for argo workflows#1422

Open
YufanKambang wants to merge 6 commits into
developfrom
cron-orch-argo-workflow
Open

feat: add cron orchestration for argo workflows#1422
YufanKambang wants to merge 6 commits into
developfrom
cron-orch-argo-workflow

Conversation

@YufanKambang
Copy link
Copy Markdown
Contributor

@YufanKambang YufanKambang commented Mar 17, 2026 •
edited
Loading

Description

This is for this epic with a stfc cron orchestration service: UserOfficeProject/issue-tracker#1540
This will close this sub-issue: UserOfficeProject/issue-tracker#1552
This is to demo the use of argo-workflow with our own services to schedule in CRON the asyncJobs that

Motivation and Context

How Has This Been Tested

Fixes

Changes

Depends on

Tests included/Docs Updated?

  • I have added tests to cover my changes.
  • All relevant doc has been updated

@YufanKambang YufanKambang requested a review from a team as a code owner March 17, 2026 13:30
@YufanKambang YufanKambang requested review from Bhaswati1148 and removed request for a team March 17, 2026 13:30
@YufanKambang YufanKambang changed the title feature: cron orchestration argo-wroflow entrypoint script feat: cron orchestration argo-wroflow entrypoint script Mar 17, 2026
@YufanKambang
Copy link
Copy Markdown
Contributor Author

YufanKambang commented Mar 17, 2026

In the apps/backend/package.json we have defined new scripts.

When commanding:

cd apps/backend
npm run test:job:local -- checkAllCallsEnded

We where able to know that our code compiled successfully, the DI container initialized ("Invalid or no value was provided for the DEPENDENCY_CONFIG"), the job tried to run.
However, it hit a native library issue (muhammara for PDFs) that requires musl libc, which your local Ubuntu doesn't have. I think this is fine as the docker container will have the right libc.

@YufanKambang YufanKambang changed the title feat: cron orchestration argo-wroflow entrypoint script feat: cron-orchestration-argo-workflow Mar 17, 2026
@YufanKambang YufanKambang mentioned this pull request Mar 17, 2026
Open
@YufanKambang YufanKambang changed the title feat: cron-orchestration-argo-workflow feat(argo): cron-orchestration-workflow Mar 17, 2026
@YufanKambang YufanKambang changed the title feat(argo): cron-orchestration-workflow feat: add cron orchestration for argo workflows Mar 17, 2026
This was referenced Mar 18, 2026
Closed
Closed
@YufanKambang
Copy link
Copy Markdown
Contributor Author

YufanKambang commented Mar 18, 2026

✅ Created runJob.ts a CLI entrypoint for Argo jobs
✅ Built Dockerfile.worker a lean Docker image for job execution
✅ Built and deployed the image to my local Minikube
✅ Created Kubernetes Secret for database credentials, local database needs to be running for this to express
✅ Created and deployed CronWorkflow to Argo, made a specific one for user-office-core/apps/backend/src/asyncJobs/jobs/checkAllCallsEnded.ts
✅ Tested the job successfully in Argo UI and the terminal
✅ Verified logs showing job execution in the Argo UI and terminal

Scott-James-Hurley
Scott-James-Hurley reviewed Mar 27, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@Scott-James-Hurley Scott-James-Hurley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this a proof of concept or something intended to be merged?

Comment thread apps/backend/Dockerfile.worker
Comment thread apps/backend/Dockerfile.worker Outdated
Comment thread apps/backend/Dockerfile.worker
RUN cd apps/backend && npm run build

# Set working directory to backend
WORKDIR /app/apps/backend
Copy link
Copy Markdown
Contributor

@Scott-James-Hurley Scott-James-Hurley Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Couldn't this be moved up to when you start running backend scripts?

Comment thread apps/backend/Dockerfile.worker Outdated
Comment thread apps/backend/src/asyncJobs/runJob.ts
);

await checkAllCallsEndedJob.functionToRun(callDataSource);
console.log('Job completed successfully');
Copy link
Copy Markdown
Contributor

@Scott-James-Hurley Scott-James-Hurley Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we probably want to use the user office logger instead of console log.

@YufanKambang
Copy link
Copy Markdown
Contributor Author

YufanKambang commented Mar 30, 2026

This is a POC.
I have presented the demo successfully today to Deepak and Hassan.
I have been guided to look more into docker-orchestration to see how this can be integrated. As we use argo-CD and that lives in docker-orchestration

@YufanKambang YufanKambang mentioned this pull request Mar 31, 2026
Open
@YufanKambang
Copy link
Copy Markdown
Contributor Author

YufanKambang commented Apr 21, 2026 •
edited
Loading

This is to not be merged in, However there are file that are vital to be able to use Argo-CD for us. Such as the entrypoint scripts that will let argo run jobs based on the cron-wrokflow yamls.

The recommendation made by Scott regarding the logging and the Dockerfile.worker, are very helpful

@YufanKambang
Copy link
Copy Markdown
Contributor Author

YufanKambang commented May 20, 2026

Reopening this pr to test this branch on dev cluster

@YufanKambang YufanKambang reopened this May 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Scott-James-Hurley Scott-James-Hurley Scott-James-Hurley left review comments

@Bhaswati1148 Bhaswati1148 Awaiting requested review from Bhaswati1148 Bhaswati1148 is a code owner automatically assigned from UserOfficeProject/stfc-user-programme

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@YufanKambang @Scott-James-Hurley