`feature/integration-tests` → `develop` by lbeckman314 · Pull Request #1263 · calypr/funnel

lbeckman314 · 2025-12-03T23:46:48Z

Important

This PR can be revisited after #1404 is merged

Need to add note on how it's different than the integration_tests_on_kind.yaml, for example:

Smaller focus/scope (only tests basic TES task submissions + nf-canary workflow as opposed to full Gen3 stack)

Overview

This PR adds initial support for integration tests against the Gen3 data platform!

Caution

Couple issues with Integration Tests + K8s Tests:

Only triggered on pushed (and not PR’s), so failures would only be reported after merging to develop (as opposed to before!)
Still using old Helm Chart Repo (need to update https://ohsu-comp-bio.github.io/helm-charts → https://calypr.org/helm-charts)

Integration Test Workflow

Tip

integration_tests.yaml

uc-cdis/fence / 31590da

➜ wget https://raw.githubusercontent.com/uc-cdis/fence/refs/heads/master/.github/workflows/integration_tests.yaml -P .github/workflows
‘.github/workflows/integration_tests.yaml’ saved

netlify · 2025-12-03T23:59:34Z

✅ Deploy Preview for funnel-dev ready!

Name	Link
🔨 Latest commit	`038b539`
🔍 Latest deploy log	https://app.netlify.com/projects/funnel-dev/deploys/69377a735ffc1b000856c268
😎 Deploy Preview	https://deploy-preview-1263--funnel-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Added commands to initialize and tidy Hugo modules in the website build process.

…n-tests

Signed-off-by: Liam Beckman <lbeckman314@gmail.com>

netlify · 2026-03-30T20:16:05Z

✅ Deploy Preview for funnel-dev ready!

Name	Link
🔨 Latest commit	`97720f4`
🔍 Latest deploy log	https://app.netlify.com/projects/funnel-dev/deploys/69cb0f02cc89250008153e53
😎 Deploy Preview	https://deploy-preview-1263--funnel-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify project configuration.

[no ci]

paulineribeyre · 2026-05-04T21:25:15Z

This is being done in #1402 instead

paulineribeyre · 2026-05-06T20:40:46Z

Replaced by #1404

Signed-off-by: Liam Beckman <lbeckman314@gmail.com>

+    runs-on: ubuntu-latest
+
+    steps:
+      # -----------------
+      # 1. Cluster Setup
+      # -----------------
+
+      - name: Create Kind cluster
+        uses: helm/kind-action@v1
+        with:
+          cluster_name: funnel-gen3
+
+      - name: Install Helm
+        uses: azure/setup-helm@v4
+
+      # ---------------------------
+      # 2. Install Funnel via Helm
+      # ---------------------------
+
+      - name: Add ohsu Helm repo
+        run: helm repo add ohsu https://calypr.org/helm-charts && helm repo update
+
+      - name: Install Funnel
+        run: |
+          # 'standard' is the default StorageClass created by Kind
+          helm upgrade --install funnel ohsu/funnel \
+            --set storage.className=standard \
+            --set storage.provisioner=rancher.io/local-path \
+            --wait --timeout=60s
+
+      - name: Wait for Funnel server
+        run: kubectl rollout status deployment/funnel-server --timeout=60s
+
+      # ----------------------------------
+      # 3. Install Gen3-Workflow via Helm
+      # ----------------------------------
+
+      - name: Install gen3workflow
+        run: |
+          helm upgrade --install gen3workflow ohsu/gen3workflow \
+            --set funnelUrl=http://funnel:8000 \
+            --wait --timeout=60s
+
+      - name: Wait for gen3workflow
+        run: kubectl rollout status deployment/gen3workflow --timeout=60s
+
+      # -------------------------------------
+      # 4. Expose services for local testing
+      # -------------------------------------
+
+      - name: Port-forward Funnel
+        run: kubectl port-forward svc/funnel 8000:8000 &
+
+      - name: Port-forward gen3workflow
+        run: kubectl port-forward svc/gen3workflow 8080:8080 &
+
+      # ----------------------------
+      # 5. Run Nextflow + nf-canary
+      # ----------------------------
+
+      - name: Setup Java
+        uses: actions/setup-java@v4
+        with:
+          distribution: temurin
+          java-version: '21'
+
+      - name: Install Nextflow
+        run: |
+          curl -s https://get.nextflow.io | bash
+          chmod +x nextflow
+          mkdir -p $HOME/.local/bin
+          mv nextflow $HOME/.local/bin/
+          echo "$HOME/.local/bin" >> $GITHUB_PATH
+
+      - name: Clone nf-canary
+        run: git clone https://github.com/seqeralabs/nf-canary
+
+      - name: Configure nf-canary for TES (Funnel)
+        run: |
+          cat >> nf-canary/nextflow.config <<'EOF'
+          plugins {
+            id 'nf-ga4gh'
+          }
+          process.executor = 'tes'
+          tes.endpoint = 'http://localhost:8000'
+          EOF
+
+      - name: Run nf-canary tests
+        id: nf_canary
+        run: |
+          cd nf-canary
+          nextflow run main.nf -with-report report.html 2>&1 | tee nextflow.log
+          echo "exit_code=${PIPESTATUS[0]}" >> $GITHUB_OUTPUT
+
+      # -------------------
+      # 6. Verify K8s Jobs
+      # -------------------
+
+      - name: Verify Kubernetes jobs completed
+        run: |
+          echo "=== All jobs in default namespace ==="
+          kubectl get jobs -o wide
+
+          FAILED=$(kubectl get jobs \
+            --field-selector=status.failed!=0 \
+            -o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true)
+
+          if [ -n "$FAILED" ]; then
+            echo "Failed jobs: $FAILED"
+            for JOB in $FAILED; do
+              echo "--- Logs for $JOB ---"
+              kubectl logs job/$JOB --tail=50 || true
+            done
+            exit 1
+          fi
+
+          SUCCEEDED=$(kubectl get jobs \
+            --field-selector=status.successful!=0 \
+            -o jsonpath='{.items[*].metadata.name}' 2>/dev/null || true)
+
+          if [ -z "$SUCCEEDED" ]; then
+            echo "No jobs completed successfully — did any tasks run?"
+            kubectl describe jobs || true
+            exit 1
+          fi
+
+          echo "All Kubernetes jobs completed successfully: $SUCCEEDED"
+
+      # ------------------------------
+      # 7. Upload test logs + reports 
+      # ------------------------------
+
+      - name: Upload Nextflow logs
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: nextflow-logs
+          path: |
+            nf-canary/nextflow.log
+            nf-canary/report.html
+            nf-canary/.nextflow.log
+          if-no-files-found: ignore


Add an explicit workflow-level permissions block near the top of .github/workflows/integration-tests.yaml (after on: and before concurrency:) so all jobs inherit least-privilege defaults.
For this workflow, the best minimal non-breaking baseline is:

contents: read (CodeQL-recommended minimum for checkout/read access patterns)

actions: read (safe for reading workflow/action metadata)

packages: read (commonly needed when pulling package/container artifacts; low-risk read scope)

No imports, methods, or definitions are needed (YAML workflow change only).

+      # -----------------
+
+      - name: Create Kind cluster
+        uses: helm/kind-action@v1


+          cluster_name: funnel-gen3
+
+      - name: Install Helm
+        uses: azure/setup-helm@v4


Signed-off-by: Liam Beckman <lbeckman314@gmail.com>

+          EOF
+
+      - name: Setup tmate session
+        uses: mxschmitt/action-tmate@v3


paulineribeyre · 2026-05-14T14:33:25Z

@@ -0,0 +1,31 @@
+name: Integration Tests (Gen3)


hey @lbeckman314 , this one needs to be deleted

docs: Add initial Integration Tests documentation

fbffbc5

Copilot AI review requested due to automatic review settings December 3, 2025 23:46

lbeckman314 changed the base branch from main to develop December 3, 2025 23:47

lbeckman314 self-assigned this Dec 3, 2025

lbeckman314 added the enhancement label Dec 3, 2025

lbeckman314 added this to Funnel Dec 3, 2025

lbeckman314 moved this to In Progress in Funnel Dec 3, 2025

lbeckman314 added docs feature labels Dec 3, 2025

feature: Add initial integration_tests.yaml workflow

0123d40

lbeckman314 added 2 commits December 3, 2025 16:18

fix: Use correct GitHub Actions workflow

65445f0

debug: Add workflow permissions

8785552

Copilot AI reviewed Dec 4, 2025

View reviewed changes

fix: Hugo mod website

f496c2a

Added commands to initialize and tidy Hugo modules in the website build process.

lbeckman314 changed the title ~~develop ← feature/integration-tests~~ feature/integration-tests → develop Dec 5, 2025

paulineribeyre and others added 3 commits December 5, 2025 17:59

Pin to feat/external-repo-support (#1266)

ff9e038

Merge remote-tracking branch 'origin/develop' into feature/integratio…

038b539

…n-tests

feat: Add initial Gen3 Integration Tests

4266f05

Signed-off-by: Liam Beckman <lbeckman314@gmail.com>

github-advanced-security AI found potential problems Mar 30, 2026

View reviewed changes

Comment thread .github/workflows/gen3-integration-tests.yaml Fixed

Comment thread .github/workflows/gen3-integration-tests.yaml Fixed

chore: Refactor gen3 integration workflow with improved timeouts

389e9b1

github-advanced-security AI found potential problems Mar 31, 2026

View reviewed changes

Comment thread .github/workflows/gen3-integration-tests.yaml Fixed

chore: Update gen3-intergration-tests.yaml

97720f4

[no ci]

github-advanced-security AI found potential problems Mar 31, 2026

View reviewed changes

Comment thread .github/workflows/gen3-integration-tests.yaml Fixed

debug: GitHub Actions

4a4c393

github-advanced-security AI found potential problems May 13, 2026

View reviewed changes

Comment thread .github/workflows/gen3-integration-tests.yaml Fixed

Comment thread .github/workflows/gen3-integration-tests.yaml Fixed

chore: Rename integration test workflows

d922bd2

Signed-off-by: Liam Beckman <lbeckman314@gmail.com>

github-advanced-security AI found potential problems May 13, 2026

View reviewed changes

lbeckman314 added 2 commits May 13, 2026 15:46

debug: GA

6b5cc68

Signed-off-by: Liam Beckman <lbeckman314@gmail.com>

debug: GitHub Actions

64d7cbb

github-advanced-security AI found potential problems May 13, 2026

View reviewed changes

Comment thread .github/workflows/integration-tests.yaml Fixed

lbeckman314 added 2 commits May 13, 2026 16:38

debug: GA

b759e9d

Signed-off-by: Liam Beckman <lbeckman314@gmail.com>

debug: GitHub Actions

8cbce05

Signed-off-by: Liam Beckman <lbeckman314@gmail.com>

github-advanced-security AI found potential problems May 13, 2026

View reviewed changes

paulineribeyre reviewed May 14, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`feature/integration-tests` → `develop`#1263

`feature/integration-tests` → `develop`#1263
lbeckman314 wants to merge 16 commits into
developfrom
feature/integration-tests

lbeckman314 commented Dec 3, 2025 •

edited

Loading

Uh oh!

netlify Bot commented Dec 3, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

netlify Bot commented Mar 30, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paulineribeyre commented May 4, 2026

Uh oh!

paulineribeyre commented May 6, 2026

Uh oh!

Uh oh!

Uh oh!

Check warning

Copilot Autofix

Check warning

Check warning

Uh oh!

Check warning

paulineribeyre May 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

@@ -3,6 +3,11 @@
             on:
               push:
+            permissions:
+              contents: read
+              actions: read
+              packages: read
             # Cancel redundant jobs
             concurrency:
               group: ${{ github.workflow }}-${{ github.ref }}

Conversation

lbeckman314 commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Integration Test Workflow

Uh oh!

netlify Bot commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for funnel-dev ready!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

netlify Bot commented Mar 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Deploy Preview for funnel-dev ready!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

paulineribeyre commented May 4, 2026

Uh oh!

paulineribeyre commented May 6, 2026

Uh oh!

Uh oh!

Uh oh!

Check warning

Copilot Autofix

Check warning

Uh oh!

Check warning

Uh oh!

Uh oh!

Check warning

Uh oh!

paulineribeyre May 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lbeckman314 commented Dec 3, 2025 •

edited

Loading

netlify Bot commented Dec 3, 2025 •

edited

Loading

netlify Bot commented Mar 30, 2026 •

edited

Loading