feat: workflow overhaul

[jekabs-karklins]

Description

This PR

• introduces new database structures to improve the workflow management system
• extends functionality by making workflows more capable
• improves UI to enable new capabilties and align with material design guidelines

Motivation and Context

The current workflow management system has limitations and inefficiencies that are addressed by the introduction of new database structures. This change is required to enhance the system's performance and user experience.

State machine

This workflow overhaul aims to align the state changes of proposal to classical state machine behaviour. The implementation of abstract state machine can be found in stateMachnine.ts. It matches the popular xState library interface just is much more simple.

There are 5 main concepts in the state machine.

• Status (State): A specific, stable stage in a proposal's lifecycle (e.g., DRAFT). The proposal stays in this state until a transition is made.
• Event: A signal or trigger that initiates a potential transition (e.g., PROPOSAL_SUBMITTED). It informs the state machine engine that an action has occurred and a state change may be required (guard might abort the transition).
• Guard: A validation or condition (e.g., isProposalInstrumentSelected) that must return true for the transition to proceed. It acts as a gatekeeper to ensure business rules are met.
• Action (Side-effect): An automated task executed during the transition itself (e.g., notifying RabbitMQ). These are the reactive behaviors that happen as a result of the state change.
• Transition: The process of moving from one status (source) to another (target). A transition is triggered by an Event, must clear any Guards, and may execute one or more Actions as it resolves.

ER diagram of the new Workflow Data Structure

Note: in order to keep the PR smaller I have not moved some tables into the DB but kept them in the code (color gray), but it makes sence to depict them here in the ER diagram.

Entity Tables:

• workflows: contains the main record for workflows containing name, description and workflow_type
• statuses: contains all statuses that the entity (proposal, exp.safety) can be in
• proposals: the propoals table
• workflow_status_connections: defines the transitions (see Transition under "State machine" section)
• workflow_status_changing_events: defines the events (see Event under "State machine" section)
• workflow_status_actions: defines the actions/side-effects (see Action under "State machine" section)
• workflow_status_changing_guards: defines the guards effects (see Guard under "State machine" section)

Changes

This PR removes a lot of unneeded code in a way simplifying the implementation

Here are the main changes to pay attention when reviewing

1. Cleanup of the namings DefaultStatus and InitialStatus.
   Initial status is the status of the entity when it enters the workflow. DRAFT for proposal and AWAITING_ESF for safetyReview.
   Defaultstatus is status which system is aware of and is not created by user with through "Create status" button. These two namings were mixed and used interchangebly, but now they are separated.

2. The table proposal_events and experiment_safety_events has been dropped and their alternative implementation has been implemented in guards which now checks if the condition is true during the transition. (see apps/backend/src/workflowEngine/guards/*.ts)

2.1 There might be an important change for STFC. Since I have removed the proposal_events table and along with it this proposalEvents.proposal_all_fap_reviews_submitted. Since this is not available any more I have changed to check if all fap reviews are submited by inspecting the fap review table columns https://github.com/UserOfficeProject/user-office-core/pull/1297/changes#diff-f3b982a00651344db8c6516a09c950e0e30df0ac84e1cd543826a0d45875588eR209

3. Status primary key now uses the semantic string statusId (formerly short_code), dropping the surrogate int. This removes number => string mapping in the code, and keeps IDs readable in logs/DB dumps and tables that reference status. This prevents mismatch bugs where int keys and short codes drifted. Migration 0205_RenameShortCodeToStatusId.sql rewires FK references across proposals/experiment_safety/workflow tables and reconstitutes dependent views.
   Operationally this makes debugging and API payloads self-describing and aligns status identity with the business identifier.

4. Status actions are now attached on transition. Before they were attached the status. This means, that if you change the status manually, the status action will not be run.

How Has This Been Tested?

Added and updated e2e tests

Fixes Jira Issue

https://jira.esss.lu.se/browse/SWAP-4949

[jekabs-karklins]

Just to echo my reply from yesterday, I do think we should avoid duplicating the status field so that we aren't storing the value in multiple places that could fall out of sync

Hi @ACLay . I have now normalized te database and removed the column status_id and solely relying in workflow_status_id, this way there is no duplicationl

[ACLay]

Given that we only seem to have the one state machine, and it seems designed to cover all our types of workflow, the "simple" in it's directory name seems unnecessary. Can we rename "simpleStateMachine" to just "stateMachine"?

[ACLay]

Read through to the end of /e2e

[simonfernandes]

I haven't reviewed the code but this looks fantastic 👍

I was just getting this running locally and there are some workflow changes that STFC would need to make, where we have proposals in statuses that the workflows don't contain:

-- Add SUBMITTED_LOCKED status to 'Import workflow'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(5, 15, 1, 1, 0);

-- Add SUBMITTED_LOCKED status to 'Import workflow for Xpress'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(92, 15, 1, 1, 0);

-- Add UNSUCCESSFUL status to 'Import workflow for Xpress'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(92, 20, 1, 1, 0);

-- Add FINISHED status to 'Import workflow for Xpress'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(92, 21, 1, 1, 0);

-- Add EXPIRED status to 'Import workflow'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(5, 9, 1, 1, 0);

-- Add EXPIRED status to 'ISIS Rapid workflow'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(4, 9, 1, 1, 0);

-- Add EXPIRED status to 'ISIS Xpress workflow'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(91, 9, 1, 1, 0);

These statuses are all added to the workflows as unconnected and I don't see a problem with doing this.

Automatic status changes (e.g. on proposal submission) are working for Xpress/Technique proposals and regular proposals, and status actions are being executed.

But manual status changes are only partly working for Xpress/Technique proposals and regular proposals. The status is changing successfully but I'm seeing the below error and status actions are not being executed:

[2026-05-06T13:03:36.187+01:00] ERROR - Unhandled NODE exception
{"exception":{"name":"Error","message":"Undefined binding(s) detected when compiling FIRST. Undefined column(s): [workflow_status_connection_id] query: select * from "workflow_status_connections" where "workflow_status_connection_id" = ? limit ?","stack":"Error: Undefined binding(s) detected when compiling FIRST. Undefined column(s): [workflow_status_connection_id] query: select * from "workflow_status_connections" where "workflow_status_connection_id" = ? limit ?\n at QueryCompiler_PG.toSQL (/app/backend/node_modules/knex/lib/query/querycompiler.js:112:13)\n at QueryBuilder_PostgreSQL.toSQL (/app/backend/node_modules/knex/lib/query/querybuilder.js:84:44)\n at ensureConnectionCallback (/app/backend/node_modules/knex/lib/execution/internal/ensure-connection-callback.js:4:30)\n at Runner.ensureConnection (/app/backend/node_modules/knex/lib/execution/runner.js:318:20)\n at processTicksAndRejections (node:internal/process/task_queues:105:5)\n at Runner.run (/app/backend/node_modules/knex/lib/execution/runner.js:30:19)"},"level":"error","message":"Unhandled NODE exception","trace_id":"835d6d47d64dbb656316046cb9748618","span_id":"c8c52cfd98351876","trace_flags":"01","service_name":"proposal-backend","timestamp":"2026-05-06T13:03:36.187+01:00"}
[ERROR] 13:03:36 Error: Undefined binding(s) detected when compiling FIRST. Undefined column(s): [workflow_status_connection_id] query: select * from "workflow_status_connections" where "workflow_status_connection_id" = ? limit ?

[ACLay]

Okay, initial readthrough complete.

[ACLay]

Starting to look over the changes made since I did my initial comments

[ACLay]

The first couple of reject calls were replaced with throw new GraphQLError(...), but from here it looks like they're replaced with throw new Error(...). Is that difference significant?

[jekabs-karklins]

This change is also not part of this Pull Request

[ACLay]

Okay, finished reading through all the changes since I started reviewing.

[ACLay]

This feels like we're writing our own xml/html parser, why do we need this?

[jekabs-karklins]

Hi, @ACLay, I'm sorry but this is not a change of this Pull Request. I am not sure why is it showing up here. Github bug?

About this changhe. Three weeks ago we were having some problems with this library due to depdendencies, and we decided to remove it completely as we were using only one function.
#1476

[jekabs-karklins]

Given that we only seem to have the one state machine, and it seems designed to cover all our types of workflow, the "simple" in it's directory name seems unnecessary. Can we rename "simpleStateMachine" to just "stateMachine"?

Yes, I think it makes sense. Thanks

[jekabs-karklins]

I haven't reviewed the code but this looks fantastic 👍

I was just getting this running locally and there are some workflow changes that STFC would need to make, where we have proposals in statuses that the workflows don't contain:

-- Add SUBMITTED_LOCKED status to 'Import workflow'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(5, 15, 1, 1, 0);

-- Add SUBMITTED_LOCKED status to 'Import workflow for Xpress'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(92, 15, 1, 1, 0);

-- Add UNSUCCESSFUL status to 'Import workflow for Xpress'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(92, 20, 1, 1, 0);

-- Add FINISHED status to 'Import workflow for Xpress'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(92, 21, 1, 1, 0);

-- Add EXPIRED status to 'Import workflow'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(5, 9, 1, 1, 0);

-- Add EXPIRED status to 'ISIS Rapid workflow'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(4, 9, 1, 1, 0);

-- Add EXPIRED status to 'ISIS Xpress workflow'
INSERT INTO workflow_connections(workflow_id, status_id, pos_x, pos_y, sort_order)
VALUES(91, 9, 1, 1, 0);

These statuses are all added to the workflows as unconnected and I don't see a problem with doing this.

Automatic status changes (e.g. on proposal submission) are working for Xpress/Technique proposals and regular proposals, and status actions are being executed.

But manual status changes are only partly working for Xpress/Technique proposals and regular proposals. The status is changing successfully but I'm seeing the below error and status actions are not being executed:

[2026-05-06T13:03:36.187+01:00] ERROR - Unhandled NODE exception
{"exception":{"name":"Error","message":"Undefined binding(s) detected when compiling FIRST. Undefined column(s): [workflow_status_connection_id] query: select * from "workflow_status_connections" where "workflow_status_connection_id" = ? limit ?","stack":"Error: Undefined binding(s) detected when compiling FIRST. Undefined column(s): [workflow_status_connection_id] query: select * from "workflow_status_connections" where "workflow_status_connection_id" = ? limit ?\n at QueryCompiler_PG.toSQL (/app/backend/node_modules/knex/lib/query/querycompiler.js:112:13)\n at QueryBuilder_PostgreSQL.toSQL (/app/backend/node_modules/knex/lib/query/querybuilder.js:84:44)\n at ensureConnectionCallback (/app/backend/node_modules/knex/lib/execution/internal/ensure-connection-callback.js:4:30)\n at Runner.ensureConnection (/app/backend/node_modules/knex/lib/execution/runner.js:318:20)\n at processTicksAndRejections (node:internal/process/task_queues:105:5)\n at Runner.run (/app/backend/node_modules/knex/lib/execution/runner.js:30:19)"},"level":"error","message":"Unhandled NODE exception","trace_id":"835d6d47d64dbb656316046cb9748618","span_id":"c8c52cfd98351876","trace_flags":"01","service_name":"proposal-backend","timestamp":"2026-05-06T13:03:36.187+01:00"}
[ERROR] 13:03:36 Error: Undefined binding(s) detected when compiling FIRST. Undefined column(s): [workflow_status_connection_id] query: select * from "workflow_status_connections" where "workflow_status_connection_id" = ? limit ?

Hi, @TCMeldrum Regarding the status actions on manual changes there is another PR addressing it
#1441