[Feature]: Comprehensive integration testing framework

[PawelPlesniak]

Description

Currently the integration testing framework focuses heavily on the processes that drunc orchestrates, and not on the functionality that drunc provides. We would like to extend this within the drunc repository under a new integtest directory for focusing on the features that drunc provides.

The integration tests (e.g. minimal_system_quick_test.py, daqsystemtest_integtest_bundle.sh) currently do something like

• boot
• conf
• start
• enable-triggers
• disable-triggers
• drain-dataflow
• stop-trigger-sources
• stop
• scrap

without any options provided. We will want tests that do the above with the additional parameters, e.g.

• command targeting and subsequent behaviour
• including and excluding
• flushing processes and subsequent behaviour
• command sequences
• manual error introductions
  Note the above is non-exhaustive, and will require a series of developments. For now lets start with the base case and build up all the permutations of behaviours we are interested in.

Potential impact radius

Small/Isolated

Reason for change

Several issues have been found with the drunc tooling, and in the direction of providing a resilitent and fault tolerant platform, we want to ensure that all changes that are made do not break current functionality.

Suggested implementations

Using the click CLI runner to run drunc, and a set of commands. Capture tty output with tee.
These are suggestions, not requirements. If you think of/know of better tools for this, happy to discuss.

Testing suggestions

This is the biggest question. The way I see it, we should have tests that support the following

• base configuration, SSH shell PM, all commands, can run everywhere, marker --integtest-ssh-shell
• base configuration, SSH paramiko PM, all commands, can run everywhere, marker --integtest-ssh-paramiko
• base configuration, k8s PM, all commands, can run everywhere, marker --integtest-k8s
• base configuration, subprocess PM (this is a WIP), all commands, can run everywhere, marker --integtest-subprocess
• The same as above but with a host-specific configuration, marker --integtest-np0x

Anything else?

This will likely have to get broken down, but the main idea is to have testing framework to continue discussing all the use cases for the run control and all the behaviours that we want to have

[PawelPlesniak]

This can rely heavily on this infrastructure from daqsystemtest. We should adapt their code that drives integration tests and run the MSQT, adapting the minimal configuration to test a distributed system. @miruuna, lets discuss next week

[bieryAtFnal]

@PawelPlesniak , @miruuna , would it be possible for me to be included in the discussion next week?

If I understand what Pawel has suggested (copying the daqsystemtest_integtest_bundle.py script [e.g. to a drunc_integtest_bundle.py script], and copying a sample integtest from the daqsystemtest repo and modifying/copying it to build up the tests that you want), that sounds good to me.

The part that I would like to advertise is that I'm thinking about a general purpose 'bundle' script.

For now, I believe that copying the daqsystemtest bundle script to a drunc bundle script is the right thing to do. At some point in time, though, I would expect to circle back to chat with you about replacing all of the repo-specific bundle scripts with a general one that could run the drunc integtests and/or the daqsystemtest integtests, etc, and a big part of that discussion would be to make sure that we keep all of the functionality that people are used to in the repo-specific scripts.

Thanks

[PawelPlesniak]

@miruuna @bieryAtFnal This sounds like a good idea, unless there are objections, I would suggest Monday 5pm CEST (4pm BST, 10am CST). We should discuss

• Current test structure implementation and configuration files, including now to define the sequence of commands that is executed in the current implementation
• Planned organization of the structure to run all integration tests from all repositories, so we can follow the plan for simple integration for running all tests in a suite

I recognise that the model of executing all the tests may be a work in progress, and as such if it is more beneficial, we can schedule this for later in the week to not get meeting fatigue.

[bieryAtFnal]

The suggested time of 5pm CEST on Monday sounds good to me.

[miruuna]

@PawelPlesniak Agree, a meeting would be very useful. Monday at 5pm CET works for me.