-
-
Notifications
You must be signed in to change notification settings - Fork 8.6k
Add guidance for testing conventions in each language #16734
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: trunk
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hyperlinking to each binding readme would also be beneficial.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, this is a good idea. I was going to do that, but our READMEs aren't consistent right now with what they do and don't contain, and I was thinking it might make sense to update them to mostly point to our website docs, and right now the coding agents we're targeting can't access the internet.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I added an issue to track this - #16740
and run specific test with bazel
| | `@NoDriverBeforeTest` | Test needs custom capabilities or tests driver creation itself. Driver is destroyed, must use `createNewDriver(capabilities)` in the test to create one. | | ||
| | `@NoDriverAfterTest` | Test leaves browser in a bad state. Driver is restarted after the test. Also accepts `failedOnly`. | | ||
|
|
||
| For tests needing two browsers simultaneously (e.g., multi-user scenarios), create a second instance with `localDriver = new ChromeDriver()`. This driver is automatically quit after the test. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nope! This will (at best) create a Chrome instance instead of the browser being used for the tests. The only reason why the browser is closed after tests is because the entire process tree is killed.
The correct way to start a new browser instance is via something like:
RemoteWebDriver.builder().oneOf(Browser.detect()).build();There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you! This is exactly why we need this file. (this is much more helpful than the generic warning I made about being careful hard-coding creation of new drivers)
Also, if you name it localDriver, it is automatically checked in after hook: https://github.com/SeleniumHQ/selenium/blob/trunk/java/test/org/openqa/selenium/testing/JupiterTestBase.java#L90
| ## Build Files | ||
|
|
||
| * Adding tests shouldn't require Bazel changes—`rb_integration_test` uses glob patterns. | ||
| * Make sure `*_spec.rb` files are in a directory with a `BUILD.bazel` containing `rb_integration_test`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think we should add a section on how to use environment variables especially for BiDi testing @titusfortner :
Environment Variables
Environment variables control test execution behavior and enable specific features.
BiDi Testing
To run tests with BiDi (Bidirectional) protocol enabled:
# Enable BiDi for all tests
WD_REMOTE_BROWSER=chrome BIDI=true bazel test //rb/spec/integration/...
# Run BiDi-specific tests
bazel test //rb/spec/integration/selenium/webdriver/bidi/...Available Variables
| Variable | Purpose | Values | Example |
|---|---|---|---|
BIDI |
Enable BiDi protocol | true, false |
BIDI=true |
WD_REMOTE_BROWSER |
Specify browser for remote tests | chrome, firefox, edge, safari |
WD_REMOTE_BROWSER=firefox |
HEADLESS |
Run tests in headless mode | true, false |
HEADLESS=true |
DEBUG |
Enable debug logging | true, false |
DEBUG=true |
Examples
# Run Chrome tests with BiDi enabled
BIDI=true bazel test //rb/spec/integration/... --test_tag_filters=chrome
# Run headless Firefox tests
HEADLESS=true bazel test //rb/spec/integration/... --test_tag_filters=firefox
# Run remote tests on Edge with BiDi
WD_REMOTE_BROWSER=edge BIDI=true bazel test //rb/spec/integration/... --test_tag_filters=remote
# Combine multiple variables
BIDI=true HEADLESS=true DEBUG=true bazel test //rb/spec/integration/selenium/webdriver/bidi/...Testing Guard Behavior
Environment variables interact with test guards. For example:
# This test only runs when BiDi is enabled
it 'uses BiDi feature', only: {bidi: true} do
# Test code
end
# This test is excluded when BiDi is enabled
it 'classic WebDriver only', exclusive: {bidi: false} do
# Test code
endRun with BIDI=true to see these guards in action.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Please add away, you should be able to commit to this PR directly. Everything here - https://github.com/SeleniumHQ/selenium/blob/trunk/README.md#ruby-1 should probably be included.
Thanks!
| bazel test //dotnet/... --flaky_test_attempts=3 --pin_browsers=true | ||
| bazel test //dotnet/... --test_output=all --pin_browsers=true | ||
| ``` | ||
|
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A few lines about running tests in the IDE.
| ### IDE Support | |
| The test infrastructure supports Visual Studio's and Rider's IDE, which can be more convenient in the inner loop when developing features and writing tests. Simply open the solution after building with `bazel` and the IDE should work like normal. | |
| Note that any bugs involving a discrepancy between the `.csproj` and `BUILD.bazel` files would manifest in running the tests through the IDE rather than with `bazel test`; any changes that affect either build systems should be tested with `bazel test` to ensure proper functionality. |
Reason for PR
Each language has its own tribal knowledge right now for how it works, from annotations to helper methods to common locations for things. This is intended to document those things to make it easier to work in each language.
💥 What does this PR do?
Removes language specific testing details from primary README and moves everything to separate
TESTING.mdfiles💡 Additional Considerations
This is in draft because they were auto-generated as a place to start. I updated some of them more than others, but they can all use a lot of work. Please assist, thanks.