Run WebGPU CTS Tests Locally: A Comprehensive Guide for Chrome

Testing WebGPU implementations is crucial, and running the Compatibility Test Suite (CTS) locally with Chrome enables faster iteration and debugging of your projects. This guide provides step-by-step instructions to run the WebGPU CTS effectively.

Setting Up Your Environment: Chromium Checkout

To run the WebGPU CTS locally, you’ll first need a Chromium checkout. Building Chrome from source grants you full access to harness functionalities like symbolizing stack dumps. If you skip building Chrome, you can still execute the CTS using an existing Chrome executable.

• Follow the official Chromium build instructions to get the source code.
• Build the telemetry_gpu_integration_test targets:

  ./content/test/gpu/run_gpu_integration_test.py webgpu_cts --browser=exact --browser-executable=path/to/your/chrome-executable
  

  Copy

Command-Line Arguments: Tailoring the CTS to Your Needs

Several command-line arguments will help you customize your testing sessions:

• --passthrough --show-stdout: Displays browser output for detailed insights.
• --test-filter: Specifies which tests to run, allowing focused debugging.
• --jobs=1: Limits execution to a single browser instance, ideal for debugging. For running multiple tests, increase this number if necessary.
• --force_low_power_gpu and --force_high_performance_gpu: Choose the specific GPU for WebGPU testing, particularly useful on dual-GPU systems. You can use --use-webgpu-power-preference to control which one WebGPU chooses.
• --enable-dawn-backend-validation: Enables extra validation layers from Dawn.
• --use-webgpu-adapter=[default,swiftshader,compat]: Selects a particular WebGPU adapter within the browser.
• --help: Shows all the available commands.
• -l: Lists all the tests that will be executed.

Running WebGPU CTS on Android: A Detailed Walkthrough

Executing the full CTS on Android with expectations demands additional setup.

1. Build the Target: Build the telemetry_gpu_integration_test_android_chrome target within the out/Release or out/Debug directories.
2. Execute Tests: Use the command line argument --browser=android-chromium (defaults to out/Release) along with --jobs=1.

   ./content/test/gpu/run_gpu_integration_test.py webgpu_cts --show-stdout --browser=android-chromium --stable-jobs --jobs=1 --extra-browser-args="--enable-logging=stderr --js-flags=--expose-gc"
   

   Copy

Warning: Android testing can take upwards of 4 hours.

Troubleshooting Android Issues

• Root Access: Running with root access (userdebug image) is generally preferred. If unavailable, use --compatibility-mode=dont-require-rooted-device, but be aware that this is unsupported and might lead to errors, especially if other Chrome versions are running.
• Proxy Errors: If you encounter ERR_PROXY_CONNECTION_FAILED, clear Chrome's command-line proxy settings:

  build/android/adb_chrome_public_command_line ""
  

  Copy

Testing on Chrome's Infrastructure: Using Swarming

Swarming enables testing your local builds on Chrome's infrastructure, which is beneficial for reproducing bugs that are difficult to replicate locally.

1. Build the Isolate: Create an "isolate" package of your local build. Make sure you are building on the same OS as the target swarming bot.

   vpython3 tools/mb/mb.py isolate out/Release telemetry_gpu_integration_test
   

   Copy

2. Upload the Isolate: Archive and upload the isolate.

   ./tools/luci-go/isolate archive -cas-instance chromium-swarm -i out/Release/telemetry_gpu_integration_test.isolate
   

   Copy

   Save the generated hash.

3. Trigger the Swarming Job: Use the isolate hash and dimensions of the target bot. Retrieve these dimensions and commands from an existing task.

   ./tools/luci-go/swarming trigger -S https://chromium-swarm.appspot.com -d "pool=chromium.tests.gpu" -d "cpu=x86-64" -d "gpu=8086:9bc5-20.0.8" -d "os=Ubuntu-18.04.6" -digest 95199eb624d8ddb6ffdfe7a2fc41bc08573aebe3d17363a119cb1e9ca45761ae/734 -- vpython3 testing/test_env.py testing/scripts/run_gpu_integration_test_as_googletest.py content/test/gpu/run_gpu_integration_test.py --isolated-script-test-output=${ISOLATED_OUTDIR}/output.json webgpu_cts --browser=release --passthrough -v --show-stdout --extra-browser-args="--enable-logging=stderr --js-flags=--expose-gc --force_high_performance_gpu --enable-features=Vulkan" --total-shards=14 --shard-index=0 --jobs=4 --stable-jobs
   

   Copy

Reviewing Failure Expectations

Dawn's CTS test results, categorized into core and compat functionalities, are stored as failure expectations files:

• expectations.txt: Core WebGPU functionality.
• compat-expectations.txt: WebGPU “compat” functionality.
• Use failure_browser/index.html for interactive review.

By following this comprehensive guide, you can effectively run the WebGPU CTS locally, debug issues, and contribute to the ongoing development of WebGPU in Chrome.