ai_testing

This module shall provide an AI Agent that is capable of taking human natural language instructions and automatically generating end-to-end UI tests, using browser actions that emulate those of a real user. The agent shall use Computer Vision / OCR to identify the elements on the page with which to interact. This will mean that the test creation agent will be able to react to changes in the DOM and modify the generated tests accordingly, if the changes do not break the functionality being tested but simply change the look/feel of the page.

This is of course a drastically different approach than simply using Selelenium-based tests and having to update them manually for every minor UI change. The tests that are generated will be dynamic, intelligent, and potentially able to integrate with a suite of code-generating functions in such a way that the site can be self-healing. This means that in certain cases when the Agent encounters certain specific errors occuriing, it will be able to analyze the reason for the failure of the test and modify the application code to fix the error. This certainly will not be one of the first features to be developed, but that shall be the goal that we will be shooting for.

The result of the runs that the Test Agent completes shall be generated test case code that uses familiar technologies already integrated into Drupal - such as Behat/Selenium frameworks - but the difference will be that now the tests will be created dynamically after code changes occur, and the AI that is generating the tests will be able to react to changes to the UI and modify the Behat tests accordingly, and in some cases even orchestrate code generation to fix bugs that are encountered during the run. The functionality can be expanded to generate test cases not only in Behat/Selenium, but also potentially Playwright or Cypress or any other End-to-End testing framework that can be integrated with Drupal. Behat has the advantage because of its ability to execute Drush commands (and regular Drupal commands too), but the module can definitely support multiple frameworks eventually.

Features

- AI agent with the capability of turning natural language into a set of actions, then executing these actions on the Drupal site utilizing Computer Vision to interact with the browser, and then generating complete test cases in Behat.
- AI Agent is able to react to changes in the UI that do not break the functionality being tested, and modify the existing test cases accordingly, leading to less need for manually changing the test cases
- Self-healing site: when the AI Agent encounters errors, if it can analyze the error and the existing application code and determine a solution then it can trigger code generation functions that will modify the application code and fix the error. Again, this automates processes that were previously done manually and saves developer time.

Post-Installation

- There will be a config page
- The timing of the AI Agent's runs will have to be configured to execute at a specific point in the CI/CD pipeline, or after a deployment to a server
- Ations that every test case must perform will have to be provided to the Agent in plain English, and at least initially it will be important to monitor the Agent to ensure that it doesn't get lost.. or bored like Anthropic's Claude during the presentation when it just decided to browse images of national parks instead of paying attention to what was going on.

Additional Requirements

Two LLMs - one for translating human language into browser-action instructions for the Agent, and one that provides OCR functionality so that our agent can find elements on the page. This is much more similar to how a human user finds elements on the web page (unless they are really into climbing around in the DOM)

Recommended modules/libraries

Will need something that provides Agentic AI functionality.. I believe such a module is already being developed for Drupal as part of the ai suite.

Similar projects

There are plenty of examples now of proprietary, commercial end-to-end testing systems that operate this way leveraging AI and OCR, but I don't believe there is anything like it that's open source and can plug into Drupal.

Supporting this Module

Sudo - The Agency is the primary force behind the development, but we welcome any and all other devs who see value in this initiative to come and contribute.

Community Documentation

Coming soon!

As of this moment this is a placeholder page and we plan to release the full project in the future. We will start checking in code. We are also eager to network with other potential contributors