Sergii Gorbachov’s Slack Engineering report deserves primary credit for making agentic end-to-end testing inspectable rather than promotional. It documents more than 200 experimental runs, compares Playwright MCP, Playwright CLI, and generated Playwright tests, and reports reliability, duration, and cost. Leela Kumili’s InfoQ report brought that work to my attention and accurately emphasized that Slack treats agent-driven execution as a complement to deterministic testing.
The experiment exposes a real distinction. A conventional test prescribes a journey: locate this element, perform this action, then assert this state. An agentic test can instead receive a goal, observe the interface, and choose a viable sequence of actions. This makes execution more adaptable—but it does not remove the need for a trustworthy oracle.
The mechanism allowed to explore a workflow should not automatically own the authority to declare that the workflow is correct.
Adaptation and verification are different responsibilities
Selector changes often break scripted UI tests even when user-visible behaviour remains valid. An agent can sometimes recover by choosing a different control, navigation path, or input method. That capability is useful because the test intent may be more stable than one encoded journey.
But adaptation creates a new ambiguity. If the agent cannot perform the intended action, it may find a different outcome that looks semantically close. It may skip an intermediate obligation, accept stale state, misread a label, or satisfy a weak success condition through an unintended path. A flexible executor can therefore reduce superficial failures while increasing the importance of the final oracle.
Consider a goal such as “send a thread reply and verify that it appears.” The executor may choose how to open the thread and enter the message. It should not be free to redefine what counts as the correct channel, author, content, persistence state, or visibility. Those conditions belong to the test contract.
The safest division is:
- the agent chooses actions under bounded authority;
- the oracle evaluates independently observable postconditions;
- the policy layer restricts data, environments, and side effects;
- the trace records the path actually taken and the evidence observed.
The same model or session may technically participate in several layers, but their evidence should not collapse into one uncheckable statement such as “the agent reports success.”
What Slack’s results actually support
Slack’s experiments do not claim that agents should replace the regression suite. The reported trade-offs point in the opposite direction. Agent-driven runs were slower and materially more expensive than ordinary deterministic execution. Reliability varied with workflow complexity and execution mechanism. The authors position the approach primarily for exploration, debugging, and complex workflows rather than high-frequency CI.
That boundary is important. Agentic execution appears strongest where the path is genuinely part of the uncertainty:
- reproducing a production report whose exact interaction sequence is unknown;
- exploring whether a goal remains achievable after interface changes;
- diagnosing a flaky journey by trying alternative paths;
- inspecting an unfamiliar part of a large application;
- generating a candidate deterministic test after a stable path is found.
It is less persuasive for a critical invariant that already has a cheap, repeatable, deterministic check. Replacing such a test with an agent adds variability, latency, and a larger failure surface without adding useful exploration.
An architecture for inspectable agentic tests
A production design should make five artifacts explicit:
| Artifact | Required content |
|---|---|
| Intent | the user or system outcome being investigated |
| Constraints | allowed actions, environment, data, budget, and stopping conditions |
| Oracle | independently checkable postconditions and forbidden outcomes |
| Trace | observations, actions, tool responses, timing, and model/tool versions |
| Verdict | pass, fail, blocked, or inconclusive, with referenced evidence |
“Inconclusive” is essential. A deterministic test usually has a narrow failure model. An agentic run can be unable to authenticate, exhaust its budget, enter an unexpected state, or lack enough evidence to distinguish product failure from executor failure. Mapping all of these outcomes to pass or fail damages the signal.
The trace also needs semantic information. A video alone shows what appeared on screen but may not establish which workspace, account, API object, or backend state was involved. Where possible, the final oracle should use a separate observation channel: an API query, database read, event record, or domain-specific invariant rather than the agent’s interpretation of the UI it just manipulated.
Test the tester as an adaptive system
An agentic test harness introduces behaviours that the product test suite must itself challenge:
- Can the agent reach a passing state by violating an intermediate rule?
- Does a misleading label or prompt-like user content redirect its actions?
- Will it continue after entering the wrong tenant or account?
- Does it repeat an irreversible action after a timeout?
- Can a weak visual signal override a stronger API-level contradiction?
- Are alternative paths still bounded by the same authorization policy?
- Does the result remain reproducible enough for a person to diagnose?
These are not conventional locator tests. They are tests of planning, authority, observation, and evidence. The harness should include adversarial environments where the easiest apparent path is deliberately invalid.
The right place in the stack
Slack’s work is valuable precisely because it does not present agentic testing as a universal successor to scripted automation. It supplies experimental evidence for a new layer of exploration and shows that the execution environment can matter as much as the model.
The practical conclusion is to keep deterministic checks as the stable regression foundation, use agents where path discovery has real value, and convert repeatable discoveries into cheaper tests when possible. Most importantly, preserve an independent oracle. An agent may adapt the journey; it must not be allowed to adapt correctness until the run passes.