ATDD

ATDD stands for “Acceptance Test-Driven Development”. ATDD finds its root in the TDD practice. The ATDD idea seems to emerge with Lisa Crispin [Crispin 2000] who already suggests automating Acceptance Tests (AT) written before the code [Kaner 2002].

ATDD consists in [Axelrod 2018]:

defining the ATs for each new User Stories (US)

implementing the ATs with production code

the minimal production code is generated to pass the ATs along with the Definition of Done (DoD)

when all ATs pass, the US passes; it is “Done” if the DoD also passes [Schwaber 2020]

ATDD is not only for GUI but for any kind of interface the US refers to such as REST APIs , RPC or other mechanisms to address Testability of the Product Backlog .

ATs can [Axelrod 2018]

help the organization focussing on the Customer since it improves the communication between Business Owner and the Development Team

help the Development Team understand progress and what to test

help both customers and Development Teams to make sure that the project is progressing in the right direction [Crispin 2000] [Deng 2009]

force to start planning testable production code

ensures that Testers and Scripters are involved as early as possible, allowing them to really influence the quality of the product

be used as a regression test and allows automatic execution on a regular basis and allows the code to be safely refactored

improve the accuracy of the development effort estimate if ATs are written beforehand (for example, during a 3 Amigos or an Example Mapping session at Sprint Refinement time).

be used as documentation the “Agile way” and provide “executable specifications” [Ambler 2005]

be run right after the simplest line of code supposed to implement the AT instead of waiting for a release candidate of the whole solution as per the “ Shift Left ” testing principle

be repeated at higher rates if automated, rather than at the end of the SDLC, reducing thus non-compliance risks very soon

ATDD are executable specifications, it means ATDD is like a “ Möbius strip ”: both sides of AT are actually the same; the automation of the test script makes them runnable [Moustier 2020].

‍

‍

‍

automation reduces the so-called transaction costs [SAFe 2021-06]: in iterative delivery model, each time a delivery batch to be deployed bears some costs which should be kept low to enable frequent releases

low transaction costs also reduces the holding cost [SAFe 2021-06]: delaying release will generate profit loss from Customers and Competitors may take advantage from undelivered added value

automation provide fast, fully repeatable and precise tests [Axelrod 2018]

it help running regression testing without any knowledge of the product

Agile, and overall DevOps, are based on heavy automation strategy

‍

“Test automation” has of course some drawback such as

scripts maintenance to fit changes on interfaces such as UI or API

test data must be maintained/migrated to match business changes and their implementation

false positives (aka “Flaky tests”) must be chased in a Jidoka approach

scripting is not testing and relying only on scripted test cases is not enough [Bach 2014]

if test automation is done in silo mode , bottlenecks will raise

automated tests must start from an initial well-known state, because a failure in the scenario aborts the test and jumps to the next one; therefore, the failure of a test should not affect the next tests and dependencies between automated tests are strongly discouraged [Axelrod 2018].

ATDD is often confused with Behavior-Driven Development (BDD). BDD is a methodology which emphasizes bridging the gap between the description of a behavior and its implementation. The behavior is described in natural language and replaces formal specifications. It comes along with tests that verify the feature [North 2006] [Axelrod 2018] [Moustier 2019-1].

While BDD is focussed on behaviors from a Customer point of view, ATDD aims to test acceptance criteria. On one hand, BDD describes the business (outside-in) and provides criteria thanks to so-called “Specifications by Examples” (SBE) [Fowler 2004]; on the other hand, ATDD enables the technical side of the AC, it transitions from outside-in to inside-out for TDD [Moe 2019].

Along with BDD, that was created by Dan North [North 2006], came a tool named “ Cucumber ” [Wynne 2012] [Ye 2013] which compiles a formatted natural language behavior specification into code stubs ready to embed the implementation of the test. BDD enables thus ATDD [Axelrod 2018]

This “natural language” used by Cucumber has been nicknamed “ Gherkin ”. Here is the structure of this natural language:

Feature :

As a

I need to

So that

Scenario X :

Given

When

Then

‍

The amount of scenarios may differ from one US to another; providing too many for the same US is not a good practice since it will generate a US that would be too big to be handled within a Sprint.

‍

From the Gherkin description, most BDD frameworks generate the skeleton of the unit test code. The generated unit test skeleton needs then to be filled with instructions and calls to define [Axelrod 2018]

the initial condition - the “Given” part

the product to the expected situation - the “When” part

the defined checks to be done - the “Then” part

These tools are actually based on keywords that receive the implementation, just like the Keyword-Driven Testing (KDT) approach where each part of the Gherkin is considered as a keyword [Deng 2009].

Other BDD tools allow you to incorporate the documentation into the test code itself, for example:

RSpec for Ruby

Spectrum for Java

MSpec for .Net

Jasmine and Mocha for JavaScript

ATDD is strongly linked with TDD, because of its origin [Crispin 2000] but also because ATDD reinforces TDD by providing the skeleton of unit tests that need to be filled with calls to production code. This generates a double loop learning system [Argyris 1977] [Smith 2001] which enables providing the right solution instead of the sole compliance with unit tests.

ATDD and TDD are a double loop learning system [Ambler 2006]

‍

This double loop learning system should also be included in a higher level loop such as the Lean Startup [Ries 2012]. This would ensure to deliver the right product instead of something that would simply be compliant with the defined US.

‍

Lean Startup as a double loop learning mechanism over the double loop ATDD/TDD [Moustier 2020]

These double loop systems may include intermediate loop levels as per Extreme Programming [Beck 2004] which provides feedback at many levels and many frequencies [DonWells 2013].

Extreme Programming requires planning and feedback at many levels and many frequencies [DonWells 2013]

Since ATDD is based on BDD, non-programmers are then able to create tests [Kaner 2002] and liaise seamlessly with workers at lower levels. Both Gherkin and SBE aim to provide a means of communication that combines usage and acceptance criteria between AT and unit testing, but the "Test First" mindset can be extended to the connected dual-loop systems implemented in the organization. From the ATDD prism, the organizations come in different flavors:

Dedicated team : complete responsibility but the Scripting Team will be disconnected from the Development Team. This configuration introduces issues some issues in a DevOps context [Axelrod 2018]; this leads to merging Scripters within Development Teams

Isolated Scripters within Development Teams would create bottlenecks inside the Team and would reinforce the Conway’s law [Conway 1968], thus increasing issues when integrating parts

T-Shape Team Members : to overwhelm the testing bottleneck, Developers contribute to testing by automating scripts

Feature Teams : once skills are no longer an, the whole Team may handle E2E tests with ATDD - usually, a common goal, few relevant metrics to monitor success, and common values help the Team go beyond the Feature Team paradigm [North 2016].

‍

Another topic that test automation draws is Test Data Management (TDM).

TDM is a major topic in test automation. Data can be generated from production data through some BI and eventually thanks to some IA [Legeard 2021]. When it comes to new features, production data will probably miss; unfortunately, ATDD is mostly regarding new features, making thus an antipattern. Moreover, Data-Driven Testing is easy at testing units level [Axelrod 2018], but when it comes to complex systems and testwares, it can be really difficult because of the amount of data to handle. To help this, data may be organized thanks to a dictionary where each test case owns its own data that should be injected (and eventually anonymized or obfuscated) in the test environment [Axelrod 2018].

At the end of the scenario, data should also be reset to enable test repetition. If tests run in parallel would touch the same data, multiple environments must be made available [Axelrod 2018] notably thanks to:

cloud management along with a 12 Factors approach;

data storage strategies such as shared data in read-only [Axelrod 2018], using database replication mechanism [Wiesmann 2000] or applying a CQRS design [Fowler 2011];

dark launching to route data changes to specific storages when it comes to isolate testing changes

TDM is something complex and hard to tackle; therefore it should be addressed little by little at any time, from the beginning.

Because Agilitest is a #nocode scripting tool, Developers usually don’t trust the technology; therefore, many Agilitest Users are business-oriented people. This situation often leads Managers to set a silo oriented organization to quickly achieve test automation in their company. However, this achievement is not the actual goal. As shown above, the mindset introduced by ATDD is basically a matter of communication; spreading tools that enable ATDD is vital for the delivery flow. Sharing concerns about ATDD tools between Developers and Scripters aims at merging the development / test automation ecocycles . This goes notably through

connecting Agilitest to an existing Jenkins server

extracting data from databases to take advantage of the Data-Driven feature in Agilitest

‍

Finally, test scripts automation infers minor failures that can block the more important parts of the test from running [Axelrod 2018]; therefore, tests must be short. Some of the Agilitest Users own scripts with a lot of steps: a 3000+ steps script has been recorded! The good news is that it proves the tool is robust; however, it is a better practice to design scripts that verify only one thing.

‍

To discover the whole set of practices, click here .

‍