Test Tetrahedron?
Ok, I have listened to Test Pyramid, indeed, there is plenty of content around this concept. Roughly, the test pyramid concept asserts that the first layer of test, Unit, is cheaper, faster, and has the desired confidence property [1,2,3]. As we climb the pyramid through Integrated and End2End (E2E) tests, in the benefit of completeness, these three properties are compromised more and more. Other properties related to the test pyramid are automation and coverage as well as correctness — the property that is asserted when the system under test is correct with respect to its specification.
A pyramid is a polyhedron (a 3D shape), e.g., the Egyptian pyramids built from 2700 B.C. Nonetheless, the common mental model in the IT domain to depict the test pyramid is a triangle (a polygon, a 2D shape) [1,2,3], in fact, the Test Triangle. It is well-known the mental model applied to face a situation plays a major part in the outcome.
And so? The point is to introduce the Test Tetrahedron, in an attempt to enable a better understanding of the test phenomenon, which includes, at least, a better allocation of test efforts.
What is a Tetrahedron?
As previously introduced, a polyhedron is a 3D shape with flat polygonal faces, straight edges, and vertices. A particular type of polyhedron with four triangular faces, six straight edges of the same size, and four vertices is a regular tetrahedron. See Fig. 1 to depict its shape.
The regular tetrahedron is an ancient polyhedron, indeed, one of the Platonic solids. Plato wrote about it in the dialogue Timaeus c. 360 B.C., in which he associated the tetrahedron with the classical element fire since the heat of fire feels sharp and stabbing (like little tetrahedra). It was used also by Kepler in 1596 when he proposed a Solar System model. More importantly, tetrahedra arise naturally in science, e.g., the methane gas, CH4, is a tetrahedral molecule with a carbon atom in the center of the tetrahedron and one hydrogen atom in each one of the four vertices.
The net of a tetrahedron is shown in Fig. 2. Fig. 2 depicts that is possible to “build” a regular tetrahedron using four equilateral triangles.
Test Tetrahedron
Let us start considering a system described by a multitiered architecture in a given company, in which three tiers are prevalent: (1) client-side - it runs in devices out of the control of the company, e.g., a Single-Page Application (SPA) using Angular 7; (2) server-side - the set of services that provides the core business and manages the state (nowadays it can be composed of hundreds of microservices each one with different datastores); and, (3) middle-side - the set of services that provides the mean for the client-side to access the server-side, e.g., proxies, gateways, and API Gateways.
As we did not define the three layers of test in the classical Test Triangle, let us give it a try. Recall these definitions can be highly controversial [3]. (I) Unit - a single behavior (method/function) is tested and all dependencies (they must be a small number) are mocked/stubbed, moreover, they are focused on the actual business behavior logic. (II) Integrated - they test a component stimulating it using some input data and inspecting its output data, by definition, it depends on the availability of the dependencies (several components working together, integrated), which does not mean that such dependencies cannot be managed. Recall some concerns can only be appropriately tested using integrated tests, e.g., transactions, and contracts. (III) E2E - they test the system from the "user perspective" in doing so they test all the system, usually, there is no room for mocked/stubbed dependencies what decreases its confidence property.
Equipped with the architecture tiers and the layers of test, it is time to depict the Test Tetrahedron using the net in Fig. 3. Note the property of correctness is at the "top" of the tetrahedron while the other three properties, namely automation, confidence, and coverage, can interchange their positions in the other three vertices.
Initial Discussion
Firstly, it should be clear that each face requires a complement in the definition of the respective test layers. The most prominent example is the middle-side face since it is questionable to invest effort in the definition of Unit tests in a face that by definition is based on integration and should not have business behavior logic. By the same token, E2E tests take different flavors in each face, e.g., in the server-side face it means that one can test the contract of APIs that expose the business behavior for the other faces, and in the client-side it requires to navigate in a browser for SPAs (possibly a set of browsers).
Secondly, as the definition of the layers required complements for each face, the area of these layers (one triangle at the "top" followed by two trapezoids) in each face can be different, in such a way, that the distribution of the efforts and tests through these three layers of test can be different.
In the client-side and server-side faces, it is usual to divide the integrated tests into, at least, two sublayers: (II.1.) Integrated mocked - in which all the dependencies are in the control of the developer so the development architecture provides alternatives to setup datastores with data and to mock external dependencies enabling the evaluation of transactions and contracts, in such a way, that a developer can locally run such integrated mocked tests without any external dependency as well as DevOps pipelines can run such tests with a medium-high degree of confidence ; (II.2) Integrated hot - in which the dependencies are the likely to be the final ones, such tests are the first to allow the testing with other systems.
Finally, each face can be composed of additional tetrahedra just placing the "base" over one face and so on. The most visible example is the server-side face since it usually has services focused on the middle-side, e.g., GraphQL Servers and Backend-For-Frontend (BFF) services. A composite solid of tetrahedra is shown in Fig. 4.
Pragmatics - Initial Technical Review of the client-side face
Let us focus on an initial technical review of the client-side face using the Test Tetrahedron for a SPA based on Angular 7 archetype app [4], such instantiated archetype has two types of tests: (I) a “unit” test defined by app.component.spec.ts, and (II) an E2E test defined by app.e2e-spec.ts.
DISCLAIMER: Angular 7 is an outdated version, nonetheless, the following discussion is about concepts. In that sense, it does not matter the tool that supports E2E tests (the default tool for Angular 7 is Protractor, or its alternatives, e.g., Cypress) or Unit Tests (the default tool for Angular 7 is Karma, or its competitors, e.g., Jest). Additionally, the following observations on the code do not mean any conceptual problem in the archetype itself but a naive usage of it.
Let us assert the "unit" test and the E2E test have a relevant overlap, as the following extractions show.
"Unit test" (Karma with Jasmine)
// app.component.spec.ts
it('should render title in a h1 tag', () => {
const fixture = TestBed.createComponent(AppComponent);
fixture.detectChanges();
const compiled = fixture.debugElement.nativeElement;
expect(compiled.querySelector('h1').textContent).toContain('Welcome to test-e2e-app!');
});E2E test (Protractor based on Selenium with Jasmine)
// app.po.ts
getTitleText() {return element(by.css('app-root h1')).getText() as Promise<string>;}// app.e2e-spec.ts
it('should display welcome message', () => {
page.navigateTo();
expect(page.getTitleText()).toEqual('Welcome to test-e2e-app!');
});
The bold lines of code show that the same expectation (in the Jasmine sense) is present in the "Unit" test and E2E test. The "Unit" test depends on the rendering of the html of the component to check an expectation concerning a visual concern. Generalizing, this "unit" test (‘should render title in a h1 tag’) is a pattern, in which unit testing is used to evaluate the integration between "parts", perhaps using mocked/stubbed components, a common real-world example is when something is computed in the component then the stubbed visual component is expected to show a message. Therefore, to turn app.component.spec.ts a unit test such expectation about "visual behavior" could be removed so it would be only focused on the actual business behavior logic.
Indeed, this is a tradeoff since such a strict view of the responsibilities of the Unit layer have many consequences, three of them are: less coverage in the unit tests, the same expectation is slower, and exhibits less confidence when run as an "E2E test" [5], and, from the viewpoint of Lines of Code (LoC), it is cheaper to perform it in the "E2E test".
Although coverage is a favorite metric among software stakeholders (in some projects, professionals naively call for 100% of coverage), it is a well-known bad practice to pay excessive attention to test coverage [3]. Nonetheless, a compromise solution is to complement the coverage of unit tests with the coverage of E2E tests, which can be done using the approach present in [4]. In that approach, one runs 'npm run alldevtests' [4] and gets two complementarily lcov files (target/unit/lcov.info and target/e2e/lcov.info).
Back to the responsibilities of the Unit tests which is the actual business behavior logic (at least, in this work), a fair question is how to describe integrated tests, e.g., that ones focused on visual behavior or dependent on calling the server-side through the middle-side. The answer comes from the Angular 7 archetype app [4], the usual ‘ng e2e’ (‘npm run integratedmockedtests’ in [4] defined to run a test suite called core) runs an HTTP server to respond to the HTTP requests regarding the static files, i.e., the development architecture “stubs” a component from the middle-side face of the Test Tetrahedron in order to allow (II.1.) Integrated mocked tests. The same approach is possible for calling the server-side face, e.g., to run a mock server with adequate data ([4] uses json-server to run an "API server" in complement to the HTTP server, see ‘npm run mock-server’ and ‘npm run integratedmockedtests’). This is in accordance with the Test Tetrahedron since it is needed to climb the client-side face but not too high, at this moment.
Up to now, the unit tests and the integrated mocked tests combined must cover the majority of flows of the SPA what is exhibited by the coverage superior of 87% of the statements by the command ‘npm run alldevtests’ [4]. It is important to emphasize that these tests are the best compromise so they are cheaper, faster, and with medium-high confidence.
The next level of the client-side face in the Test Tetrahedron demands hot integration but before discussing it the same approach should be applied, at least, to the server-side, e.g., each microservice should have automated (I) Unit and (II.1.) Integrated mocked tests, in the sense that quality is asserted (in the developer local environment as well as in the Continuous Integration pipeline of DevOps) from the bottom/up approach in each face of the Test Tetrahedron.
In conclusion, the last two levels of the Test Tetrahedron, namely (II.2.) Integrated hot tests and (III) E2E tests, are focused on tests that depend on an infrastructure, perhaps expensive, to assess the correctness.
In real-world projects, we have seen, for some good reasons, testers and architects preferring to use a dedicated tool for integrated hot tests and E2E tests. On the other hand, we have seen testers and architects choose tools/frameworks without a clear rationale what invariably has led to difficulties in the development and maintenance of the (II) Integrated and (III) E2E tests. For example, in a real-world project, in which the client-side was based on an Angular 7 SPA, there was a large number of (II.1.) Integrated mocked tests (defined using Protractor, integrated into the CI pipeline, and resulting in a coverage together with unit tests of the app greater than 80%), none (II.2.) Integrated hot, and another huge number of (III) E2E tests defined using Java/Selenium to test the same SPA. Therefore, the overlap of the (II) Integrated and (III) E2E tests was tremendous what compromised the total cost. Nonetheless, the (II.1.) Integrated mocked tests in the client-side can be reused, the most critical ones (as a good practice as you climb the tetrahedron the number of tests is fewer, consequently, you must prioritize the critical ones [3]), in the (II.2) Integrated hot even in the (III) E2E tests. The major benefits of such an approach are: (a) to reduce costs of developing and maintenance of such tests, and (b) to allocate the responsibility of the test’s development and maintenance as close as possible to the development of the functionalities using the same set of tools/frameworks. In [4], this is achieved by the commands 'npm run integratedhottests' and ‘npm run fulle2etests’ (using a specific test suite to focus only on critical tests) by changing the target “address” (assumption: data in the target datastores were prepared possibly in CI/CD pipelines).
Wrapping up
Test Tetrahedron can represent the classical element fire, according to Plato, in the quality arena since it can be viewed as a challenge for the well-established Test Pyramid. Recall these concepts can be highly controversial [3] as well as the properties allocated in each vertex of the Test Tetrahedron, namely correctness, automation, coverage, and confidence. Finally, as canary deployments gain acceptance it is clearer that a part of E2E tests can only be executed, with a high degree of confidence, in production what emphasizes the good practice of focusing E2E tests on the critical behavior of a system while major investment should be on Unit and Integrated tests.
References
[1] https://martinfowler.com/bliki/TestPyramid.html
[2] https://martinfowler.com/articles/practical-test-pyramid.html
[3] http://blog.codepipes.com/testing/software-testing-antipatterns.html
[4] https://github.com/romgerale/TestE2E
[5] https://testing.googleblog.com/2017/04/where-do-our-flaky-tests-come-from.html
