Table of contents
My team is using testify as the testing framework for Go. We noticed that many methods provided by testify/suite can be easily achieved by the built-in testing package. For example, using Cleanup can do the same of AfterTest provided by testify/suite. It makes me wondering if we can just use Cleanup to replace as much methods from third-party packages as possible.
TL;DR
Cleanup can replace methods that are not relevant to subtests, which are AfterTest, TearDownAllSuite and TearDownTestSuite.
Cleanup can’t replace TearDownSubTest because there is a risk that a subtest pollute its parent test.
The experiment
I use a global variable externalResource as the external resource. It’s value is 1.
When preparing a subtest at SetupSubTest, externalResource is changed to 2. At the same time, a cleanup function is registered to restore externalResource to 1.
I expect that externalResource is set to 2 before the subtest and restored to 1 once the subtest is finished. So, from the point of parent test, externalResource is always 1. From the point of subtest, externalResource is always 2.
The code is
var externalResource = 1
type TestUnexpectedCleanupOrder struct {
suite.Suite
}
func (s *TestUnexpectedCleanupOrder) TearDownTest() {
fmt.Println("TearDownTest")
}
func (s *TestUnexpectedCleanupOrder) SetupSubTest() {
externalResource = 2
fmt.Printf("SetupSubTest. The externalResoure is: %d\\n", externalResource)
s.T().Cleanup(func() {
externalResource = 1
fmt.Printf("SetupSubTest. The externalResoure is: %d\\n", externalResource)
})
}
// Test case
func (s *TestUnexpectedCleanupOrder) TestCaseA() {
fmt.Printf("Start TestCase A. The externalResoure is: %d\\n", externalResource)
s.Run("Sub-TestCase A", func() {
fmt.Printf("Finish Sub-TestCase A. The externalResoure is: %d\\n", externalResource)
})
fmt.Printf("Finish TestCase A. The externalResource is: %d\\n", externalResource)
}
The result
The output of the above code is (with line number)
1 Start TestCase A. The externalResoure is: 1
2 SetupSubTest. The externalResoure is: 2
3 Finish Sub-TestCase A. The externalResoure is: 2
4 Back to TestCase A. The externalResource is: 2
5 TearDownTest
6 Cleanup subTest. The externalResoure is: 1
Notice line 4 and line 6. The externalResource is still 2 when the subtest is finished and the parent test is resuming! externalResource is not cleaned until the parent test is finished.
According to the documentation of Cleanup:
Cleanup registers a function to be called when the test (or subtest) and all its subtests complete. Cleanup functions will be called in last added, first called order.
It only guarantee that a registered function is called after something.
It does NOT guarantee when a registered function is called before something.
The solution
If we add a TearDownSubTest to reset externalResource, then its value is cleaned as we need
func (s *TestUnexpectedCleanupOrder) TearDownSubTest() {
externalResource = 1
fmt.Printf("\\tTeardown subTest. The externalResoure is: %d\\n", externalResource)
}
Start TestCase A. The externalResoure is: 1
SetupSubTest. The externalResoure is: 2
Finish Sub-TestCase A. The externalResoure is: 2
Teardown subTest. The externalResoure is: 1
Back to TestCase A. The externalResource is: 1
TearDownTest
Cleanup subTest. The externalResoure is: 1
Summary
We can use Cleanup to clean up resources for normal tests. However, we need to use TearDownSubTest to clean the resource held by subtests.
The source code
https://github.com/xuanyuwang/testify-examples/tree/main/unexpected-cleanup-order