Writing Tests - Advanced Topics
- Advanced Assertion Topics
Local Severity Levels
- Test-Resource Management
Error handling for Create, Setup, TearDown, and Destroy
Loading external files
- Manual fixtures (without reflections)
This version of vbUnit introduces severity levels for assertions. This allows you to determine the effect of a failed assertion. In previous versions of vbUnit, a failed assertion would have no effect on subsequent assertions, while an error would exit the current test method and continue in the next one. This default behavior has now been changed such that a failed assertion will exit the current test method. Execution will then continue in the next test method. Hence, a failed assertion is now treated like an error.
If this conflicts with your needs, Don't Panic! The new method IAssert.SetSeverity allows you to change this behavior:
Sub IAssert.SetSeverity(Severity As vbUnitScope, Optional Duration As vbUnitScope = vbUnit_Method)
This method takes two arguments of the the type vbUnitScope:
Public Enum vbUnitScope
vbUnit_Assertion = 0
vbUnit_Method = 1 'this is the default
vbUnit_Fixture = 2
vbunit_Suite = 3
vbUnit_AllTests = 4
The severity level determines the effect of a failed assertion. In vbUnit2, each assertion had severity 0, which means that a failed assertion would have no further effect. The default severity level in vbUnit3 is now 1 (vbUnit_Method). This means that a failed assertion will break out of the current testmethod and execution will continue at the next testmethod. For severity 2 (Fixture), a failed assertion will stop execution in the current fixture, severity 3 will break out of the current suite, and severity 4 will stop all further tests.
For example, if you acquire a complicated resource without which all further tests in a given fixture would be meaningless, set the severity to vbUnit_Fixture before checking if the resource was acquired sucessfully:
'Critical Initialization Test 'in Fixture class TestMyExpensiveResource.cls: Public Sub TestInitialization() m_assert.SetSeverity vbUnit_Fixture 'a failed assertion in 'this method will stop 'testing in this fixture m_assert.Verify m_resource.IsValid End Sub Public Sub TestUseResource() 'resource is valid, we can use it now! '... End Sub
The second, optional parameter of SetSeverity determines the duration of the severity assignment. When execution leaves the scope of the duration, the severity level goes back to the default. The duration level 0 applies only to the next assertion, duration 1 applies to all assertions in the current method that come after the SetSeverity statement, duration 2 applies to all assertions in the current fixture, etc.
For example, to make this version of vbUnit3 behave like vbUnit2, include the following statement in the first test method:
'vbUnit2 Compatibility Public Sub TestVBUnit2Compatibility() m_assert.SetSeverity vbUnit_Assertion, vbUnit_AllTests 'now there can be several failed assertions in a single method ' for all tests in this run ... End Sub
Conversely, to speed up development cycles, make new fixtures the first fixture in your suite and set the severity to high. This will stop testing upon encountering the first problem, and you don't have to run the whole suite every time. This way of testing is very fast during development, but you can still be sure to run all tests for the complete project when the new fixture passes everything.
'Fastest development: Stop testing at first error or failure 'in the TestSuite: Private Function ISuite_Suite() As ITest Dim suite As New TestSuite suite.SuiteName = TypeName(Me) suite.AddFixture New NewestFixture suite.AddFixture New OlderFixture1 suite.AddFixture New OlderFixture2 Set ISuite_Suite = suite End Function 'in the newest fixture: Private Sub IFixture_Setup(assert As IAssert) Set m_assert = assert m_assert.SetSeverity vbUnit_AllTests, vbUnit_AllTests End Sub
The assertion methods Verify, LongsEqual, DoublesEqual, StringsEqual, and VariantsEqual have an aditional optional parameter that can override the current severity setting for this assertion. For example:
' stop all tests if the
resource is invalid
m_assert.Verify m_resource.IsValid, "testResource", vbUnit_AllTests
This is basically a shorthand for setting the severity with a separate call to m_assert.SetSeverity.
In addition to the security levels, IAssert also offers the method
Sub IAssert.StopTesting(scope As vbUnitScope)
This will break execution to the end of the required scope. For example,
m_assert.StopTesting vbUnit_Fixturewill exit the current fixture. Execution will continue at the next fixture.
In most cases, this can be achieved by having assertions fail. The only difference of the StopTesting method is that it won't list any failures in the TestResult.
In most cases, the initialization for each test should be done in IFixture_Setup, and the cleanup in IFixture_TearDown. However, if a test fixture aquires very expensive resources such as database connections that take a long time to create, it may be desirable to do this only once for a fixture, or even just once for the complete test run.
A fixture may implement the optional interface IFixtureFrame, which
defines the methods Create and Destroy. These methods will be
called before and after ALL TestMethods in a given fixture have been called.
Class_Initialize and Class_Terminate are not good places to handle resource management, because these methods are already called when the fixture is added to a suite:
suite.AddFixture New TestVBUnit ' Class_Initialize for TestVBUnit is called here!
After the Fixture has been registered, it is destroyed again. Later, when the test is run, a second instance of the fixture class is created. Therefore, Class_Initialize should not acquire any expensive resources!
Here is the sequence of calls made to a fixture during its lifetime:
' run the actual Tests
[ IFixtureFrame_Create ] ' only called if class implements IFixtureFrame
[ IFixtureFrame_Destroy ] ' only called if class implements IFixtureFrame
Click here to find out how to do initialization and cleanup before and after running several suites.
Each of those methods can throw failed assertions or errors, and the corresponding error or failure will be added to the TestResult. An error or failure in Setup or Teardown will stop all further tests in this fixture. In other words, the severity level (see below) for Create, Setup, Teardown, and Destroy is "vbUnit_Fixture". The following table lists the methods that are being called after an error or failure in each respective fixture method:
|Error or Failure in:||Further methods called:|
|Class_Initialize||none (VB won't call Class_Terminate if there is an error in Class_Initialize)|
|Create||Destroy, Class_Terminate (no tests will be run at all)|
|Setup||Teardown, Destroy, Class_Terminate|
|(run TestMethod)||Teardown, Destroy, Class_Terminate|
The vbUnit3 TestRunner will set the current directory to the location of the compiled
TestDll. All relative file references in your Test Project should be
relative to that location. This works both in debug and in compiled mode, but
obviously the TestDll needs to have been compiled at least once to indicate the
Using relative instead of absolute file references is highly recommended, since it will allow you to move your test folders to a different location or machine much more easily. For example, the vbUnit3 folder can be installed anywhere without breaking the tests, since they only use relative paths.
New for version 3.05.00: If the compiled TestDll is not registered or not present, vbUnit will take the path of the target executable defined in the project settings. This is the file that VB suggests when you compile the project.
In some cases it may be desired to run tests without the use of reflections. This will allow to make the test fixture a private class of a project (Instancing = private), and it will also allow tests to be run on machines where Visual Studio is not installed. Manual fixtures use the interface IManualFixture instead of IFixture.
In addition to Setup and TearDown, IManualFixture has a third method that explicitly binds the test methods. Otherwise, it can be used exactly like the IFixture interface:
Private Sub IManualFixture_GetTestNames(tests