# Testability

* 30-50% of software costs (money/time) taken up by testing
  * Reducing this cost is a highish priority
  * Includes
    * writing automated tests
    * manual testings
    * setup for tests
    * maintaince for testing (servers)
    * redoing testing after fixing defacts from test or bugs
    * Helping QA/testers learn or solve their problems
* Software testability refers to the ease with which software can be made to demonstrate its faults through (typically execution-based) testing
  * testability refers to the probability, assuming that the software has at least one fault, that it will fail on its next test execution
  * a system is testable if it "gives up" its faults easily. If a fault is present in a system, then we want it to fail during testing as quickly as possible.
* Typical model
  * includes
    * inputs to program
    * output of program
    * internal state before/after program has ran
    * Quality attributes ie performance
    * Oracle (tester/test harness) who compares the above outputs with the program specs (expected results) then approves or rejects
* For a system to be properly testable, it must be possible to control each component's inputs (and possibly manipulate its internal state) and then to observe its outputs and internal state
* Testing of code is a special case of validation, which is making sure that an engineered artifact meets the needs of its stakeholders or is suitable for use
* Generally done via test harness,
  * which is specialized software (or in some cases, hardware) designed to exercise the software under test.
  * Exampels
    * a record-and-playback capability for data sent across various interfaces
    * a simulator for an external environment in which a piece of embedded software is tested, or even during production
  * IT can
    * provide assistance in executing the test procedures and recording the output
* Portions of the system or the entire system may be tested.
* Carried out by
  * developers
  * QA
  * Users
* TEsts can be written by
  * developers
  * testing group
  * customers
* The response measures for testability deal with how effective the tests are in discovering faults and how long it takes to perform the tests to some desired level of coverage
*   Testing can be done after feature has been done or during developement(TDD)

    \-

## Testability General Scenario

* Source of stimulus.
  * The testing is performed by
    * unit testers
    * integration testers
    * system testers (on the developing organization side)
    * acceptance testers
    * end users (on the customer side).
  * The source could be human or an automated tester.
* Stimulus.
  * A set of tests is executed due to
    * the completion of a coding increment such as a class layer or service,
    * the completed integration of a subsystem
    * the complete implementation of the whole system
    * the delivery of the system to the customer
* Artifact.
  * A unit of code (corresponding to a module in the architecture)
  * a subsystem
  * the whole system is the artifact being tested.
* Environment.
  * The test can happen at
    * development time
    * compile time
    * deployment time
    * while the system is running (perhaps in routine use)
  * The environment can also include the test harness or test environments in use
* Response.
  * The system can be controlled to perform the desired tests and the results from the test can be observed
* Response measure.
  * Response measures are aimed at representing how easily a system under test "gives up" its faults.
  * Measures might include
    * the effort involved in finding a fault or a particular class of faults,
    * the effort required to test a given percentage of statements
    * the length ofthe longest test chain (a measure of the difficulty of performing the tests)
    * measures of effort to perform the tests
    * measures of effort to actually find faults
    * estimates of the probability of finding additional faults
    * the length of time or amount of effort to prepare the test environment
    * the ease at which the systetn can be brought into a specific state.
    * the reduction in risk of the remaining errors in the systetn can be used.
      * attempt to rate the severity of faults found (or to be found)
      * Not all faults are equal in terms of their possible impact

## Tactics for Testability

* The goal is to allow for easier testing when an increment of software development is completed.

### Control and Observe System State

* Come as a pair
* The simplest form of control and observation is to provide a software component with a set of inputs, let it do its work, and then observe its outputs
* These tactics
  * cause a component to maintain some sort of state information
  * allow testers to assign a value to that state information
  * and/or make that information accessible to testers on demand
* state information might be
  * an operating state
  * the value of some key variable
  * performance load
  * intermediate process steps
  * anything else useful to re-creating component behavior
* Tactic: Specialized interfaces
  * allows you to control or capture variable values for a component either through a test harness or through normal execution
  * Examples
    * A set and get method for important variables, modes, or attributes (methods that might otherwise not be available except for testing purposes)
    * A report method that returns the full state ofthe object
    * A reset method to set the internal state (for example, all the attributes of a class) to a specified internal state
    * A method to tum on verbose output, various levels of event logging, performance instrumentation, or resource monitoring
  * Specialized testing interfaces and methods should be clearly identified or kept separate from the access methods and interfaces for required functionality, so that they can be removed if needed
    * in performance-critical and some safety-critical systems, it is problematic to field different code than that which was tested. If you remove the test code, how will you know the code you field has the same behavior, particularly the same timing behavior, as the code you tested?
  * Record/playback.
    * The state that caused a fault is often difficult to re-create.
    * Recording the state when it crosses an interface allows that state to be used to "play the system back" and to recreate the fault.
    * Record/playback refers to both capturing information crossing an interface and using it as input for further testing
  * Localize state storage.
    * To start a system, subsystem, or module in an arbitrary state for a test, it is most convenient if that state is stored in a single place.
      * By contrast, if the state is buried or distributed, this becomes difficult if not impossible.
    * The state can be fine-grained, even bit level, or coarse-grained to represent broad abstractions or overall operational modes.
    * The choice of granularity depends on how the states will be used in testing.
    * A convenient way to "externalize" state storage (that is, to make it able to be manipulated through interface features) is to use a state machine (or state machine object) as the mechanism to track and report current state.
  * Abstract data sources.
    * Similar to controlling a program's state, easily controlling its input data makes it easier to test.
    * Abstracting the interfaces lets you substitute test data more easily.
      * For example, if you have a database of customer transactions, you could design your architecture so that it is easy to point your test system at other test databases, or possibly even to files of test data instead, without having to change your functional code.
  * Sandbox.
    * "Sandboxing" refers to isolating an instance of the system from the real world to enable experimentation that is unconstrained by the worry about having to undo the consequences of the experiment.
    * Testing is helped by the ability to operate the system in such a way that it has no permanent consequences, or so that any consequences can be rolled back.
    * This can be used for
      * scenario analysis
      * training
      * simulation.
    * Example, the Spring framework comes with a set of test utilities that support this.
    * Tests are run as a "transaction," which is rolled back at the end.
      * Thus removing any state changes
    * A common form of sandboxing is to virtualize resources.
      * Testing a system often involves interacting with resources whose behavior is outside the control of the system.
      * Using a sandbox, you can build a version ofthe resource whose behavior is under your control
        * ie control the clock, as waiting for the action for the resource undert test to occur is a waste of time
          * Can allow the system (or components) to run at faster than wall-clock time, and to allow the system (or components) to be tested at critical time boundaries (ie daylight savings)
      * Similar virtualizations could be done for other resources, such as memory, battery, network, and so on
      * Forms of virtualisation
        * Stubs
        * mocks
        * dependency injection
  * Executable assertions.
    * Using this tactic, assertions are (usually) hand-coded and placed at desired locations to indicate when and where a program is in a faulty state.
    * The assertions are often designed to check that data values satisfy specified constraints.
    * Assertions are defined in terms of specific data declarations, and they must be placed where the data values are referenced or modified.
    * Assertions can be expressed as pre- and post-conditions for each method and also as class-level invariants.
    * This results in increasing observability, when an assertion is flagged as having failed.
    * Assertions systematically inserted where data values change can be seen as a manual way to produce an "extended" type.
      * Essentially, the user is annotating a type with additional checking code.
      * Any time an object of that type is modified, the checking code is automatically executed, and warnings are generated if any conditions are violated. To the extent that the assertions cover the test cases, they effectively embed the test oracle in the code assuming the assertions are correct and correctly coded.
* All of these tactics add capability or abstraction to the software that (were we not interested in testing) otherwise would not be there
* Tactics for adding abstractions and capabilities for testing
  * Tactic: Component replacement
    * which simply swaps the implementation of a component with adifferent implementation that (in the case of testability) has features that facilitate testing.
    * Component replacement is often accomplished in a system's build scripts
    * Using DI framework to replace component with a testing componement (ie stub, with same interface but added testing features)
  * Tactic: Preprocessor macros
    * that, when activated, expand to state-reporting code or activate probe statements that return or display information, or return control to a testing console
  * Tactic: Aspects (in aspect-oriented programs)
    * that handle the cross-cutting concern of how state is reported

### Limit Complexity

* by the definition of complexity, its operating state space is very large and (all else being equal) it is more difficult to re-create an exact state in a large state space than to do so in a small state space.
* testing is not just about making the software fail but **about finding the fault that caused the failure so that it can be removed**, we are often concerned with making behavior repeatable
* Tactic: Limit structural Complexity
  * includes
    * avoiding or resolving cyclic dependencies between components
    * isolating and encapsulating dependencies on the external environment
    * reducing dependencies between components in general
      * for example, reduce the number of external accesses to a module's public data
  * In object-oriented systems, you can simplify the inheritance hierarchy:
    * Limit the number of classes from which a class is derived,
    * or the number of classes derived from a class.
    * Limit the depth ofthe inheritance tree, and the number of children of a class.
    * Limit polymorphism and dynamic calls.
  * One structural metric that has been shown empirically to correlate to testability is called the response of a class.
    * The response of class C is a count of the number of methods of C plus the number of methods of other classes that are invoked by the methods of C.
    * Keeping this metric low can increase testability.
  * Having high cohesion, loose coupling, and separation of concerns all modifiability tactics can also help with testability.
    * They are a form of limiting the complexity of the architectural elements by giving each element a focused task with limited interaction with other elements.
    * Separation of concerns can help achieve controllability and observability
      * as well as reducing the size of the overall program's state space
    * Controllability is critical to making testing tractable,
      * A component that can act independently of others is more readily controllable
        * With high coupling among classes it is typically more difficult to control the class under test, thus reducing testability
        * If user interface capabilities are entwined with basic functions it will be more difficult to test each function
  * systems that require complete data consistency at all times are often more complex than those that do not.
    * If your requirements allow it, consider building your system under the "eventual consistency" model,
      * where sooner or later (but maybe not right now) your data will reach a consistent state.
    * This often makes system design simpler, and therefore easier to test
  * some architectural styles lend themselves to testability.
    * In a layered style, you can test lower layers first, then test higher layers with confidence in the lower layers.
* Tactic: Limit nondeterminism
  * The counterpart to limiting structural complexity is limiting behavioral complexity
  * Nondeterministic systems are harder to test than deterministic systems.
  * This tactic involves finding all the sources of nondeterminism, such as
    * unconstrained parallelism, and weeding them out as much as possible.
  * Some sources of nondeterminism are unavoidable
    * for instance, in multi-threaded systems that respond to unpredictable events but for such systems, other tactics (such as record/playback) are available.

## A Design Checklist for Testability

* Allocation of Responsibilities
  * Determine which system responsibilities are most crrtical and hence need to be most thoroughly tested.
  * Ensure that additional system responsibilities have been allocated to do the following:
    * Execute test sune and capture results (external test or self-test)
    * Capture (log) the activity that resulted in a fault or that resulted in unexpected (perhaps emergent) behavior that was not necessarily a fault
    * Control and observe relevant system state for testing
  * Make sure the allocation of functionality provides high cohesion, low coupling, strong separation of concerns, and low structural complexity
* Coordination Model
  * Ensure the system's coordination and communication mechanisms:
    * Support the execution of a test suite and capture the results within a system or between systems
    * Support capturing activity that resulted in a fault within a system or between systems
    * Support injection and moni1oring of state into the communication channels for use in testing, within a system or between systems
    * Do not introduce needless nondeterminism
* Data Model
  * Determine the major data abstractions that must be tested to ensure the correct operation of the system.
    * Ensure that It is possible to capture the values of instances of these data abstractions
    * Ensure that the values of instances of these data abstractions can be set when state is injected into the system, so that system state leading to a fault may be re-created
    * Ensure that the creation, initialization, persistence, manipulation, translation, and destruction of instances of these data abstractions can be exercised and captured
* Mapping among Architectural Elements
  * Determine how to test the possible mappings of architectural elements (especially mapplngs of processes to processors, threads to processes, and modules to components) so that the desired test response is achieved and potential race conditions identified.
  * determine whether it is possible to test for illegal mappings of architectural' elements
* Resource Management
  * Ensure there are sufficient resources available to execute a test suite and capture the results.
  * Ensure that your test environment is representative of (or better yet. identlcal to) the environment in which the system will run.
  * Ensure that the system provides the means to do the following:
    * Test resource limits
    * Capture detailed resource usage for analysis in the event of a failure
    * Inject new resource timits Into the system for the purposes of testing
    * Provide virtualized resources for testing
* Binding Time
  * Ensure that components that are bound later than compile time can be tested in the late-bound context.
  * Ensure that late bindings can be captured in the event of a failure, so that you can re-create the system's state leading to the failure.
  * Ensure that the full range of binding possibilities can be tested
* Choice of Technology
  * Determine what technologies are available to help achieve the testability scenarios that apply to your architecture.
    * Are technologies available to help with regression testing. fault injection, recording and playback, and so on?
  * Determine how testable the technologies are that you have chosen (or are considering choosing in the future) and ensure that your chosen technologies support the level of testing appropriate for your system.
    * For example, if your chosen technologies do not make it posstble to inject state, It may be difficult to re-create fault scenarios.

## Issues with testing

* Test Data
  * is how to create large, consistent and useful test data sets.
  * particularly for
    * integration testing (that is, testing a number of components to confirm that they work together correctly) and
    * performance testing (confirming that the system meets it requirements for throughput, latency, and response time).
  * For unit tests, and usually for user acceptance tests, the test data is typically created by hand.
  * This data has to be sufficiently varied to make testing worthwhile
  * It has to conform to all the referential integrity rules and other constraints of your data model
  * you need to be able to calculate and specify the expected results of the tests.
  * Solutions
    * you either write a system to generate your test data
      * Ideal
      * in full control
      * cover edge cases
      * High effort, high upfront costs
    * you capture a representative data set from the production environment and anonymize it as necessary.
      * Anonymizing test data involves removing any sensitive information
      * Capturing data from the live environment is easier
      * dont know what data you may get, so not all cases are covered
* Test Automation
  * In practice it is not possible to test large systems by hand because of the number oftests, their complexity, and the amount of checking of results that's required
    * In the ideal world, you create a test automation framework to do this automatically, which you feed with test data, and set running every night, or even run every time you check in something
      * the continuous integration model
  * A test automation framework
    * costs
    * Should be budgeted alongside prod budget
    * Should not be thought of as part of prod and can be absorbed into it
  * how the framework will invoke functions on the system under test,
    * particularly for testing user interfaces, which is almost without exception a nightmare
      * use of selenium
      * can be hard to test, and maintain these tests
