6.3.3 How the main product testing methods are applied?

Depending on where you are in the design process, there are three types of usability tests to choose from. These are the exploratory, assessment, and validation tests. A fourth type, the comparison test, can be used at any point in the design life cycle.1

Please stop and think: Does product testing play a role in your company’s strategy? If yes, what methods you have used so far? Do you know about the different methods?

The following statements are based on Innosupport 6.2 component.2

Exploratory tests

Carried out early in the development process during the fuzzy front end, when the problem is still being defined and potential solutions are being considered, preferably once the development team has a good understanding of the user profile and cus-tomer needs. The objective of the exploratory test is to examine and explore the poten-tial of preliminary design concepts and answer some basic questions, including:

  • What do the users think about using the concept?
  • Does the basic functionality have value to the user?
  • Is the user interface appropriate and operable?
  • Are our assumptions about customer requirements correct?
  • Have we misunderstood any requirements?

Assessment tests

Whilst the exploratory test aims to explore the appropriateness of a number of poten-tially competing solutions, the assessment test digs into more detail with a preferred solution at a slightly later stage of development. The main aim of an assessment test is to ensure that assumptions remain relevant and that more detailed and specific design choices are appropriate. The assessment test will tend to focus on the usability or level of functionality offered and in some cases, may be appropriate for evaluating early lev-els of performance. Assuming that the right concept has been chosen, then the as-sessment test aims to ensure that it has been implemented effectively and answer more detailed questions, such as:

  • Is the concept usable?
  • Does the concept satisfy all user needs?
  • How does the user use the product and could it be more effective?
  • How will it be assembled and tested and could this be achieved in a better way?
  • Can the user complete all tasks as intended?

Please check: Are you able to answer the questions above for your products? If not, what could you do in order to get the required information?

Validation tests

The validation test is normally conducted late in the development process to ensure that all of the product design goals have been met. This may include usability, per-formance, reliability, maintainability, assembly methods and robustness. Validation tests normally aim to evaluate actual functionality and performance, as is expected in the production version and so activities should be performed in full and not simply walked through.
It is probable that the validation test is the first opportunity to evaluate all of the com-ponent elements of the product together, although elements may have been tested in-dividually already. Thus, the product should be as near to representing the final item as possible, including packaging, documentation and production processes. Also in-cluded within validation tests will be any formal evaluation required for certification, safety or legislative purposes. Compared to an assessment test, there is a much greater emphasis on experimental rigour and consistency. It may be preferable for evaluation to be carried out independently from the design team, but with team input on developing standards and measurement criteria.

Comparison tests

This test compares the users’ reactions to multiple examples of a tool or implementa-tion. This test can be used at any stage in the design process to compare radically dif-ferent designs or implementations against each other. This test is used to determine which design is easiest to use and what are the advantages and disadvantages be-tween designs. This test can be an exploratory test where multiple designs are com-pared qualitatively. The usual result is an improved product which combines the best of many different ideas. The best results normally come from comparing examples of wildly differing implementations.

Product testing case study

A combination of validation and comparison tests from a Polish SME

Acoustic Testing of Automotive Exhaust Systems

ASMET was founded in 1989, is located in Czersk, Poland. The company has grown rapidly from its origins as an automotive spare parts supplier to become a major Euro-pean manufacturer of high quality automotive exhaust systems. The product range is wide with exhaust systems for over 600 different models in current production. To design and develop an exhaust system, ASMET buys at least two complete units from the original equipment manufacturer.
The units are carefully measured and CAD drawings produced. The aim is to achieve as close a shape to the original as possible.  The next step is that acoustic measure-ments are made in the anechoic room. The unit with the best noise parameters is cut to reveal its construction and the interior dimensions are again measured and CAD drawings made.

Prototypes of the new design are made and these are thoroughly tested, both for their physical dimensions but also their acoustic properties. Subject to these being within the specified limits, the new model is released for production. They can also develop their own designs based upon their years of experience.3

1 Guide to Planning and Conducting Usability Tests
2 InnoSupport: Supporting Innovations in SME. 6.2 Rapid Prototyping and Product Testing Strategies. 2005
3 ASMET Case study: www.bksv.com/doc/ba0722.pdf