Techniques and tools
Multivariate Testing

Multivariate Testing is a planned approach for determining cause and effect relationships. It can be used to optimise any business process, which has controllable inputs and measurable outputs. It is based on a discipline called Design of Experiments (DOE) which has traditionally been used to optimise manufacturing and industrial processes. However, it has numerous applications in the business, marketing and market research fields.

It is particularly useful where the outputs we wish to optimise depend on combinations of factors. The main tenet of the process is to assess the impact across all of these factors simultaneously, rather than one at a time. It achieves this through using experimental design techniques, to maximise the information obtained whilst minimising the number of trials or runs required to fully understand the process.

MVT is often used when it is impractical or unrealistic to run research exercises with individual decision makers. Instead of administering choice or preference exercises to decision makers, as is the case with traditional Conjoint research, we experiment with real-world processes by changing specific combinations of input factors and measuring how these changes affect the outputs of interest. MVT can be used to optimise sales, response rates, conversion rates, response times, waiting times, number of inbound customer contacts, satisfaction, number of errors, number of repeat visits or indeed any other numerical quantity. It enables an excellent understanding of which combination of factors influence an output - placing an emphasis on understanding interactions between factors as well as their “main effects” (i.e. impact regardless of other factors). It achieves this using generalised linear modelling, the family of techniques of which multiple regression and ANOVA members. By carefully designing a series of trials or runs, individual effects and interactions can be precisely identified using only a minimal number of factor combinations.

The input factors which are varied can be a mixture of qualitative factors (e.g. presence or absence of a feature) and/or continuous quantitative factors (e.g. time). Different experimental design techniques are used depending on the number and type of factors, the process under investigation, objectives and practical considerations. Designs are available which allow up to 31 factors to be assessed, though most applications of MVT investigate fewer factors in this.

Some applications of MVT are outlined below[1] :

Direct Marketing / Advertising

• A travel club wanted to increase response to direct mail. It tested 17 factors in 20 mail pieces, including: copy on envelope and cover letter, the offer, graphics, and even fonts and logos. They found that short and simple works best, text on envelope helped, but a free offer made no difference. An extra insert actually lowered response. The MVT increase predicted response from 0.3 to 0.5%, worth $20 to $40 million in annual revenue. (Multivariable Testing Methods in Marketing, Gordon H. Bell)



• An MVT investigated three factors (commercial length, repetition and delay before recall) on the average number of products recalled by 20 subjects viewing television commercials. (Journal of marketing research, 25, 1988, pp.72-88: “Recognition versus Recall as Measures of Television Commercial Forgetting")

Billing/Finance

• A large US telephone company did a two level factorial to design a better telephone bill. They varied more than a dozen factors, including: colour, fonts, shading, alignment and orientation. The new layout garnered a 78% preference rating versus 48% of the old bill. Savings of $2 million in postage will result from the more efficient bill. (Michael Berry, Southwestern Bell, Austin, Texas)



• A large company reduced their receivables from 200 to only 44 days, generating a large cash flow in the process. They studied four factors: billing with the shipment on a separate invoice, automation, follow up by letter or telephone, contract or in-house being service. They ran only eight of the combinations. Two of the factors were highly significant. (Experimental Design, Frigon, Matthews, J. Wiley, 199, p.266)

Marketing/Sales Mix

• A major shoe retailer used MVT to simultaneously test sales techniques, advertising, separation by product colour, discounts and display configurations. They found a combination that pushed trainer sales up 33%. (Forbes, March 11, 1996: The New Mantra: MVT”)



• A study showed that the choice of high cost brands is enhanced when the consumer is given little time to choose. This interaction could only be revealed via multivariate testing. (Marketing letters, 6:4, 1995: "the effect of time pressure on the choice between brands that differ in quality, price and product features")



• A sales team wanted to improve its success rate using percent of successful closures to measure performance. They did an MVT on the following factors: attire (suit or casual), number of salespeople (one or two), presentation (high-pressure or low) and brochure (old or new). (From talk by Rip Stauffer on “Six Sigma in a Non-Manufacturing Environment" at 49th annual Minnesota quality conference, 2002)

Customer Service

• A major telecommunications provider made use of screening MVTs to reduce network outage duration time; service order processing time; response times to customers and increase sales for call centres. (From talk by Harry Rever on "the Application of Large Screening Design of Experiments in the Service Industry to Improve Key Metrics of the business" at ASQ's 2004 Six Sigma conference)



• As a more general example, there are many applications of MVT to optimise the performance of call centres, help desks and switchboards. Variables to optimise include reducing the number of unwanted inbound calls, increasing the proportion of problems resolved in one call, improving response times, increasing new sales, minimising call length, decreasing waiting times, increasing customer satisfaction. There are whole variety of potential factors to test, including type of training provided to agents, configurations of call screening technology, shift patterns, type of prerecorded message/instructions provided to customers, type of cross selling techniques and instructions provided to agents - to name but a few.

Please contact Gary Bennett for further information (garyb@logitresearch.com).

[1] I am indebted to Mark Anderson of Stat Ease for providing these examples

< Back to Techniques and Tools Home Page