A practice common among successful companies may still be a poor basis for adoption
When many successful companies use the same sales practice, copying it can seem like a route to more contracts. Yet if companies that used the practice and missed their targets are absent from the research, you cannot calculate the success rate among adopters.
A practice being common among successful companies and a practice causing success are different questions. Success stories can teach you how sales activities were carried out. Assessing effectiveness also requires unsuccessful cases using the same method and results from companies that did not use it.
Survivorship bias occurs when surviving companies or successful cases remain visible while other outcomes are missing, distorting judgment. Companies can disappear through closure, and researchers can also omit failure by selecting only successful companies for interviews. (Denrell on undersampling failure and management beliefs)
What missing failures can make us misunderstand
Denrell's 2003 paper shows that risky practices can appear positively associated with performance among surviving companies even when they have no association with performance in the full population. This is an argument about how selection changes inference, rather than a measured percentage increase in sales following a particular sales intervention. (Abstract of the 2003 paper)
Why does information disappear? A Stanford University article featuring an interview with the researcher explains that unsuccessful companies may close or change their approach, leaving less information about unsuccessful strategies. When a practice occurs in both success and failure but the latter is less visible, it can become known as a successful method. (Stanford's interview with Denrell)
This problem can arise even when every success story is accurate. An accurate account of one company's experience does not establish that the cases available to read represent all companies.
Two thirds of successful companies can adopt a practice with a 20% success rate
The following figures are hypothetical and illustrate different proportions. They are not company outcomes observed in the research.
Suppose you have complete records for 100 companies that began sales activities at the same time. You know whether each adopted a particular practice at the outset and whether it won at least ten new contracts in the following six months. Here, success means ten or more contracts; below target means fewer than ten. This classification concerns contracts in that period, rather than whether the company closed.
| Sales practice at the outset | Companies | At least ten new contracts | Fewer than ten new contracts | Success rate within each group |
|---|
| Adopted | 80 | 16 | 64 | 20% |
| Did not adopt | 20 | 8 | 12 | 40% |
| Total | 100 | 24 | 76 | 24% |
Interview only the 24 successful companies, and 16 of them adopted the practice. Dividing 16 by 24 lets you report that about 66.7% of successful companies adopted it.
Among all 80 adopters, however, only 16 succeeded. Dividing 16 by 80 gives a success rate of 20%. Among the 20 companies that did not adopt, eight succeeded, giving 40%. In this hypothetical dataset, adopters' success rate is 20 percentage points lower than that of companies that did not adopt.
The adoption share among successful companies uses successful companies as its denominator. The success rate among adopters uses adopters as its denominator. If you want the success rate after adoption, counting only the 24 successful companies cannot answer the question. You also need records covering the 76 companies that fell below target.
Comparing success rates still does not establish the effect of the practice
The 20% and 40% figures alone do not establish that the practice reduced success. Perhaps companies struggling to acquire customers were more likely to adopt, while companies already receiving many referrals did not. Their sales circumstances would then differ at the outset.
The table therefore describes the association between adoption and success in the hypothetical 100 companies. It does not measure what would happen to the same company with and without the practice.
As a practical application, if comparison within your sales organization is feasible, you could randomly assign teams to adopt or not adopt during the same period. Markets served, sales resources, and teams sharing the practice can also affect results, so the assignment unit and comparison need to fit the work. This is a proposal in this article, rather than a sales procedure tested in the cited paper.
Adding failure stories does not automatically provide a success rate
Collecting ten success stories and ten failure stories does not establish a 50% success rate if someone deliberately selected those twenty cases. Publishing equal numbers of each is different from knowing the outcomes of all companies that tried the practice.
For data intended to estimate success rates, one approach is to define company inclusion criteria and the evaluation period before outcomes are known, then retain results for companies that stop using the practice. If closure or discontinuation prevents follow-up, report how many outcomes are missing. A missing record alone does not justify counting a company as either successful or below target.
When reading public case studies, complete company outcomes may be unavailable. Use the cases to understand how a practice was implemented or how a failure developed. Avoid converting the proportions of published cases into a probability of success after adoption.
Learn methods from success stories without treating them as success rates
Success stories can reveal proposals and work that companies carried out. When considering adoption, ask how many companies using that method succeeded and how companies not using it performed. Shared characteristics of successful companies alone determine neither the success rate after adoption nor the effect of the practice itself.
When publishing customer case studies on your website, explain which customer changed what, over which period, and what result followed. This helps readers understand the case. Describe one customer's result as that customer's experience rather than the average outcome for all users.
SpreadSite is a no-code website builder that lets you create, publish, and update a website from Google Sheets using ready-made templates.
To create a website explaining customer use cases and implementation stories in concrete terms, give SpreadSite a try!
Build a website with SpreadSite →
