Satisfaction Among Respondents May Differ From Satisfaction Among All Customers
If 90% of customers who answer a survey say they are satisfied, it does not follow that 90% of all customers are satisfied. The satisfaction of customers who did not answer remains unknown. Decisions about service improvements require information about who received the question and how many answered, alongside the percentage satisfied.
When respondents and nonrespondents differ in the satisfaction being measured, results based only on respondents differ from the value for all customers. This problem is called nonresponse bias. The American Association for Public Opinion Research, or AAPOR, explains that response rates alone cannot establish whether nonresponse error exists or how large it is. (Standard Definitions)
Distinguish the evaluations of customers who answered from estimates about all customers.
Eighteen Satisfied Respondents Out of Twenty Do Not Establish the Rate for One Hundred Customers
The following is a hypothetical calculation. A service sends a question to all 100 customers who used it during a particular month, and the invitation reaches everyone. Each customer can answer once. Twenty answer the satisfaction question, with 18 choosing satisfied and two choosing dissatisfied. These are the only two response options; the other 80 customers do not answer.
The satisfied share among respondents is 18 ÷ 20 = 90%. The response rate among the 100 invited customers is 20 ÷ 100 = 20%. These percentages use different denominators.
Even with those same responses, satisfaction across all customers depends on which of the following assumptions holds for the other 80 people.
| Assumption about nonrespondents | Satisfied respondents | Satisfied nonrespondents | Satisfied share of all 100 customers |
|---|
| 90% of nonrespondents are satisfied too | 18 | 72 | 90% |
| 40% of nonrespondents are satisfied | 18 | 32 | 50% |
The first row gives (18 + 72) ÷ 100 = 90%; the second gives (18 + 32) ÷ 100 = 50%. The same 18 observed satisfied customers produce different overall percentages under different assumptions about nonrespondents. Neither 90% nor 50% has been established as the overall rate by this survey alone.
You also cannot conclude that dissatisfied customers declined to answer. Nonrespondents could be satisfied in the same proportion as respondents. An absent response differs from a response of dissatisfied.
More Responses and Representation of All Customers Are Different Questions
Efforts to collect more responses are worthwhile. However, an increase in total responses alone does not reveal the satisfaction difference between respondents and nonrespondents.
Suppose a survey of the same 100 customers instead receives 80 answers, with 72 choosing satisfied. The response rate is 80%, and satisfaction among respondents is 90%. If all 20 nonrespondents are dissatisfied, overall satisfaction is 72%; if all are satisfied, it is 92%. Fewer customers have unknown satisfaction than in the example with 20 responses, yet overall satisfaction is still not established as 90%.
When collecting feedback, send the same question to customers from the same usage period rather than requesting answers only from satisfied customers. For example, surveying only customers who renew the following month excludes customers who canceled after using the service. Before considering why invited people did not answer, consider who received an invitation at all.
Investigate Missing Feedback Through Follow-up Questions and Usage Records
Estimating satisfaction across all customers requires more information about people who have not answered. One practical approach is to resend the same question to nonrespondents and request answers through an existing contact method. AAPOR's guidance also lists repeated contact and contact through different methods among ways to increase participation. (Best Practices for Survey Research)
Calculate satisfaction among these additional respondents separately from the initial respondents. A lower satisfied share among later respondents shows that the additional customers answering are less often satisfied than the initial respondents. Customers who still have not answered remain unknown, however. The later respondents' percentage cannot automatically be assigned to every nonrespondent.
Customer usage records can provide further clues. For customers from the same month, count invitations and responses separately for those who renewed and those who canceled. A lower response rate among customers who canceled indicates that their feedback is underrepresented. Cancellation may also reflect price or the end of a customer's need, so do not substitute cancellation for a dissatisfied response.
Changing question wording or the response method can change answers even for the same customers. To assess changes from the previous month, keep wording, answer options, invitation timing, and methods consistent. AAPOR likewise recommends keeping question wording and survey methodology as similar as possible for comparisons over time. (Best Practices for Survey Research)
Accompany Satisfaction Percentages With Response Counts and the Customers Invited
A 90% satisfied share among respondents does not establish a 90% satisfied share among all customers. In the initial example, the observations are that 100 people were invited, 20 answered, and 18 selected satisfied. Estimating the overall evaluation requires information about missing customers, including through further questions to nonrespondents.
When publishing survey results on a website, include both the number choosing satisfied and the number answering that question. Include the service usage period covered, invitation count, question wording, and response options so readers can understand which customers and answers produced the 90%. This distinguishes observed feedback from estimates about all customers.
SpreadSite is a no-code website builder that lets you create, publish, and update a website from Google Sheets using ready-made templates.
To present your service alongside customer survey results and the methods behind them, give SpreadSite a try!
Build a website with SpreadSite →
