To better manage water resources, it is useful to understand how people value water quality. One way to assess this value is to estimate (WTP) for improved water quality. Stated preference (SP) surveys, which use experiments to quantify WTP, are useful in estimating such values because they are the only method available to estimate non-use values (values for quality that are unrelated to observable behaviors). Our research estimates values for freshwater ecosystem services in Michigan using a non-market valuation survey that asks respondents whether they would vote for a proposed change in water quality at a stated cost.
An important step in SP survey design is choosing an appropriate collection method (Champ, 2017; Johnston et al., 2017). The traditionally preferred methods rely on probability sampling, such as address-based sampling (ABS) or random-digital dialing (RDD). Probability sampling ensures that every member of the population of interest has a known probability of being selected to participate. Probability samples are preferred since they reduce the risk of systematic bias related to representation (Baker et al., 2010). While probability sampling methods are the gold standard for survey sampling, they are often cost prohibitive. Thus, in the internet era, non-probability online samples have seen rising predominance due to their speed and cost-effectiveness. Non-probability samples are different from probability samples because not everyone in the population has a known and equal chance of being selected. Most non-probability online samples use opt-in panels, where members choose to participate in the panel and are recruited online. While non-probability online samples offer advantages such as rapid collection times and lower costs, they are criticized for their potential biases that may not be mitigated by balancing samples to “represent” population demographics, such as through propensity weighting (Baker et al., 2010).
Understanding trade-offs related to non-probability samples is particularly pertinent to SP surveys, which are often complex and costly. Similarly, understanding differences in SP surveys and results due to sampling differences is an important part of best practices for SP (Johnston et al., 2017) and contributes to our understanding of valuation validity (Bishop & Boyle, 2019). Some survey literature has found that non-probability samples are less accurate and more variable in their accuracy than probability samples (Yeager et al., 2011). Although a recent review of best practices for SP recommends probability sampling, it also notes that data collection mode may not considerably affect results citing several SP studies with mixed findings (Johnston et al., 2017). As methods change and access and familiarity with the Internet increases, continuing research is important since bias between non-probability online samples and traditional methods may change.
Given these tradeoffs, the aim of this research is to investigate how sociodemographic, attitudinal, and WTP values from a SP survey compare across probability and non-probability samples. We build on previous literature by providing evidence of the extent to which responses gathered via non-probability sampling differ from those gathered via a representative sample of the population. Furthermore, this research specifically focuses on environmental SP research which is underrepresented in the literature comparing probability and non-probability internet samples. The SP survey was implemented for three sample types: one probability sample and two nonprobability samples. The probability sample is an address-based sample from the USPS (United State Postal Service) postal delivery sequence file and used a push-to-web design with mailed invitations to visit a website (Dillman, 2017). The two non-probability online samples are from the opt-in panels MTurk and Qualtrics.1 MTurk is an Amazon web-service where workers complete tasks, such as surveys, for small payments. Qualtrics is an opt-in panel using a range of proprietary methods and incentives. The SP survey used a single binary referendum contingent valuation question with respondents voting on a water quality change at a cost to their household. To understand differences in economic values across samples, we compared results of logit models that relate the referendum vote to cost and water quality indices. To test for differences across survey sources, we compared parameters, marginal willingness to pay (MWTP), and total WTP (TWTP) for a range of non-marginal changes. Overall, we have mixed findings, but generally conclude that the non-probability methods generate different valuation results than the probability-based sample, with the majority of the differences stemming from the MTurk sample.
