I’m a Ph.D. health economist, now retired. I have significant professional experience in both conducting and analyzing surveys. I’m only bringing that up so that you’ll know that I have the bona fides to discuss what does and doesn’t work in surveys.
And I don’t think it’s very smart to do what the title of this posting says. That is, to use a low-quality survey to make decisions affecting roughly five billion dollars’ worth of residential property in the Town of Vienna.
But that’s exactly what the Town of Vienna is doing.
Worse, for a few hundred dollar’s worth of post cards and stamps, they could easily check the results of their low-quality survey.
But I’m betting that they won’t do that.
Read on if you want to know further details. First, I’ll describe the Town’s survey and the risk they take in accepting the results. Then I’ll describe how, for a few hundred dollars, they could avoid that risk entirely. And why they really, really ought to want to do that.
No screeds this time, other than to point out that I’ve been saying the same thing for years now. And that we used to have people on Town Council who understood the basics of surveys. But apparently we don’t any more. Or they aren’t speaking up. And that’s a pity.
The Town’s survey and what can go wrong.
The Town is seeking input from the citizens regarding the rewrite of the entire zoning code. One of those rewrites involves allowing greater lot coverage in the residential zones, as proposed by Councilmember Patel. The Town conducted a survey to gauge citizen reaction to that, the results of which you can see at this link. This is the proposal that I wrote about yesterday, as opening the door to substantially larger houses in Vienna.
(And, to be clear, of course the larger-house issue was not mentioned on the Town’s survey, but that’s not the issue being addressed in this post. I noted that as initially reported, the Patel proposal would include driveways. That was omitted from the key survey question. So the Town more-or-less sidestepped the entire issue of the impact of the Patel proposal on increasing house size in Vienna.)
The heart of the issue is that this isn’t a random-sample survey. It was open to anyone who wanted to take it. This sort of non-random-sample internet survey is a poor tool in the best of circumstances due to the likelihood of selection bias. The self-selection means that people who knew about and bothered to take the survey might be quite different from the average Vienna resident. And in cases where money is involved, and vested interests are around, the chances for biasing the results by intentionally biasing the respondents go up exponentially.
It’s not like that’s some sort of shocking news. I’ve been posting about survey methods here for years. (E.g. ,Post #462, Post #415, and many others). In fact, the prior writeups pretty much cover the key points about what can go wrong with a self-selected survey, emphasis mine:
Suggestion 2: Use a random sample. If possible, include efforts to correct for non-response bias. The last thing you want to do is a voluntary internet survey. Sure, sometimes you can get some information out of a survey of that type. But more typically, those are prone to substantial selection bias. That is, the people who answer them are not a true cross-section of your target population, which in this case is Vienna citizens.)
Selection bias can occur naturally, per Councilwoman Patel’s point about using a paper-copy survey to avoid excluding those uncomfortable with or unable to use an on-line survey. But in addition, public-facing small-scale internet surveys are relatively easy to attack. Interest groups can easily create bias in any number of ways, starting with just passing the word to all your friends to go answer that survey.
Please note that I wrote that a year and a half ago. And, sure enough, apparently, the owner of one of the highest-membership Facebook pages about the Town of Vienna told her membership how they should answer that survey.
And here’s the awful part: That’s just the one you know about, because that was done brazenly. Most people who want to bias a survey will have the sense to do it quietly. And considering the commercial aspects of what’s at stake, such as the right to build bigger houses, vested interests would be remiss if they didn’t try to bias the results.
You note that I’m not blaming anyone for trying to bias the results. As an economist, I expect that to happen. That’s one reason you shouldn’t do a self-selected survey where somebody’s livelihood is affected. If you offer a self-selected survey that hits topics near and dear to vested commercial interests, you deserve what you get. Attempting to influence the information stream is simple, straightforward, profit-maximizing behavior.
Just looking at the answers, I have to tell you, they look a bit odd to me. The first question is an apparently minor technical issue of allowing porch roofs to reach into the mandatory setback from the road. Which, as these things go, is something of a snoozer. I’d bet the average Vienna resident doesn’t even have an opinion on that, let along a strongly held and firmly expressed agreement with that. Yet more then 50% strongly agreed with that.
What to do?
So, against advice from prior (and current!) Town Council members, Town staff did a low quality self-selected survey. If you go back to when this sort of thing was first being discussed (the 10/9/2019 Town Council meeting), you can find this in my writeup of that meeting, Post #413.
Several Town council members wanted to do something better than an on-line survey. Councilmembers Noble and Patel both advocated for a proper random-sample survey (rather than an on-line survey for whoever cared to answer the questions.)
(And so, once upon a time, we had a Town Council that was smart enough to know that a self-selected on-line survey was not a good way judge the opinion of the average Vienna resident (or maybe average Vienna voter). But Town Staff ignored them, and now all discussion of the problems with self-selected surveys seems to be forbidden, or perhaps just forgotten.)
There’s an easy solution here: Give it to a small random sample of Town residents after-the-fact. It’s incredibly easy to do, it’s incredibly cheap to do. Given the stakes here, I can’t understand why the Town wouldn’t do it.
The step-by-step directions.
The only point here is that this is easy to do, with Excel or equivalent, and either Microsoft Word (if you want to do the printing and mailing in-house), or by sending the resulting list to a printer (if not).
1: Take the spreadsheet list of eligible voters as your sample frame. That doesn’t hit every person in Vienna, but it’s close. And, really, if they can’t vote, in some literal sense, they don’t count. This survey is for a public purpose, and addresses a key issue for voters, so I am sure this would be an allowable use of the registered voter database.
2: Append a random number to every line. Add the Excel RAND() function to the end of every line. Hit return. Then freeze those values.
3: Sort by that random number, and take the first N hundred persons. Congratulations, you’ve just picked your random sample. And the voter eligibility list provides you with your mailing list for that random sample.
4: Optional security step. If you want, you can give people a security code to type in when they go to answer your survey. It’s easy enough to generate (say) five-digit random numbers as TRUNC(100000*RAND()). Now your mailing list has a random five-digit number that can be used in your mailing. (But that’s not really necessary, because you’ll know the name and address of every person you’ve asked to take the survey, anyway. It’s really more of a convenience for you than anything else.)
5: Deal with the people who are in your random sample but have already answered the survey. I have a preferred method, but you could do this either of two ways. One is to just to ask them to fill out the survey again. A more elegant solution is to pull them off the list, use their existing answers, but generate the same non-response percentage as the rest of your survey respondents by dropping them in proportion to the non-response rate for the rest of the survey.
6: Mail out a postcard to each person in the random sample, asking them to take your existing SurveyMonkey survey. (And have the common sense to put this second copy of the SurveyMonkey survey at a different URL from the original copy.)
7: Wait two weeks, and mail out a second post card to the non-respondents.
8: Optionally: Do a third followup postcard. These usually yield next-to-no additional responses.
9: Compare the answers from your random sample to the answers from your self-selected sample. SurveyMonkey does all the work there, I think. You just have to push the button. If you’ve used the five-digit “security code”, stripping this down to just the random sample is easy. Otherwise, you have to do a name-address match to do that.
I used to do this sort of thing for a living, so the first four steps would take me about 10 minutes. Then it’s just a question of (e.g.) using the mail-merge function of Microsoft Word to print the postcards.
In my experience, the most time-consuming step is putting the stamps onto the post cards. If you’re paying for the labor, it might be cheaper to buy pre-stamped post cards.
And if printing and mailing that many post cards is a problem, you can just send that whistle-clean mailing list to a printer. I’d be surprised if it cost more than $1 per card, for first class post cards printed and delivered to the USPS.
That’s all it would take to eliminate most of the risk of looking at mis-information when you look at your survey. It’s not 100% guaranteed. There will still be some non-response bias in a random-sample survey. But you will have eliminated the selection bias inherent in self-selected surveys.
Conclusion.
From what I can tell based on people I’ve talked to, a lot of the Town government involved with this really does not understand basic issues with surveys and sampling. That’s not hugely surprising, as I think that would be true of any cross-section of people. In any case, we seem to have multiple members of Town government saying that because they had a fairly large number of responses, the survey must be “good”, that is, it probably provides a valid representation of the views of the typical Vienna resident. That’s absolutely not correct, and anyone who believes that should take a few minutes to look that up on the internet. Else I will be forced to do yet one more boring posting on survey methods.
But if I had to sum up the situation, I would say this.
- Town Council members originally asked for this to be done as a random-sample survey, because they explicitly recognized the shortcomings of a self-selected survey.
- Town Staff simply refused to do that.
- We have public evidence that some vested interests tried to bias the results.
- We have no idea how many vested interested did that privately instead of publicly.
- And we have at least one answer that seems highly improbable to me. Just take a look at the answer to that first question, and think about it some. Does the average Vienna resident really have a strongly held opinion about that?
But the real point is, we don’t have to guess. We don’t have to act like a bunch of bumpkins who are too dumb to understand basic statistics. For an expenditure that would be lost in rounding error in the Town budget, we could use a standard methodology.
And as an economist, that last point really highlights the issue. Clearly, it’s not the cost that’s preventing this. Or, even at some level, the lack of understanding. It’s that Town Staff simply do not want to do a straight-up random-sample survey. It’s not that the Town can’t find the answer here. Maybe it’s that they don’t want to know.
Addendum
Why do I know this is easy to do? Because I’ve done it. Here, in the Town of Vienna.
And, because I tend to put my money where my mouth is, when I did a survey of Town if Vienna residents, regarding MAC zoning, of course it was a random-sample survey. In fact, my little survey is still the only random-sample survey of Town of Vienna residents ever conducted that directly asked citizens what zoning changes they would like to see.
Which is just pitiful, when you think about it. Given how cheap it would be to do a proper random-sample survey. Given the value of the property that is going to be affected by these decisions. It just floors me that the Town would half-ass its way through this.
Anyway, my survey yielded no huge surprises. When asked, straight up, citizens pretty clearly wanted more green space and smaller buildings than MAC zoning was providing. Full writeup and significant caveats for my cheap little survey are given in this posting.
