The Town would like to survey Vienna residents to get their opinion about key aspects of Maple Avenue development. See the relevant section of Post #413 for the writeup of that. I applaud the fact that the Town is going to do such a survey.
But the Town is just asking for trouble in how they have chosen to go about that. It’s a real drag to have to say that. But what I’m going to say here needs to be said, by somebody. Even though people are not going to like it, and I will be accused of being a mean person for saying it.
Well, tough.
What’s wrong with this picture?
First, what is the cardinal rule of survey design?
Here are some hints. It has nothing to do with random sampling. It’s not about statistical technique. It has nothing overtly to do with choice of wording. It’s not a rule about survey length, or use of graphics, of the types of scales used to elicit responses, screener questions, data edits, or anything like that.
The cardinal rule of survey design is that the entity performing the survey should have no vested interest in the results.
And that’s the rule that the Town is violating, big time. Because they’ve asked the Department of Planning and Zoning to do the survey.
Just go back to the 10/9/2019 Planning Commission meeting, listen to it, and start counting all the times the Director of Planning and Zoning a) pushed for having larger buildings and b) attempted to quash any attempt at mandating smaller buildings under MAC.
Objectively, this is not a person who has a neutral attitude toward the outcome of this survey. Objectively, if one of the key questions is about the height of the buildings, this is a person who clearly has a very strong opinion about what she would like, and what she would not like, as a result of this survey.
She wants bigger buildings, period. Bigger buildings means more development. She vehemently does not want a three-floor limit on MAC buildings, period, based on the last work session. I believe the phrase was something like “If there’s a three-floor limit, there’s no point to having MAC zoning at all”.
And Town Council has put the Director of Planning and Zoning in charge of their survey. The survey where they re going to ask the citizens about their preferences for, among other things, building height.
This is simply wrong. If you’ve never done surveys professionally, you may not understand this. But if you have, you know what I’m saying is correct. And it’s just common sense, to boot.
And I believe the Town Council itself may have some inkling already, because they made it clear at that meeting that they needed to review the draft of of the survey before it was fielded. I don’t think it’s normal policy for the Town Council to check staff work. But on this one, it is.
Worse, they’ve given free rein to use pictures when asking these questions, because, with the best of intentions, the Mayor was strongly in favor of a graphics-oriented survey. Heavy use of graphics — particularly photographs — opens up all kinds of opportunities for creating bias in the results. I’ll show you one, below. It’s far harder to examine for bias than a text-based survey, where plainly-biased language is much easier to spot.
To sum up this section, if you expect our Director of Planning and Zoning to do a straight-up, honest and unbiased survey regarding MAC zoning, you have not been paying attention, at all. I’ve said it before, and I’ll say it again: Near as I can tell, the DPZ acts as if its goal is to maximize the amount of development in the Town of Vienna. The Director has a clear and clearly expressed preference for larger buildings, and in the 10/9/2019 meeting openly scoffed at the idea of a three-story limit under MAC. And accordingly, DPZ staff are not the right people to draft, execute, and analyze this survey on behalf of Town Council.
So this is a risk. Whether or not Town Council proceeds with this approach, depends, I guess, on their taste for risk. But for sure, this is not what anyone would choose to do under ideal circumstances.
What’s the risk? I mean, if the survey provides the wrong impression of typical citizen sentiment, what’s the big deal? If the zoning is not an accurate reflection of the true cross-section of public opinion, so what? It’s not as if every Town policy has to be tailored to the wishes of the electorate.
The risk is that, eventually, reality matters. As in, eventually, there’s going to be another Town election. Despite my small-scale survey, and the tenor of the comments on the Town’s visual preference survey, and the last election results, there are still people who think there’s some silent majority strongly in favor of big and bigger MAC buildings. But at this point, I’m reasonably confident that they are wrong about that. And if Town Council mistakenly opts for much larger buildings, based on survey results that do not correctly reflect the true cross-section of Vienna voters, and if that matters to the citizens? Well, that mistake may end up being corrected in the next Town election.
Brief anecdote on biased language
I worked for about a decade as staff to a federal legislative-branch advisory body. At one point, I had to write up a policy for which the potential cost or cost savings (to the taxpayer) was unknown. I made the mistake of writing the following:
“This policy might raise spending.”
I thought that was fine. Instead, I was taken to the woodshed by the (then) representative of the AARP on that commission. He was, as it turns out, a retired high school English teacher. And, after pointing out the clear bias in that statement — that the reader would get the impression that this policy was likely to raise spending — he correctly had me change that to:
“This policy might or might not raise spending.”
My only point is that sometimes it takes a lot of work, and a lot of opposing viewpoints, to achieve truly neutral and unbiased language.
Obviously, any survey that I did is potentially subject to the same criticism I have levied at the Town’s survey. I, too, have a strong opinion on this. But take a quick look at the table at the start of this posting. Note the carefully balanced language. Every “pro” line has a matching “anti” line. To the greatest extent possible, language in those line pairs was identical. That’s not by chance. And that’s not even because I want to be fair. Nobody wants a fair fight. That’s because I’ve gotten my @# kicked too many times for failing to be as careful as possible when trying to achieve a truly neutral word choice.
A few suggestions
The Town isn’t made of money. I get that. So if it’s full speed ahead on a survey run and analyzed by the Department of Planning and Zoning, despite the cardinal rule, then let me offer a few suggestions.
Suggestion 1: Do not use photographs or drawings of actual buildings or street scenes. Use simple line drawings instead, with a stick figure of person drawn in to scale. I discussed this in my critique of the visual preference survey method.
With drawings, you can isolate the one thing at issue — height, say — while leaving all else unchanged. And you can do so cleanly, without introducing bias via the ancillary details of the picture.
With pictures, that’s not possible. In fact, they practically beg for you to use the picture details to determine the outcome.
For example, here are two pictures from the Town’s own Visual Preference survey, contrasting a narrow sidewalk to a broad sidewalk. These were among the pictures that Town staff used to conclude that people in Vienna wanted broader sidewalks.
Note that the narrow sidewalk appears filthy, in poor repair, and devoid of people or vegetation. And just in case you didn’t get the message, there’s an empty (For Lease) store front, the sky is overcast and the pavement is wet. Seriously, all it needs is a couple of zombies shuffling down the street to complete the picture.
Source: Town of Vienna, VA visual preference survey. Original credit: “Biz Photo by Don Simmons Dec 17, 2003”
Whereas the broad sidewalk is clean, well-populated, with brightly-lit shop interiors, trees, and so on. Cute. In the background, I hear Julie Andrews singing “Raindrops on roses and whiskers on kittens, … “.
Source: Town of Vienna, VA visual preference survey. Original credit: Unknown.
Let me re-emphasize that these are actual photos from the Town’s last survey. The people who gave you that contrast, and then interpreted the results as showing a strong preference for broad sidewalks, are the same folks who will do the next survey. At some point, I don’t care if that was intentional or not. You ignore the cardinal rule at your peril.
Suggestion 2: Use a random sample. If possible, include efforts to correct for non-response bias. The last thing you want to do is a voluntary internet survey. Sure, sometimes you can get some information out of a survey of that type. But more typically, those are prone to substantial selection bias. That is, the people who answer them are not a true cross-section of your target population, which in this case is Vienna citizens.)
Selection bias can occur naturally, per Councilwoman Patel’s point about using a paper-copy survey to avoid excluding those uncomfortable with or unable to use an on-line survey. But in addition, public-facing small-scale internet surveys are relatively easy to attack. Interest groups can easily create bias in any number of ways, starting with just passing the word to all your friends to go answer that survey.
It’s not hard to do a random-sample survey. Councilwoman Patel hit on one obvious approach, which is simply to mail one to every Vienna household as part of the Vienna Voice (Town newsletter). By contrast, I started from a commercial mailing list and randomly chose 400 addresses using the Excel ranuni() function. Then I mailed out 400 envelopes, with an invitation to take an on-line survey, and a unique, short, random passcode for each address. That way, sure, anybody could answer the survey, but I knew which ones were in the random sample.
(Just a fine point here, but in practice, for either of these examples, the sample frame for the survey is not Vienna residents, it’s Vienna households. You allow (at most) one response per household. Individuals in multi-person households are under-sampled relative to individuals in single-person households. Usually, that’s not enough to create a material bias in the results. But you need to be sensitive to the potential for bias. You in effect over-sample the very young (solo apartment dwellers) and very old (widows and widowers) relative to the median Vienna household (couples).)
It is so much cheaper to do surveys on-line (as opposed to paper-copy) that there is no doubt that the Town would like to get all or most of the responses that way. Both the Town and I used SurveyMonkey for our respective surveys, and my guess is that this would be the Town’s choice for the subsequent survey. I believe SurveyMoney typically charges $1 per response, which in my experience is far less expensive than having a paper-copy survey keypunched (a quaint term for entering the answers into a database, originating in the era when computers literally used punch cards.) And, because it’s on-line, you avoid what can be substantial printing costs, particularly if the survey includes a large amount of graphics.
You do need to be mindful of the $1 per response charge. If you can get an adequate sample size without surveying everyone in Town, then you should do so, and thereby keep your costs down. (Determining what is an adequate sample size requires taking a guess at the response rate (what fraction will actually take your survey), and then doing a “power calculation” to determine how to achieve an acceptable margin of error, given that anticipated response rate.)
But how do you handle the small number of people who prefer to take the survey on paper, and then (e.g.) mail in the results? You have someone type their answers into the SurveyMonkey system. In effect, somebody takes the on-line survey for them, using their paper-copy answers.
Suggestion 3: If it were up to me, and there were no legal or other issues, I’d use the Town’s voter roll as the sample frame for my survey. It’s better data than commercially-available mailing lists, voters are what matter, and it contains a modest amount of demographic data (age, gender). The demographic information could be used to adjust for non-response bias.* It could also be used to create a stratified random sample if you are going to sample just a subset of the entire Town population.**
* and **: You could, for example, divide the population into age-sex cells (Men under 30, 30-40, …. 85+, women under 30 … 85+), determine the response rate for each cell, and re-weight the responses inversely proportional to the response rate. E.g., if the under-30’s were only half as likely to respond as others, count each of their responses twice in the final tabulation. Similarly, if you do a random sample, you can make it more precise by randomly sampling the exact same fraction of the population in each cell.)
Suggestion 4: It’s easy enough to execute what Councilwoman Patel suggested, in full, with just a few simple steps, at modest cost. She’d like to use the Vienna Voice for the survey, and allow those who prefer a paper copy survey to take it that way, rather than on-line. The trick is, you don’t mail out 6000 copies of the survey. You mail out 6000 invitations to take the survey.
Exactly how you do that depends on a number of factors: How persnickety you wish to be about treating on-line and paper-copy surveys identically, how capable the printer is, and whether you think that reliance on US Mail for part of the process would itself create some bias. And, to a limited degree, whether or not you wanted to attempt any corrections for non-response bias.
Here’s one way to do it, ignoring the finer points. You’d have an article in the Vienna Voice announcing this survey. The article would clearly state the purpose of the survey, the number of questions, and the approximate amount of time it should take to fill out the survey. The article would list the URL for accessing the SurveyMonkey site where the on-line survey was located. And that issue would have a business reply postcard or envelope that individuals could pull out and mail back to the Town of Vienna.
In addition, for that issue, you would augment the “Resident” line of the existing mailing list by adding a unique random five-character code (letters and numbers) for each address. That’s a snap to do if the mailing list file is available as an Excel file. But, conversely, this step may be where the printer is unable to handle this task. Assuming the printer can do it, the Town keeps a list of those codes, because those are the main security feature for the survey. (Case-insensitive, there are about 60M such codes possible, so only about 1 in 10,000 would be a valid code — so it’s tough to get one by guesswork.)
Respondents wishing to take the survey on-line would be asked to go to the SurveyMonkey URL and carefully type in the five-character code. They would then be allowed to proceed with the survey. (In fact, an invalid code would not prevent you from taking the survey, so that individuals who attempted to cheat the system by trying random five-character combinations would be given no feedback. The person analyzing the results would throw out all responses with invalid codes, or duplicate responses with the same valid code. Note that this requires someone who is capable of handling the SurveyMonkey response-level file.
But if you would prefer paper copy, just pull out the reply card, put your address on it, copy the five-character code, and send it in. (If the printer were capable, you’d pre-print all of that on the reply card, which would make it easier for the recipient). Town staff would then manually address and mail a printed copy of the survey to each such individual, with a postage-paid return envelope. Note that you only pay for the postage on these items — the pull-out card in the Vienna Voice, and the reply envelope — if they are actually used. Assuming a low desire for the paper-copy version, this would not cost much or involve much staff time. Staff would also have to type those responses into the SurveyMonkey database, once received.
There are other, less secure ways to try to prevent someone from “gaming” the survey that would not require this five-character code. You could, as with the Town’s visual preference survey, require the person to type in their address, check for a valid address, and in addition rely on SurveyMonkey’s own methods for preventing duplicate submissions from the same URL. My feeling is that this is less secure than supplying a random code.
For the record, even though the arithmetic does not say so, I suspect that surveying only a fraction of Town residents would actually be the more secure approach. (In addition to being potentially cheaper, depending on mailing costs.) Anyone trying to organize a “fill out the survey” campaign would (hopefully) be discouraged by the fact that only a small subset of their friends and acquaintances would be eligible to take the survey.
Suggestion 5: Practice, practice, practice.
5.0 Review this for compliance with common standards. Every abbreviation must be spelled out first. It cannot take longer than (maybe) ten minutes. Ideally it would take less time than that. Any series of questions must have the questions asked in a parallel format. It must be crystal-clear which end of the response scale is low, and which one is high (or which is strongly disagree versus strongly agree, and so on).
5.1 Seek review by those on all sides of the issue. Send a copy of the survey to several people who do not see eye-to-eye with the survey creators, and have them scrutinize it. The survey must be acceptable to persons on all sides of this issue. If necessary, written comments need to go to a neutral third party for adjudication.
5.1 Have several people who are completely unfamiliar with this issue first take the survey, and then separately identify all the things that they find confusing or ambiguous. The rule here is that for every idiot-proof approach, there’s always a bigger idiot. I can’t tell you how many times I’ve thought I was perfectly clear, only to get garbage back from many respondents.
