The general principle of question wording is that every respondent should understand the question and be able to answer it with reliability – that is, if she were asked the same question again, she would give the same answer. So question wordings are evaluated in terms of whether they can provide reliable information. A number of common problems have been identified, as well as solutions for dealing with them.
Open versus Closed
End: In an open end form, a question is asked and the respondent
answers in his own terms. In a closed end form, the respondent is
provided with a list of answers and then selects one. One example of
these differences can be found in the “most important
problem” question. This is asked most commonly in the open end
form used by the Gallup organization:
- What is the most important problem facing the country today?
But some polling organizations ask a question of the form:
- Which of the following problems is the most important one facing the country today?
This question wording produces a shorter list of problems, based upon the length of the list offered, and it may also produce other differences based upon the order in which the “problems” are listed.
Advantages and disadvantages exist in using either open or closed ended questions. Open-end questions are good for really getting at what is on people’s minds and having respondents talk about issues in their own words. On the other hand, open-end responses can be hard to code into meaningful categories particularly in tight time frames, take more time to administer so that the researcher must ask fewer questions, and can be hard to draw conclusions from if only a small number of people say any given response. Close-end questions are considerably easier to administer and analyze, but can sometimes make the respondent feel constrained in their answers if the responses categories do not include a response that they want to provide. The vast majority of polling questions are asked in close-end formats.
Double negative: In this form, the use of a double negative can confuse a respondent, especially when they are asked to provide a simple “Agree” or “Disagree” response. Take this example:
- On occasion, I am unable to express how interested in politics I am.
When the respondent answers “Agree” does it mean he is not interested in politics? Or he is interested but just can’t express himself well? This question would be better worded as:
- I am usually interested in politics.
Double-barreled: In this form, a pair of options is offered but the response alternatives are only “Yes” or “No.” In this case, it is not possible to determine whether the respondent is indicating one response or the other, or both. Take this example:
- Did you vote in the 2004 and 2006 elections?
When the respondent answers “yes” does it mean that she voted in 2004 and 2006, in 2004 but not in 2006, or in 2006 but not in 2004? This question could be worded as two separate questions or in the following way:
- Did you vote in both 2004 and 2006, in 2004 but not 2006, in 2006 but not in 2004, or in neither election?
Leading: In this question form, an initial phrase leads the respondent by suggesting the position or stance of an authority with which it might be difficult for the respondent to disagree. Leading questions introduce a bias in a particular direction linked to the authority. Take this example:
- Do you support President Bush’s decision to send additional troops to Iraq?
A better question might be:
- Do you favor or oppose sending additional troops to Iraq?
Sometimes questions can be leading by providing information about only one side of an issue, making it more likely that one side will be supported (opposed) than the other. Take this example:
- If it would result in increased opportunities for educating New Jersey citizens, would you favor or oppose building a new TV transmitter at liberty science center?
Question Order: In some surveys, the order of the questions may be designed to “lead” the respondent to a kind of conclusion that produces a predictable response. This form of bias would not have been present if the prior questions had not been asked. This is also referred to as setting up a “context effect.” For example, if you ask questions about a specific issue like the economy before asking what the most important problem is facing the nation, respondents will be more likely to name the economy in that subsequent question then they would have been without having that context set up for them.
The power of leading questions asked in the “correct” order can be seen in the results of a strategy survey conducted for the tobacco industry that was opposing Attorney General Morales in Texas because he was contemplating filing a suit against them.
They asked a general question about support for his reelection, then made intervening statements that highlighted and even distorted some of his positions on a variety of issues, and then they asked the question about reelection again.
Q15. Do you think Morales has performed his job as Attorney General well enough to deserve re-election, or do you think it’s time to give a new person a chance to do a better job?
- Re-elect 42%
- New person 27
- Don’t know 31
Then a series of intervening statements were made on Morales issue positions:
- Morales supports affirmative action
- Morales supports gun control
- As Attorney General, Morales made consumer issues a higher priority than fighting crime
- Morales has said that young gang members don’t need harsh treatment and prison, they need nice recreational facilities, drug counseling, and summer jobs.
Then the re-election question was asked again, and here is the shift in supprt:
Q38. Now that you have had a chance to learn more about Dan Morales’ record, do you think Morales has performed his job as Attorney General well enough to deserve re-election, or do you think it’s time to give a new person a chance to do a better job?
- Re-elect 42%............ 21%
- New person 27.......... 58
- Don’t know 31........... 21
Balanced Questions and Response Categories:
A balanced question will
equally represent both sides of an issue and/or will provide the
respondent with answers categories with an equal number of options on
For example, the question “do you support the United States taking military action against Iraq?” is an unbalanced question. Adding in the single word “oppose” so that it reads: “do you support or oppose the United States taking military action against Iraq?” makes it a balanced question. Similarly, “did you happen to vote in the presidential election in 2004?’ is unbalanced until the words ‘or not” are inserted at the end – “did you happen to vote in the presidential election 2004, or not?” which now signals to the respondent that it is a legitimate response to say “no”.
In terms of response categories, a balanced question will have an equal number of options on each side such as “strongly agree, somewhat agree, somewhat disagree, strongly disagree,” while an unbalanced question will lead the respondents to more heavily weight towards one side or another such as “all the time, most of the time, never.” A balance question may or may not have an explicitly stated midpoint (such an offered “neither agree nor disagree”) but in every case a mid point will exist either implicitly in the options (like in the case of strongly agree, somewhat agree, somewhat disagree, or strongly disagree where a mid point is implied between the two somewhat responses) or explicitly in the question. Here’s another example of unbalanced response categories:
Q: As you know, elected officials are held to high standards in public life. Here are some reasons people are giving to vote against Dan Morales for Attorney General. Please tell me if each statement makes you much more likely to vote against Dan Morales, somewhat more likely to vote against Dan Morales, or if it makes no difference at all?
Some questions may be structured in too a complex way or may include specific language that will not be well understood by all respondents. For example: “Do you think Congress should pass legislation to facilitate single-payer cost reimbursement plans for indigent patients?” includes references to at least three concepts that are probably not commonly understood (single-payer; cost reimbursement plans; and indigent patients). Alternatively, questions can include too much information or have too complex a structure for respondents to reliably answer as in the following example:
Now I'm going to read you some pairs of statements about what George (W.) Bush and John Kerry are saying about the economy. As I read each pair, please tell me whether the first statement or the second statement comes closer to your own views.)...Kerry's economic plan would end tax breaks for companies that outsource jobs and provide tax incentives for companies creating jobs here in America. It repeals Bush's tax cut for those earning over 200,000 dollars to ensure that all kids are guaranteed health insurance and reduce health insurance costs for small businesses and for families by 1000 dollars. And it creates new jobs by investing in alternative energy to reduce our dependence on Middle East oil. Bush's economic plan includes tax cuts for all Americans to keep our economy moving forward, reduces regulations to help small businesses, and eliminates the junk lawsuits that threaten jobs and entrepreneurs across America. The Bush plan will create7 million new homeowners by providing loan assistance and removing down payment requirements to make it easier for people to own their own home.
Which statement comes closer to your views? (If First/Second statement, ask:) Do you feel strongly about that, or not so strongly?
Back to top