The following is from the MSN website on December 19, 2003:
About our Live Votes and surveys
How 1,000 people can be more representative than 200,000
One week in the middle of the Clinton-Lewinsky scandal, more than 200,000 people took part in an MSNBC Live Vote that asked whether President Clinton should leave office. Seventy-three percent said yes. That same week, an NBC News-Wall Street Journal poll found that only 34 percent of about 2,000 people who were surveyed thought so.
To explain the vast gap in the numbers in this and other similar cases, it is necessary to look at the difference in the two kinds of surveys.
Journalists use polls to gauge what the public is thinking. The most statistically accurate picture is captured by using a randomly selected sample of individuals within the group that is being targeted, typically adult Americans.
While a poll of 100 people will be more accurate than a poll of 10, studies have shown that accuracy begins to improve less at about 500 people and increases only a minor amount beyond 1,000 people.
So, in the case of that NBC-WSJ poll, only 2,005 adults were surveyed by the polling organizations of Peter D. Hart and Robert M. Teeter. The poll was conducted by telephone and had a margin of error of plus or minus 2.2 percentage points at the 95 percent confidence level. The confidence level means that if the same poll were conducted 100 times, each one randomly selecting the people polled, only five of the polls would be expected to yield results outside the margin of error.
Random selection of those polled is necessary to ensure a broad representation of the population at large. For example, a nationwide poll asking which NBA team is the best would likely yield a far different answer in Philadelphia than in Los Angeles. (And neither one would be a good sample of the population at large.)
In the NBC-WSJ survey, pollsters first randomly selected a number of geographic areas and then telephone numbers were generated in a way that allowed all numbers in those areas (both listed and unlisted) an equal chance to be called. Only one adult in each household was then selected to answer the poll.
While variation can occur depending on what questions are asked and how they are asked, similar questions tend to yield similar answers. One way to account for variation, however, is to ask the same question over a period of time.
In contrast, MSNBC's online surveys (Live Votes) may reflect the views of far more individuals, but they are not necessarily representative of the general population.
To begin with, the people who respond choose to do so - they are not randomly selected and asked to participate, but instead make the choice to read a story about a certain topic and then vote on a related question. There is thus no guarantee that the votes would reflect anything close to a statistical sample, even of MSNBC.com users: The participants in a Sports Live Vote and a Politics Live Vote may overlap, but each group is likely to be dominated by people with an interest in each particular area. In addition, while MSNBC.com's Live Votes are designed to allow only one vote per user, someone who wants to vote more than once could simply use another computer or another Internet account.
Additionally, while computer and Internet usage is growing daily, only 41.5 percent of Americans have online access at home, according to a study by the National Telecommunications and Information Administration <http://www.ntia.doc.gov/ntiahome/digitaldivide/index.html> released in October 2000. In contrast, more than 90 percent of Americans live in homes with a telephone. The study also shows a disproportionately higher rate of online access among whites and Asians, and people with higher incomes and levels of education.
This does not mean that Internet polling cannot be scientific. Harris Interactive <http://www.harrisinteractive.com/about/methodology.asp>, for example, has set up a system with checks and balances that allow it to use the Internet to obtain survey results comparable with more traditional methods.
But MSNBC's Live Votes are not intended to be a scientific sample of national opinion. Instead, they are part of the same interactive dialogue that takes place in our online chat sessions: a way to share your views on the news with MSNBC writers and editors and with your fellow users. Let us know what you think.