The Reporting of Publicly Released Polls

The post election analyses assessing the accuracy of the publicly released polls in this election cycle has passed, at least until the next one. The assessments found that, despite the criticism, many of the independent/media polls were largely on target  within the point estimates of the final election tabulation. The state polls proved for the most part to be very reliable.

The most frequent charge leveled was that the polls over-sampled Democrats. First, there should be no doubt that any publicly released poll should be subject to a fair and honest debate relative to all aspects of the poll. It’s reasonable to ask questions dealing with sampling, weighting, cell phone inclusion, screening questions, and the list goes on.

Second, this year the criticisms had a very different motive. Many pollsters were accused of deliberately “skewing” their polls in order to help the Democrats. In other words, the motives of the pollsters were questioned not just the techniques.

Third, there should be agreement that polls should not receive media coverage without at least the following information being made available: who paid for the poll, what sampling was employed, the entire questionnaire, what statistical adjustments were made, and how did the pollster determine likely voters if determined. Actually we should demand more.

For years the Association of Public Opinion Research has promulgated standards for disclosure. The complete list of those standards appears below. Pollsters should abide by the standards and the media should refuse to report on polls whose researchers refuse to abide by the standards.

Good professional practice imposes the obligation upon all survey and public opinion researchers to disclose certain essential information about how the research was conducted. When conducting       publicly released research studies, full and complete disclosure to the public is best made at the time results are released, although some information may not be immediately available. When undertaking work for a private client, the same essential information should be made available to the client when the client is provided with the results.

A.  We shall include the following items in any report of research results or make them available immediately upon release of that report.

1. Who sponsored the research study, who conducted it, and who funded it, including, to the extent known, all original funding sources.

2. The exact wording and presentation of questions and responses whose results are reported.

3. A definition of the population under study, its geographic location, and a description of the sampling frame used to identify this population.  If the sampling frame was provided by a third party, the supplier shall be named. If no frame or list was utilized, this shall be indicated.

4. A description of the sample design, giving a clear indication of the method by which the respondents were selected (or self-selected) and recruited, along with any quotas or additional sample selection criteria applied within the survey instrument or post-fielding. The description of the sampling frame and sample design should include sufficient detail to determine whether the respondents were selected using probability or non-probability methods.

5. Sample sizes and a discussion of the precision of the findings, including estimates of sampling error for probability samples and a description of the variables used in any weighting or estimating procedures.  The discussion of the precision of the findings should state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to clustering and weighting, if any.

6. Which results are based on parts of the sample, rather than on the total sample, and the size of such parts.

7. Method and dates of data collection.

B.  We shall make the following items available within 30 days of any request for such materials.

1.  Preceding interviewer or respondent instructions and any preceding questions or instructions that might reasonably be expected to influence responses to the reported results.

2.  Any relevant stimuli, such as visual or sensory exhibits or show cards.

3.  A description of the sampling frame’s coverage of the target population.

4.  The methods used to recruit the panel, if the sample was drawn from a pre-recruited panel or pool of respondents.

5.  Details about the sample design, including eligibility for participation, screening procedures, the nature of any oversamples, and compensation/incentives offered (if any).

6.  Summaries of the disposition of study-specific sample records so that response rates for probability samples and participation rates for non-probability samples can be computed.

7.  Sources of weighting parameters and method by which weights are applied.

8.  Procedures undertaken to verify data. Where applicable, methods of interviewer training, supervision, and monitoring shall also be disclosed.