How to do research with self-administered surveys
Frederick S Sierles. Academic Psychiatry.
Author(s): Frederick S Sierles
Article types: General_Information
Section: Original articles
Publication title: Academic Psychiatry.
Source Type: Periodical
ISSN/ISBN: 10429670
ProQuest document ID: 352401881
Text Word Count 4604
Article URL: http://gateway.proquest.com/openurl?ctx_ver=z39.88-2003&res_id=xri:pqd&rft_val_fmt=ori:fmt:kev:mtx:journal&genre=article&rft_id=xri:pqd:did=000000352401881&svc_dat=xri:pqil:fmt=html&req_dat=xri:pqil:pq_clntid=62763
More Like This »Show Options for finding similar articles
Full Text (4604 words)
Copyright American Psychiatric Press, Inc. Summer 2003
[Headnote]
Given the absence of brief "how to do it" articles on self-administered survey research, the author presents suggestions for conducting and publishing questionnaire research successfully. The article covers choosing a topic, framing a core research question that can be tested scientifically with survey data, respecting the study subjects' time and pride, keeping the survey materials succinct, being persistent, and guaranteeing study subjects' anonymity or confidentiality. (Academic Psychiatry 2003; 27:104-113)
Conducting survey research with self-administered questionnaires is a widely used, low-tech, relatively low-cost academic technique; however, all too often it is not done well and winds up unpublished. To my knowledge, no brief "how to do it" papers have been published on the subject. In this article I offer suggestions for successfully conducting and publishing survey research. The main themes I cover are choosing a topic that intrigues or engages you; posing a core research question that can be answered scientifically with survey data; respecting the recipient's (respondent's) time and pride; keeping the survey material succinct; being persistent; and guaranteeing study participants' anonymity or confidentiality.
The scientific principles of survey research are the same as those of other research. Indeed, questionnaires-both the self-administered type and those administered by interviewers-are fundamental research instruments. Good surveys have numerous common features. Of course, because survey research is both an art and a science, each research group contributes uniquely to the survey literature (1). The techniques and approaches presented in this article characterize my style, which generally coincides with the recommendations made in standard textbooks on survey research (2). Examples of survey-based papers-"finished products"-may be found in references (3-10).
CHOOSING A TOPIC
If you are an inexperienced researcher, it may surprise you that most of your questions about medical education or clinical care are fine starting points for a study. Most questions you might pose have not been answered decisively in the medical education or clinical literature. Yager (11), for example, provided a huge list of medical education topics worth studying. Ideally, a research question will be characterized by the following: the topic intrigues you and would interest your peers; you posed the core research question personally, although many fine papers are written as a result of answering others' questions; and the research question can be answered through data generated by a survey.
Framing the Core Research Question
Your research question should be scientifically answerable and should be interesting to your peers. For example, in 1979, when I was a new director of medical student education at Finch University, my chairman and I had to determine whether to devote faculty or staff time to proctoring examinations. The initial question-"Should we proctor our examinations?"-was scientifically untestable.
Perhaps we could have surveyed the approximately 127 directors of medical student education nationwide to ascertain whether, how, and why they proctor or have honor systems. But cataloging these systems nationwide would have been uninteresting. It would not address these systems' goals and outcomes, such as maximizing the likelihood that students will act ethically in medical school and in their practices and that they will develop sound professional values.
What we really needed to know was the frequency with which our students-or medical students nationally-cheated, and what were the types of cheating (e.g., plagiarism) and the predictors of cheating (e.g., does cheating in college, or having a cynical attitude, predict cheating in medical school?). Since the consequences of cheating-especially of being caught at it-are serious, we considered conducting an anonymous self-report survey of all medical students at our institution to ascertain the frequency of their cheating. We would guarantee anonymity in order to create a condition in which self-reported cheating could have no consequences.
If good studies of medical student cheating already existed, we might not have done the survey. Our reference librarian searched the literature, and we found that no studies of medical student cheating had been published. However, extensive literature was available on the frequency of college student cheating and on valid ways to ascertain it.
Because schools all differ in some respects from one another, we invited another director of medical student education to conduct the survey at her school at the same time. We lacked the resources to study more than two schools. Thus the core question became "What are the frequency and correlates of cheating in these two schools?" The response rate was 95%, and the paper was published (3). Because we studied only two schools, the survey's generalizability (validity) was limited (10), but it answered our research question and was a valuable preliminary contribution.
Finding Articles on the Topic
To confirm that your question has not been answered decisively, search two or three Internet databases and consult with one or two colleagues. The standard medical and medical education database is PubMed (www.pubmed.gov) which covers papers from 1966 to the present and is available at no cost from the National Library of Medicine. To learn how to use PubMed, click "Tutorial" on its web site.
The American Psychological Association's PsycInfo (www.psycinfo.com) has references that PubMed omits (there is an overlap of about 80%). The Institute for Scientific Information's web site (www.isinet.com) contains the Science Citation Index (SciSearch) and the Social Sciences Citation Index (SocSciSearch), which list the references for each cited article as well as papers and books that have cited the article. Because SciSearch and SocSciSearch are costly, some universities cannot provide them to faculty on-line, in which case you could collaborate with your reference librarian. For general (e.g., undergraduate college) education, search the Educational Research Information Clearinghouse (ERIC) (www.ericir.syr.edu).
Countless medical articles were published before 1966, when MedLINE-the precursor to and a major component of PubMed-began. Moreover, since 1966, articles in many relevant journals are still not indexed in PubMed, PsycINFO, SciSearch, or ERIC. For example, PubMed indexes Academic Psychiatry only for articles published since the Spring 2001 issue. Thus, you must also use your library's bound journals and the interlibrary loan system.
SELECTING YOUR SURVEY SAMPLE
In planning your study, consider whether the conclusions you
draw from your sample will be generalizable to the population (or universe) of possible
respondents. When you write your paper, address the extent to which your
results are generalizable. For example, if your question is "What are the
perceptions of clerkship directors about the impact of managed care on the
medical education environment?" (4), and you send questionnaires to the
clerkship directors in each core third-year specialty in every
If your question is "To what extent do
RESPECTING PARTICIPANTS' TIME AND PRIDE
Most survey recipients-indeed, most persons-perceive themselves as busy and are most apt to respond if they can answer the questions immediately at their desk or a table near the mailbox. Most recipients discard the questionnaire if it requires that they look up information, communicate with a colleague, or do anything other than complete the questionnaire then and there, within 5-10 minutes. Also, most recipients will discard the questionnaire if they perceive that responding frankly could embarrass them.
Few recipients will let anyone but a confidante (and a vague statement about confidentiality does not a confidante make) know that they are not conforming to the surveyor's-or society's, or their own-standards. Why would an education director let someone know that although he or she values education about, say, mental retardation, the department presents only one classroom hour on the subject in a 4-year curriculum? Unless I am guaranteed anonymity or confidentiality, I automatically discard questionnaires that could embarrass my department or me. For example, I toss any survey conducted by a researcher with a major interest in a topic that asks me to endorse a statement that my department does not teach that topic "enough," unless the surveyor guarantees my anonymity or confidentiality.
Many questionnaires pose questions (e.g., "Have you ever faked an illness?") that could compromise the respondent if others knew his or her answers. For these reasons, view each recipient as poised to pitch the questionnaire the moment he or she perceives that responding will waste time or cause shame.
Would you readily tell anybody-except, perhaps, a confidante-that you cheated in medical school (3) or malingered an illness (6), or that only 1% of your medical school's graduates matched in psychiatric residencies (7), or that your job is sometimes demoralizing because of excessive clinical obligations (4), or that you don't know the precise criteria for many DSM-IV diagnoses (5)? Were any of these statements true for me, I would not affirm as much unless I had a compelling academic motive to do so. I might, however, admit to them if I believed that nobody could or would try to identify me. In anonymous surveys that my colleagues and I have conducted (3-8), we have posed these questions and elicited high response rates, with many responses endorsing the socially unacceptable behavior. This occurred despite the social desirability phenomenon (13), by which respondents overreport socially desirable attitudes and behaviors and underreport undesirable ones.
Guaranteeing Anonymity or Confidentiality
To maximize the respondents' frankness, you must minimize the chances that the recipient will worry about being embarrassed, so your survey must be anonymous or confidential. You must convey that you do not know, will not try to discover, and do not care to know, who the respondent is. You must guarantee that neither you nor your colleagues will read his or her responses-and therefore could not pass judgment about them. Medical students, for example, view themselves as academically vulnerable if their illnesses (e.g., depression) are known to faculty (9).
Progressively more attention is being given to trainees as research subjects (14), and some institutional review boards ask you to specify in the informed consent form or the survey cover letter that you take personal responsibility-in essence, legal liability-if you breach confidentiality.
As a surveyor and as a recipient, I prefer anonymity to confidentiality-without-anonymity (CWA). As Bradburn (15) has written, "With sensitive questions or those associated with a high degree of social desirability, the more anonymous methods appear to work somewhat better; that is, they lower the degree of under- or over-reporting" certain behaviors.
Some of my colleagues prefer CWA to anonymity. CWA surveys cost less because follow-up mailings need to be sent only to nonrespondents, and it is easier to compare respondents with nonrespondents if one knows about the nonrespondents.
Conducting Anonymous Surveys
To convey that a survey is anonymous and that you respect the recipient and his or her time, it is crucial to create a well-planned cover letter (see Appendix 1) stating that the survey is anonymous (3). In a mail survey, a self-addressed stamped envelope, with your own name and address as the return address, should be provided (having to obtain a stamp turns many recipients into nonrespondents), and the cover letter should ask the recipient not to write his or her name on the questionnaire or the return envelope.
Anonymous questionnaires should contain no code numbers, as these suggest that you could figure out the respondent's identity. How, then, in mailing a follow-up questionnaire, could you know who has already responded, avoiding a repeat mailing to respondents? In other words, how do you ensure that only nonrespondents receive follow-up questionnaires? The answer is that if you have promised anonymity, you should not know who responded.
This means that you must send the second and third mailings to the entire sample-which has the advantageous effect of affirming that you do not know the recipients' identities. Although this procedure costs more than repeat mailings only to recipients who have not yet responded, it is more likely to yield a high response rate. Thus, conduct an anonymous survey only if you can afford three mailings. Some researchers ask respondents to fill out a separate enclosed stamped postcard to indicate that they have completed the questionnaire. Thus, anonymity is maintained, but this, too, is costly and time-consuming.
Confidential Surveys
Most of the preceding recommendations also apply to CWA surveys. You must conspicuously convey that you will not read the respondent's replies, and you should state this in the informed consent form or cover letter. In CWA surveys, you can avoid repeat mailings to those who have already responded by writing code numbers on the questionnaires and linking each code number with the respondent's name.
PREPARING SURVEY MATERIALS
The Cover Letter
In a mail survey, write your cover letter on department or
organization stationery and address it to each recipient individually (e.g.,
"
Phrase the cover letter as respectfully and neutrally as possible, the latter to convey that "socially unacceptable" responses are as agreeable to you as socially acceptable ones. As much as possible, explain the survey's purpose to the recipients, and phrase the letter to make the topic interesting to the recipient. Appendix 1 (3) provides an example.
Designing the Questionnaire Booklet
Unless your questionnaire has only one or two pages, prepare a booklet with questions on both sides of the page rather than stacking and stapling pages (1). The booklet (see Appendix 2 for an example), should be trim and compact; in a mail survey, it should fit neatly into the enclosed return envelope. A bulky appearance reduces response rates (16). The booklet should be attractive and well duplicated, with ample, balanced margins on both sides of each page and no lopsided margins, typos, smudges, or faintly duplicated type.
The survey in Appendix 2 (8) was typed on both sides of an 8 1/2 x 11 sheet of paper with a horizontal (landscape) page setup and a size 10 font. To create a four-page unstapled folded booklet from one sheet of paper, page 1 of the survey was typed on the right half and page 4 on the left half of one side of a sheet of paper. On the reverse side, page 2 was typed on the left and page 3 on the right.
Writing the Questions
The booklet should contain succinct, crystal-clear, closed-ended questions (1). Extra space may be provided with each question for respondents to elaborate spontaneously on some answers or, at the end, include an open-ended item such as "Other comments." Closed-ended questions should include multiple choices exploring all possible alternatives (see Appendix 2, items 12-17). If a sequence of questions might not apply to a respondent, there should be clear instructions (using capital letters, underlining, or italics) to skip the sequence (e.g., "If you have already taken Dr. Green's Neuroscience course, please skip to question 7."). If you must use more than one "skip sequence," the sequences must not be complicated. Group questions in a reasonable order and title each section, as in Appendix 2. You and your statistician should also review the questionnaire to ensure that data entry will be straightforward and go smoothly.
Although most survey principles (1) resemble those of medical interviewing, several do not. Survey principles that are similar to interviewing principles include the following: Keep questions short, clear, and responsive to the recipient's education and vocabulary. Make the opening questions easy and non-threatening. Do not ask double questions, such as "How satisfied are you with your working conditions and salary?" Do not ask vague questions such as "Do you read Academic Psychiatry?" Here an affirmative reply could indicate anything from having read one article in one issue to having read several articles in every issue for 12 years. Avoid double negatives. Do not be condescending. You can ask almost any question, if the question contributes to answering the core research question, flows from the questions that precede it, and is posed respectfully and nonjudgmentally. "Difficult" questions should be placed such that the respondent will already have answered other questions comfortably.
A feature of questionnaires that differs from medical interviewing principles is that in self-administered surveys, you should ask closed-ended questions primarily, if not exclusively, for reasons discussed below. An aspect of surveys that differs somewhat from medical interviews is that surveys should employ a forced-choice method, omitting "I don't know" or "I have no opinion" options. Such options let the respondents off the hook too easily when, in fact, they very likely would express an opinion if allowed to do so, and if you have conveyed that you value their answer. In clinical interviews, you follow up on many "don't know" answers by asking, "What's your best guess?" or "Nobody really knows, but how do you feel about it?" or "Because this is so important to us, let's discuss it more."
Spontaneous Narrative Comments
As you know, spontaneous narrative comments, or responses to open-ended items, such as "Other comments" at the close of the questionnaire, can be valuable (1). But do not count on quantifying the spontaneous narrative comments easily, if at all. At best, you will be able to organize the narrative comments into categories and quote several vivid, illustrative comments in your paper.
Two-Choice Scales
Probably you have filled out countless Likert scale items. In this format, a statement (e.g., "Being a clerkship director is personally fulfilling") is followed by a progression of choices, such as 1, strongly agree; 2, agree; 3, neither agree nor disagree; 4, disagree; and 5, strongly disagree-which is a 5-point Likert scale. Likert scales can range from 2-point scales, in which the respondent checks whether he or she agrees (see Appendix 2, question 28), to scales with as many as 10 gradations.
Market researchers (17,18) find that 5- to 9-point scales are best for advertising purposes, and many medical researchers prefer scales in this range because they yield data with a level of precision that is amenable to advanced statistical analysis. Scales in this range can be treated as interval-ratio data (i.e., with equal intervals between points, as on thermometers) and can be used in parametric analyses such as t tests, analyses of variance, Pearson correlations, and multiple regression analyses, whereas 2-point scales produce nominal or categorical data (i.e., data with no gradations, in which something simply is or isn't), which can be analyzed with nonparametric statistics such as the chi-square test, Fisher's exact probability test, and logistic regression analysis (19). However, it has not been demonstrated conclusively that 5- to 9point Likert scales produce true interval-ratio data: Is the interval between "strongly agree" and "agree" identical to that between "disagree" and "strongly disagree"? The main advantage of 2-point scales-and the reason I recommend them-is that they take less time to complete.
Avoiding Response Bias
In using Likert scales to elicit respondents' attitudes, balance statements to avoid biasing answers in one direction (20). For example, in a questionnaire seeking to elicit clerkship directors' attitudes about the impact of managed care on clerkships (3), we balanced pro- and anti-managed care statements, including the following: managed care has slowed the rise in health care costs for employers and the government; managed care has made health care more like a commodity; managed care has reduced professionalism in medicine; and managed care is improving the health care of most Americans.
Peer Reviews and Trial Runs
Have several colleagues comment on each draft of the questionnaire, and then administer one or two versions to a small subsample-say, five people. You cannot be sure about the user-friendliness of your survey until you have received feedback. Once you mail it, you can't fix it.
RESPONSE RATE
The higher the response rate, the closer your study sample is to the study population and the more generalizable your results are. In anonymous mail surveys, unless the initial response rate exceeds 80%, mail it again to all the original recipients. In the cover letter for the second mailing, add prominently the statement "If you filled out this questionnaire previously, please do not fill it out this time," which, again, reinforces the anonymity theme. In CWA surveys, unless the initial response rate exceeds 80%, mail it again to the nonrespondents.
Although, to my knowledge, there is no decisive standard for what constitutes an acceptable response rate for a mail survey, try for at least 70%. Over 80% is excellent. Between 60% and 70% is usually acceptable, and sometimes less than 60% is acceptable-especially if the subject is controversial or if little empirical work has been done on the topic. In such cases, note in the discussion section of your paper that the response rate is modest and, if possible, compare your respondents with the nonrespondents or with the entire sample.
Each follow-up mailing yields 30%-50% of the number of initial responses; thus, if your initial response rate was 50% of your sample, the response rate for your second mailing will be 15%-25% of your sample. If it is nearer 15%, do a third mailing.
CONDUCTING SELF-ADMINISTERED SURVEYS IN PERSON
When you have direct access to the entire sample or population (e.g., your trainees), distributing and collecting the questionnaire in person to a group yields much higher response rates than does mailing it or distributing it to take home. Schedule unencumbered time and space in advance. In-person distribution permits you to explain the survey face-to-face, to answer questions, and to demonstrate your commitment. Compared with a mail survey, this arrangement also confers more authority on you, which is legitimate if you affirm the recipients' right not to participate. For example, if a recipient leaves the room, take no notice and make no response whatever. Direct administration rarely works at education meetings, where attendance rarely approaches 80%. Also, mixing in-person with mail administration reduces reliability.
ESTIMATING COSTS
The cost in materials for conducting, writing up, and presenting the results of a typical survey with three mailings to 125 recipients is $600-$800; that includes stamped envelopes, stamped enclosed return envelopes, 375-400 pieces of department or organization stationery, 100 journal reprints, reprint mailing, and 25 slides for meeting presentations. Secretarial time is typically 10-15 hours. If neither you nor a coauthor code, enter, or analyze the data, paying a statistically savvy graduate student $15 an hour to do it will cost $300-$500. Any travel expenses should be considered as well.
CONCLUDING REMARKS
This article is by no means meant to provide a comprehensive review. For example, I included only minimal discussion of statistical analysis, and even less of writing and submitting the paper. The 1983 survey research text by Rossi et al. (2) is comprehensive. Norman and Streiner's (19) text addresses biostatistics well, and Green et al. (21) cover SPSS computer analysis well. The spring 2001 issue of Academic Psychiatry (volume 25, number 1) contains a fine set of papers on education research issues. Hopefully, though, this paper has presented practical strategies that will increase your chances of publishing scientifically meritorious papers that answer your educational or clinical research questions.
An earlier version of this paper was presented at the annual
meeting of the Association of Directors of Medical Student Education in
Psychiatry in
[Reference]
References
1. Sheatsley PB: Questionnaire
construction and item writing, in Handbook of Survey Research. Edited by Rossi
PH, Wright JD,
2. Rossi PH, Wright JD,
3. Sierles FS, Hendrickx I, Circle S: Cheating in medical school. J Med Educ 1980; 55:124-125
4. Brodkey AC, Sierles
FS,
5. Jampala VC, Sierles
FS, Taylor MA: The use of DSM-III in the
6. Sierles FS: Correlates of malingering. Behav Sci Law 1984; 2:113-118
7. Sierles FS: Medical school factors and career choice of psychiatry. Am J Psychiatry 1982; 139:1040-1042
8. Sierles FS, Magrane D: Psychiatry clerkship directors: who they are, what they do, and what they think. Psychiatr Q 1996; 67:153-162
9. Roberts LW, Warner TD, Lyketsos C, et al: Perceptions of academic vulnerability associated with personal illness: a study of 1,027 students at nine medical schools. Collaborative Research Group on Medical Student Health. Compr Psychiatry 2001; 42:1-15
10. Baldwin DC Jr, Daugherty SR, Rowley BD, et al: Cheating in medical school: a survey of second-year students at 31 schools. Acad Med 1996; 71:267-273
11. Yager J: Preparing psychiatrists to do educational research. Acad Psychiatry 2001; 25:17-29
12. Sudman S: Applied sampling, in
Handbook of Survey Research. Edited by Rossi PH, Wright JD,
13. Crowne DP, Marlowe D: The
Approval Motive: Studies in Evaluative Dependence.
14. Roberts LW, Geppert C, Connor R, et al: An invitation for medical educators to focus on ethical and policy issues in research and scholarly practice. Acad Med 2001; 76:876-885
15. Bradburn NM: Response effects,
in Handbook of Survey Research. Edited by Rossi PH, Wright JD,
16. Dillman DA: Mail and other
self-administered questionnaires, in Handbook of Survey Research. Edited by
Rossi PH, Wright JD,
17. Cox EP III: The optimal number of response alternatives for a scale: a review. J Marketing Res 1980; 17:407-422
18. Green PE, Rao VR: Rating scales and information recovery: how many scales and response categories to use? J Marketing 1970; 34:33-39
19. Norman GR, Streiner DL:
Biostatistics: The Bare Essentials, 2nd ed.
20. Guyatt GH, Cook DJ, King D, et al: Effect of framing of questionnaire items regarding training on residents' responses. Acad Med 1999; 74:192-194
21. Green SB,
[Author Affiliation]
Frederick S. Sieries, M.D.
[Author Affiliation]
Dr. Sierles is Professor of
Psychiatry and Behavioral Sciences at Finch University of Health Sciences/The
Chicago Medical School,
Copyright (C) 2003 Academic Psychiatry.
Enlarge 200%
Enlarge 400%
APPENDIX 1. An Example of a Survey Cover Letter, developed for Sierles FS, Hendrickx I, Circle S. Cheating in medical school. J Med Educ 1980; 55:124-125.
Enlarge 200%
Enlarge 400%
APPENDIX 2. An Example of a Four-Page Survey Questionaire, developed for Sierles FS, Magrane D. Psychiatry clerkship directors: who they are, what they do, and what they think. Psychiatric Q 1996; 67:153-162.
Enlarge 200%
Enlarge 400%