Questionnaire Pretest

A functioning questionnaire design does not guarantee that every participant understands the questions correctly. Therefore, the most important step to generate valid data is a pretest.

Pretests often fall short, especially in online surveys. This accounts for a large part of the perceived poor data quality of online surveys used in studies.Therefore, how to perform a proper pretest is explained below.

SoSci Survey allows for online pretests. These typically include less effort, but only reveal major issues. To grasp the more subtle issues (poor formatting, unclear wording, etc.), a face-to-face pretest shall be done.

1. Create a pretest hyperlink at Survey Project → Pretest and Preview.
2. Testers can access the questionnaire with this hyperlink. They will see the question's IDs and a text input to leave comments about the questionnaire.
3. Finally, download the testers' annotations at Collected Data → Pretest Comments.

Warning: Should you delete the collected data, the pretest annotations are deleted as well.

A pretest takes – depending on the care taken during the design of the questionnaire – between one hour and several days. Investing this time is worthwhile though, since a severalfold will be saved when analysing the data. Further, data quality will be higher and rates of discontinuation lower.

1. You need approx. 5 people who only know little about your study (just like your later participants). Ideally, this will be people from your target population – they can often be found in the family, the friend circle or the canteen. For the latter, a laptop with wireless internet connection can prove very useful.
2. Place these people in front of the questionnaire one by one. All you need is a computer with internet connection, the questionnaire URL (internet address, can be found in Compose Questionnaire) and the Pretest password.
   1. While completing the questionnaire, the person should comment on anything that crosses her mind (method of loud thinking); especially, in case something is unclear or ambiguous. You may need to remind your tester at times to think aloud while completing the questionnaire.
   2. As an observer, simply take notes of every comment made – do not comment or justify (“but generally, you understand what is meant, don't you?”).
   3. If you are uncertain whether a question was understood correctly, you can ask your tester directly what she understood. You should also ask whether longer texts (this can already be 2-3 sentences) were actually read.
   4. If your testers were satisfied with everything at first contact, then you haven't sufficiently asked them for criticism. You will always receive some criticism if asked for it properly, since you cannot cater for everyone…
3. After collecting feedback from these 5 people, think carefully about the comments made. Some small issues can be overcome easily (e.g. missing answer options), other issues may just concern one person. Of course, you do not need to change everything, if there is a good reason not to.
4. Once you have incorporated all your changes, you should repeat the pretest – otherwise you may be making matters worse. Of course you need to recruit different people, since the original testers now know how to understand your questions.
5. Even good questionnaires need a couple of alterations. Usually, 2-4 Pretest rounds are needed, to turn a good questionnaire into a really good one. However, the effort is worthwhile: better data quality, less drop-outs and less effort during the analysis and interpretation. Especially under time pressure a pretest should not be missed out on.

Useful hints on pretesting and further test methods are described, for example, in the text Kognitive Interviews by Prüfer and Rexroth (2005) from the ZUMA How-to-Reihe, Nr. 15. (German)