articles: 94

This data as json

rowid	title	contents	year	author	author_slug	published	url	topic
94	Using Questionnaires for Design Research	How do you ask the right questions? In this article, I share a bunch of tips and practical advice on how to write and use your own surveys for design research. I’m an audience researcher – I’m not a designer or developer. I’ve spent much of the last thirteen years working with audience data both in creative agencies and on the client-side. I’m also a member of the Market Research Society. I run user surveys and undertake user research for our clients at the design studio I run with my husband – Mark Boulton Design. So let’s get started! Who are you designing for? Good web designers and developers appreciate the importance of understanding the audience they are designing or building a website or app for. I’m assuming that because you are reading a quality publication like 24 ways that you fall into this category, and so I won’t begin this article with a lecture. Suffice it to say, it’s a good idea to involve research of some sort during the life cycle of every project you undertake. I don’t just mean visual or competitor research, which of course is also very important. I mean looking at or finding your own audience or user data. Whether that be auditing existing data or research available from the client, carrying out user interviews, A/B testing, or conducting a simple questionnaire with users, any research is better than none. If you create personas as a design tool, they should always be based on research, so you will need to have plenty of data to hand for that. Where do I start? In the initial kick-off stages of a project, it’s a good idea to start by asking your client (when working in-house you still have a client – you might even be the client) what research or audience data they have available. Some will have loads – analytics, surveys, focus groups and insights – from talking to customers. Some won’t have much at all and you’ll be hard pressed to find out much about the audience. It’s best to review existing research first without rushing headlong into doing new research. Get a picture of what the data tells you and perhaps get this into a document – who, what, why and how are they using this website or app? What gaps are there in existing research? What else do you need to know? Then you can decide what else you need to do to plug these gaps. Think about the information first before deciding on the methodology. The rest of my article talks mostly about running self-completion online surveys. You can of course do face-to-face surveys, self-completion written questionnaires or phone polls, but I won’t cover those here. That’s for another article. Why run a survey? Surveys are great for getting a broad picture of your audience. As long as they are designed carefully, you can create an overview of them, how they use the site and their opinions of it, with an idea of which parts of this picture are more important than others. By using a limited amount of open-ended questions, you can also get some more qualitative feedback or insights on your website or app. The clients we work with surprisingly often don’t have much in the way of audience research available, even basic analytics, so I will often suggest running a short survey, just to create a picture of who is out there. OK, what should I do first? Before you rush into writing questions, stop and think about what you’re trying to find out. Remember being in school when you studied science and you had to propose a hypothesis? This could be a starting point – something to prove or disprove. Or, even better, write a research brief. It doesn’t have to be long; it can be just a sentence that encapsulates what you’re trying to do, like a good creative brief. For the purposes of this article, I created a short, slightly silly survey on Christmas and beliefs in Father Christmas. My research brief was: To find out more about people’s beliefs about Father Christmas and their experiences of Christmas. Inevitably, as you start thinking of what questions to ask, you will find that you go off at tangents or your client will want you to add in everything but the kitchen sink. In order for your questionnaire not to get too long and lose focus, you could write lists of what it is and what it’s not. This is how I’d apply it to my Christmas questionnaire example: What it is about How people communicate with Father Christmas If someone’s background has affected their likelihood of believing in Father Christmas What it is not about What colour to change Father Christmas’s coat to Father Christmas’s elves Let’s get down to business: the questions. Kinds of questions There are two basic kinds of questions: open-ended and closed. Closed questions limit answers by giving the respondent a number of predefined lists of options to choose from. Typically, these are multiple-choice questions with a list of responses. You can either select one or tick all that apply. Another useful type of closed question I often use is a rating scale, where a respondent can assess a situation along a continuum of values. These can also be useful as a measure of advocacy or strength of feeling about something. There is a standard measure called the Net Promoter score, which measures how likely someone is to recommend your product or service to a friend or acquaintance. It’s a useful benchmark as you can compare your scores to others in a similar sector. Open-ended questions often take the form of a statement which requires a response. Generally, respondents are given a text box to fill in. It’s useful to limit this in some way so that people have an idea of how long the expected response should be; for example, a single line for an email address (Q18), or a larger text area for a longer response (Q6). If you plan to send your survey out to a large number of people, I would suggest using mostly closed questions, unless you want to spend a long time wading through comments and hand-coded responses. I’d always advise adding a general request at the end of a survey (‘Is there anything else you’d like to tell us?’). You’d be surprised how many interesting and insightful comments people will add. There are times when it’s better to provide an open-ended text box rather than a predefined list makes assumptions about your audience’s groupings. For example, we ran a short survey for our Gridset beta testers and rather than assume we knew who they were, we decided to ask an open-ended question: “What is your current job title?” The analysis took quite a bit longer than responses using a predefined list, but it meant that we were able to make sure we didn’t miss anyone. And next time we run a survey for Gridset, I can use the responses gathered from this survey to help create a predefined list to make analysis easier. What to ask The questions to ask depend on what you want to know, but your brief and lists of what the survey is and isn’t should help here. I always ask the design team and client to give me ideas of what they are interested in finding out, and combine this with a mix of new and standard questions I have used in other surveys. I find Survey Monkey’s question bank a very useful source of example questions and help with tricky wording. I always include simple demographics so I can compare my results to the population at large or internet users as a whole – just going on age, gender and location can be quite illuminating. For example, with the Christmas survey, I can see that the respondents were typical of the online design and dev community, mainly young and male. If appropriate, I add questions on disability, ethnic background, religion and community of interest. Questions about ethnicity, religion, sexual preference, disability and other sensitive subjects can feel awkward and difficult to ask. This is not a good reason to not ask them. Perhaps you’re working for a public sector client, like a local council, so it’s likely you will need to consider groups of people who maybe under-represented, who may have differing views to others, or who you need to look at specifically as a subset. How to ask Although they may seem clunky and wordy, it’s often best to use the census wording or professional body wording for such demographic questions. For example, I used the UK census 2011 wording for Wales on my Christmas questionnaire in my questions on religion [PDF] (Q16) and ethnicity [PDF] (Q17). I had to adapt them slightly for the Survey Monkey format – self-completion online, rather than pen and paper – which is why “White Welsh” came up as the first option for the ethnicity question. For similar questions for US audiences, try the Census Bureau website. When conducting a survey for a project that has a global audience, you need to consider who your primary audience is. For example, I recently created a questionnaire for a global news website. A large proportion of its audience is based in the USA, so I was careful to word things in a way Americans would find familiar. I used the US ethnic background census question wording and options, and looked at data for US competitor news websites to decide which to include. You should also consider people whose first language isn’t English. Working as an audience researcher at BBC Wales, every survey we did was bilingual. I’ve also recently run a user survey in Arabic using Google Forms. During this project, we found that while Survey Monkey supports different languages, including Arabic, the text ran left to right with no option to change it to right to left – an essential when it comes to reading Arabic! If research is a deliverable in a client project, and you know you’ll need to conduct it in a foreign language, always build in extra time for translation at both the questionnaire design and analysis stages. Make sure you also allow for plenty of checks. In this case we had to change to Google Forms after initially creating our survey with Survey Monkey to get the functionality we needed. Look and feel Think about the survey as another way your audience will experience your brand. Take care getting the tone of voice right. There are plenty of great articles and books out there about tone of voice – try Letting Go of the Words by Ginny Redish for starters, or Brand Language by Liz Doig. The basic rule of thumb is to sound like a human, and use clear and friendly language. If, like me, you are lucky enough to work with journalists or copy editors, you should ask for their help, particularly in the preamble, linking text and closing statements. I find it helpful to break my questions down into sections and to have a page for each. I then have an introductory piece of text for each section to guide the respondent through the survey. You should also make sure you check with your designers how your survey looks – use a company logo and branding, and make the typography legible. Many survey apps like Survey Monkey and Google Forms have a progress bar. This is helpful for users to see how far through your survey they are. I generally time the survey and give an indication in the preamble: “This survey will only take five minutes of your time.” You also need to think about how you will technically serve the questionnaire. For example, will it be via email, social media, a pop-up or lightbox on your website, or (not recommended but possible) in an ad space? Ethical considerations Something else to think about are any local laws that govern how you collect and store data, such as the Data Protection Act in the UK. As a member of the Market Research Society, I am also obliged to consider its guidelines, but even if you’re not, it’s always a good idea to deal with personal data ethically. If you collect personal data that can identify individuals, you must ask their permission to share it with others, and store it securely for no longer than two years. If you want to contact people afterwards, you must ask for their permission. If you ask for email addresses, as I did in question 18, you have a ready-made sample for a further survey, interviews or focus groups. Remember, you shouldn’t survey people under sixteen years old without the permission of their parents or legal guardians, so if you know your website is likely to be used by children, you must ask for verification of age early on, and your survey should close someone answers that they are under sixteen. The ESOMAR guidelines for online research [PDF] are well worth reading, as they go into detail about such issues, as well as privacy guidelines – using cookies, storing IP addresses, and so on. Tools Unless you work in-house and have proprietary software, or at a market research agency and you’re using specialist software such as Snap or IBM SPSS Statistics (previously just SPSS), you will need to use a good tool to run your survey, collect your responses and, ideally, help with the analysis. I like Survey Monkey because of the question bank and analysis tools. The software graphs your results and does simple cross-tabbing and filtering. What this means is you can slice the data in more interesting ways and delve a bit deeper. For example, in the Gridset questionnaire I mentioned earlier, I cross-tabbed responses to questions against whether a person worked in-house, for an agency or as a freelancer. Other well known online tools that I also use from time to time are Wufoo and Google Forms. Smart Surveys is a similar service to Survey Monkey and it’s used by many leading brands in the UK. Snap Surveys mentioned above is a well-established player in the market research scene, used a lot for face-to-face surveys and also on tablets and smartphones. Analysis Analysis is often overlooked but is as important as the design of the questionnaire. Don’t just rely on looking at the summary report and charts generated as standard by your form or survey software. Spend time with your data. Spend at least a week now and then if you can, looking at the data. Keep coming back to it and tweaking or cutting it a different way to see if there are any different pictures. Slice it up in different ways to reveal new insights. Here is the data from my dummy survey (apart from the open-ended responses). For open-ended questions, you can analyse collaboratively. Print and cut out the open-ended responses and do a cluster analysis or affinity sort with a colleague. Discussing the comments helps you to understand them. You will also find the design team are more likely to buy into the research as they have uncovered the insights for themselves. Always make sure to treat open-ended responses sensitively and don’t share anything publicly in a way that identifies the respondent. Write a report Never hand over a dataset to your client without a summary of the findings. Data on its own can be skewed to suit the reader’s needs, and not everyone is able to find the story in a dataset. Even if it’s not a deliverable, it’s always a good idea to capture your findings in a report of some sort. Use graphs sparingly to show really interesting things or to aid the reader’s understanding. I have written a quick dummy report using the data from the Christmas questionnaire so you can see how it’s done. I highly recommend Brian Suda’s book A Practical Guide to Designing with Data for tips on how to present data effectively, but that’s a subject that benefits a whole article (indeed book) in itself. I am not a designer. I am a researcher, so I never write design recommendations in a report unless they have been talked about or suggested by the designers I work with. More often, I write up the results and we talk about them and what impact they have on the project or design. Often they lead to more questions or further research. So that’s it: a brief introduction to using questionnaires for design research. Here’s a quick summary to remind you what I have talked about, and a list of resources if you’re interested in reading further. Top 10 things to remember when using questionnaires for design research: Start by auditing existing research to identify gaps in data. Write a research brief. Work out exactly what you’re trying to find out – what is the survey about, and what is it not about? The two basic kinds of questions are open-ended and closed. Closed questions limit responses by giving the respondent a number of predefined lists of options to choose from (multiple choice, rating scales, and so on). Open-ended questions are often in the form of a statement which requires a response. Always ask one at the end of a questionnaire. Always include simple demographics to enable you to compare your sample against the population in general. It’s best to use official census or professional body wording for questions on ethnicity, disability and religion. Be sure to think carefully about your tone of voice and the look of your questionnaire. Pay attention to guidelines and laws on storing personal data, cookies and privacy. Invest plenty of time in analysis and report writing. Don’t just look at the obvious – dig deep for more interesting insights. Some useful resources for further study Online research Design Research: Methods and Perspectives edited by Brenda Laurel Online Research Essentials by Brenda Russell and John Purcell Handbook of Online and Social Media Research by Ray Poynter ESOMAR guidelines for online research [PDF] Online questionnaires Market research books on questionnaire design Using Questionnaires in Small-Scale Research: A Beginner’s Guide by Pamela Munn Questionnaire Design by A N Oppenheim Developing a Questionnaire by Bill Gillham	2012	Emma Boulton	emmaboulton	2012-12-14T00:00:00+00:00	https://24ways.org/2012/using-questionnaires-for-design-research/	business