How to Conduct an Online Survey
As I mentioned in last week’s survey invitation, when you’re trying to build a business, blog, non-profit, or pretty much any venture, regularly checking in with your peeps is an important way to make sure you know who they are and what they want.
This is especially important when it comes to a) transition points in the growth of your community, or b) product development at any time. Surveys allow you to take the pulse of a large group of people in a short period of time– and because of how sampling works, you don’t actually need to hear from all of them to know what most of them are thinking.
Keep in mind that most readers or customers are usually in the Silent Majority. As I’ve said before, well over 95% of them (sometimes 99%+) do not usually feel the need to participate in public responses like website comments or even one-on-one correspondence through email exchanges. If you want to know what the Silent Majority thinks, a survey is a great way to bring people out of the woodworks.
Granted, surveys also attract a self-selecting group of respondents, so there is still a big “Really Silent” Majority out there, but short of stalking people on an individual basis, a survey is the best you can get. The rest of this article will explain exactly how to set up an online survey– it’s actually quite easy– along with a few points on data interpretation that I’ve learned through trial and error.
Logistics and Design
The logistics to setting up a survey are very simple. To manage the data and make it easy for the respondents, you need to either use an online service of some kind, or hack it out yourself.
I’ve used Survey Monkey for years and am happy to recommend them without any benefit for me. It costs $19.95 a month for the premium version (the free version is OK for testing but then gets pretty limited) and is worth the investment if you’re in it for the long-term. There are some free alternatives, but personally I am happy to pay $19.95 a month for a service I know well. Some of the free options have intrusive ads on their survey forms, or redirect users to the survey service homepage at the end. Even though it may be a small thing, I never want to abuse the trust of someone who wants to share their input with me. Therefore I use Survey Monkey.
If you want to do it on the cheap, you could also hack together a survey using Google Forms. I’m not the best hacker and I already know another system, so I personally don’t have any need to change anything.
After you decide on how to collect the results, you need to set up the actual survey. Here’s where you need to think very carefully about what you really need to know from the respondents. You don’t want to waste anyone’s time by asking for irrelevant information, but you also don’t want to miss anything important.
Whatever you decide, I recommend including variations of the following two questions:
1) In 1-2 sentences, why do you read the Art of Non-Conformity site?
2) What do you think is the #1 thing I can help people with?
Obviously, you adjust these questions based on whatever it is you do. Also, if you have a few ideas of upcoming projects in mind, it’s good to test them out in the form of a ranking system. I usually phrase the question like this:
Here are a few things I’m thinking about doing in the next few months, but I could be totally wrong! Please let me know what you think of each idea.
Idea #1
Idea #2
Idea #3
etc.
I then apply a simple ranking scale to each idea and ask respondents to side with their first impression. The ranking scale I use usually consists of answers like “I love it! You should do it; Sounds Interesting; Would Need to Hear More; and It’s Not for Me.”
Generally speaking, I like to keep surveys brief – less than 10 questions or so overall. To get more responses, ask less questions. To get more detailed responses, ask more questions. It’s up to you, but just make sure that whatever you ask is something you actually need to know about.
Once your survey is ready to go in Survey Monkey or whatever platform you use, you’ll get a link to share with your readers, customers, or community. Be sure you test it in a couple different browsers to make sure it’s working, then send it out to your group with a polite request asking for input.
Limiting the Responses
I limited the first AONC survey last year to 250 responses and turned it off after two days. I tried to limit the most recent one to 500 responses, but had to turn it off after just a couple of hours – even on Labor Day, when a lot of people in North America were away from their email. After setting up the survey, I was a bit late getting back to my computer and ended up receiving more than 750 responses. (You guys are so fast! Sorry to everyone who got there afterwards.)
I deliberately limit the responses because a) much of the feedback will overlap after a while, and b) I want to pay close attention to what each person says. With more than a few hundred responses, I simply won’t be able to do that. My philosophy is that if someone takes the time to think about the questions and give me their detailed feedback, I don’t want to just skim their comments; I want to pay close attention to each of them.
Also, I don’t mention this in advance, but I try to send a short thank-you note to each person who takes the survey, and I couldn’t easily do that with thousands of responses. I know that this kind of task is inefficient and difficult to scale, but I also know that each person took a few minutes to share their input, so why can’t I take a few hours to write everyone to tell them I appreciate it?
Interpreting the Data
Once you get a good range of responses (I like 100+, although if you have a smaller group you can still get valid feedback), you want to review the data and find out what people have to say. Here are a few important notes on that process.
You’ll likely learn things you never knew before. One of the most interesting and insightful outcomes from a good survey is when you find yourself surprised by some of the responses. For me, one of the most important things I have consistently heard is that people read AONC because it helps them to not feel alone. Over and over I hear variations on this theme. I never could have expected this kind of feedback in the beginning of the project, but it’s now one of the things I’ve come to identify with most closely. As long as I’m connecting on that level with some people, I know I’m on the right track– and when you do your own survey, you may find something of your own that you never expected on your own.
Some feedback will be contradictory. If I ask about products, I usually gauge the initial interest level by listing a few things I’m thinking about building over the next few months. Inevitably someone will think a particular idea is the greatest idea I’ve ever had while someone else will think it’s terrible. The truth is probably somewhere in between, or they could both be right in their own way– for one person, it’s a great idea, and for another, it’s terrible.
Often this kind of feedback arrives back-to-back, which is interesting to compare. Again, it doesn’t mean they are wrong, or that I’m wrong. It just means that different ideas will resonate with different people.
When you take a survey, look for the trends. What jumps out from several hundred responses? What are people most interested or excited about? The trends are the obvious observations that come from looking at the data in spreadsheet form, and can usually give you an idea of what to do next.
To take it further, though, you’ll want to avoid looking strictly at the majority/minority data points. What I mean by this is that many projects are best suited to a passionate-but-small base of users. For example, when I asked about topics for future Unconventional Guides, the one on Frequent Flyer Miles was rated in the middle instead of near the top. Despite the average ranking, I know that people interested in earning miles and redeeming them for free travel are quite a passionate group. If I can reach that group, I won’t hesitate to proceed with the project even though I also know it won’t be a good fit for everyone.
Asking for name + email address is good. The first time I took a survey, I didn’t ask for any personal information on the grounds that more people would be willing to respond. I then found that the problem with complete anonymity is that if you want to follow-up with someone after reading their comments, you have no way of doing so. Asking for some basic information also ensures that you’re getting respondents who care enough about what you’re doing that they don’t hesitate to share their contact details. Of course, none of the information is shared or used for any other purpose.
In the end, you have to do what you think is right. Obviously, I’m a big fan of surveys, but in the end, the group opinion isn’t everything. I don’t think your course should be set for you by other people, even if the other people are like-minded or agree with you. Among other things, you have to have your own motivation for doing whatever it is you want to do. If your motivations are based strictly on the preferences of someone else, you’ll run the risks of burnout, boredom, unhappiness, or simply being less than you could be otherwise.
What I mean by this is that you can’t simply ask and let people tell you what you should be doing in your project. Feedback is great, but in the end you are the one responsible for the outcome. Also, sometimes what people want and what they say they want are two different things– but that’s a topic for a whole different article!
Wrap-Up
I’ve always liked this quote from the mad hatter in Alice in Wonderland: “If you don’t know where you’re going, any road will take you there.”
Granted, a survey won’t necessarily help you know where to go if you don’t have at least a general idea in the beginning. The individualist in me would assert that you can chart your course without feedback from anyone and you’d be totally fine. The community-builder in me, however, knows that your work will have a deeper impact if you take the pulse of the group from time to time.
I hope this is helpful to those of you who are also trying to build something bigger than yourself. Wherever you’re going, stay the course!
***
Also Read:
Business, Blogging, and Broken Windows
Product Launch 101
Unconventional Business Ideas
What Makes a Community?
###
Survey Image by Kim Pierro
23 Comments
Hi Chris,
This is a good overview of conducting online surveys. A good portion of my work over the last ten years has been devoted to designing, administering and analyzing surveys–both online and paper-based–so I hope the following considerations (some learned through that great teacher called experience) will also be useful to any of your readers who might be considering sending a survey to their readers or clients.
First, it’s true that people appreciate a thank you for taking the time to participate in a survey (Some web-based survey tools will actually allow you to send out thank-you notes at the end of the data collection period–I can’t remember if the professional version of Survey Monkey includes that option.), and from my own experience, people also like to know that the data was actually looked at and put to good use. One way to do this might be to write up a short summary/highlights of the overall results or themes that emerged and post that on your website for the readers. (Just make sure that identifying information is removed or disguised–especially if your sample is really small.)
Second, I definitely agree with you Chris about keeping surveys as brief as possible–and not just because excessively long surveys will irritate your readers/survey respondents. The more questions you have, the more data you will have to analyze, so it’s important to give some thought beforehand as to how you want to use the data and how much time you want to spend analyzing your survey data. A lot of open-ended, text responses, while providing lots of really rich information, can be very time-consuming to analyze and summarize. If there are ways to convert open-ended questions to closed-ended, single or multiple response items (with space available to describe “other”) I’d recommend going that route. Cross-tabulating the responses to various survey items can also provide you with some really useful information to guide your decision making. (e.g., Are certain categories of readers more interested in certain products or topics, etc.
Last but definitely not least, for any of your readers in Canada (or countries other than the U.S.A.) who have micro or online businesses and are considering using online survey tools like Survey Monkey, it’s a good idea to look into (1) the FIOPOP (Freedom of Information and Protection of Privacy) regulations in your province or country, as they apply to the private sector and (2) find out where the server that holds the information is located and who else might have the legal right to access that data–especially if you are asking for identifying information from your survey respondents. It’s a big no-no in the public sector to have information of a personal or identifying nature (name, email address, city, etc) stored on servers outside of Canada and can result in a huge fine for the organization. I’m not sure what the regulations are for the private sector, but whether or not the same standards apply legally, I would argue that if your readers/clients are kind enough to provide you with identifying or other personal information about themselves or their companies, then it becomes your responsibility to ensure that the information is safe-guarded in terms of privacy.
Hi Chris,
I’ve just finished your manifesto on overnight success where you speak about online surveys.
This post is helpful and a good complement to the manifesto. I have a blog on management and I’m thinking about launching my first e-book. I’ll follow your advice on survey for sure !
Thanks for the manifesto (I’m ready for world domination 🙂
I think that survey gizmo would be a better fit than survey monkey. Having used both I now use survey gizmo. I do not work for them just a happy user. Hope this helps
I meant to say, i was considering creating a video screencast on how to create online surveys with survey monkey…. there’s really nothing better than getting real, honest feedback from your audience, i’ve found – even when it’s critical!
I’ll have to take a look at Survey Monkey, as I’m planning on running a survey by my readership in the very near future. This post couldn’t have come at a better time!
It’s funny how collecting data was always my least favorite part of Sociology class back in the day, but not I’m actually looking forward to it. I guess the automation behind online survey-conducting helps ease that strain a bit!
I did run an unofficial survey a couple of months ago where I polled people about changing my byline. I think I was keen to change it but in the end most people liked it the way it was!
I’ll be coming back here when I can squeeze in the time to run another bigger and better survey. At the moment working out how to make a podcast and create a video seem to be at the top of my list.
So much to learn, so little time! Thanks for sharing everything you’ve learned so far, it really helps:)
Thanks, Chris
Great info on how to interpret the info. I agree, in the end you must follow your inner guide.
Rasheed
@Sue, @MonsieurJ, @Jonathan,
Thanks so much for your comments.
Sue, that’s almost a post of its own! Good points, thank you.
Chris, how would you compare the value of survey feedback to direct contact to your most loyal customers?
For most blog business models, there is a hardcore group who buy the package deal and regularly participate at the membership section (or whatever). It’s often real easy to get detailed feedback from such people.
What do you think? Is it worth interviewing the loyal customers… or better to survey the folks who are still sitting on the proverbial fence a little? 🙂
… I just read your other recent post that criticises membership sites etc making my comment (above) a bad example.
What I was attempting to ask: Is it better to survey more talkative fans/customers or total newcomers to your site/following?
Chris this was helpful to me. I’m just in the very beginning stages of building my blog. And I agree with others have said in your surveys about feeling connected with you. I can relate with you as you are in the role of storyteller and mentor.
Going to check out your other recommended links now!
@ Chris — Let’s just say it’s a “practice” blog post for when I eventually launch my own blog. 😉 Sorry if it was a bit long.
@ Colin–Yup, the automated data collection process with online surveys makes it much easier and greatly reduces the likelihood of data entry errors. It wasn’t the data collection process that used to get to me as student–it was entering the data into a text file that was then uploaded into (Ready for this?) a mainframe version of SPSS. It was a real pain to deal with and enough to make one statistics and computer phobic for life. 😉
Chris, if you really want to have information from the really silent majority then try to for example give away your discontinued guide about airfare for filling in the survey. 😉 Motivation should not be underestimated.
@Peter,
I don’t think all membership sites are evil, so it wasn’t a bad question at all. The answer to the question depends on what kind of data you are looking for. It’s true that folks who fit into the “true fan” category will often provide different feedback than general followers.
All things being equal, I would value the feedback from true fans more, but I also like to hear what other people are saying too.
@Michal,
I don’t like to incentivize people to respond to surveys. Enough people are usually happy to do it that you don’t need to bribe the others.
The timing of this post couldn’t have been any better. Just this past week I’ve thought of a few things I would like to ask my blog readers but I’ve never done an online survey before.
I really appreciate all your pointers Chris, they will help me greatly!
Cheers,
Dayne
Chris, a question for you–how do you get genuinely useful feedback if you limit it to just the first batch of responses?
Isn’t it sort of like comments, in the sense that the ones that tend to appear first (the ones that are posted within minutes of a new post going up) are usually rushed, arbitrary and not all that thought out, while the comments that come in after a few hours ARE more thought out and thus more valuable?
By that logic, you might want to skip the first 100 responses, and read the next 250. Curious if you would consider doing that at all.
Dan
Casual Kitchen
Good question. The main reason why I limit the responses is not for quality of data, it’s so I won’t get overwhelmed and I want to honor everyone’s time. If I asked people to take a survey but then didn’t pay close attention to their feedback, I don’t think that would be very nice.
Keep in mind also that I’m not highly scientific with this (or anything!). I haven’t found that the first comments are rushed– it seems that most are thoughtful.
Your post came at the right time. I am mulling over the idea of including a survey for my upcoming book. The reasons you list are pretty valid. Thanks Chris.
I also recommend Google Forms. It’s a great little web app that does something quickly and easily that used to require a web developer.
This post was so helpful. THanks for offering up your expertise!
“You don’t want to waste anyone’s time by asking for irrelevant information, but you also don’t want to miss anything important.”
This is such an important point. When in University, I use to hate the course surveys they would hand out at the end of each semester.
Keep up the great posts. I really enjoy post that give advice on how to get by on your own terms.
Thank you for this article. It looks like I’m a bit late to the game. I’ve recently created a survey on Surveymonkry for a small business I am starting. I emailed my contacts and posted it on facebook which has resulted in about 100 responses.
I’m curious if there are other places to post the survey to get the attention of more than my innercircle?
Thank you for a great article!
Your comments are welcome! Please be nice and use your real name.
If you have a website, include it in the website field (not in the text of the comment).
Want to see your photo in the comments? Visit Gravatar.com to get one.