Have you ever conducted a survey and struggled with what to do with the responses to open-ended questions? As someone who has been in the survey field for over 10 years, I have too struggled at times to more efficiently analyze these valuable pieces of information. Below, are a few approaches that can be used, some of which you may have heard before and others which may be new.
- Live Coding – you can train coders to read each comment and assign them to categories which are developed from reading a sample of comments.
- Key Word Search – Manual process where key words are used to identify comments. For example, you can conduct a key word search for all comments related to “benefits”.
- Self-code – Each respondent codes their own response directly on the survey (or through the web survey form). Coding often corresponds with the survey categories.
- Computational linguistics – a less labor intensive alternative that makes use of advanced linguistic theory and technology that pulls out key themes and then categorizes open ended responses.
Depending on your resources and the volume of comments you are analyze each approach has its benefits and drawbacks.
Live Coding: Given a fixed or limited resource pool and large volume of comments (i.e., thousands), live coding can be prohibitive.
Key Word Search: With limited resources and a basic spreadsheet or word processing program you can conduct key-word searches quite efficiently but will have less robust information (and will still need to spend time tabulating the results of the key word searches).
Self Coding: If your web survey technology allows you to do so, this can be an efficient means of coding open ended comments. Two drawbacks come to min 1) comments may span more than one category and self coding usually won't capture this 2) you will need to come up with a list of categories for your respondent to select. Unless you have a sense from previous efforts, your list may not capture all of the potential categories.
Computational linguistics: This approach is an efficient and robust method for analyzing open ended comments. It can work for questions with as few as a few hundred responses to tens of thousands. It still takes time to set up and conduct the analysis, so the cost-benefit analysis for smaller data sets is a potential issue. For larger data sets, the set up time can take a little more time (e.g., spell check, formatting, etc). There's also the investment in the software which depending on the provider can be expensive.
So, depending on your situation, each approach has its merits and limitations. However, if analyze responses to open ended questions helps you and your management team (or client if you are a consultant) act on the findings, then your investment is justified.
If you'd like to respond to this post, please feel free to do so my clicking on the link below. To see an example of computational linguistics, click on the link in our link section titled "Harnessing the Power of Open-Ended Comments".
Posts
