We've been asked by a few prospective clients about the value of benchmarks, so I thought I would share our Benchmark Tutorial here with you.



1. Client Additive Benchmarks: This approach is based on the compilation of survey results from the clients of a given consulting firm. These benchmarks may contain responses from hundreds or even thousands of employees across hundreds of companies. The sheer number of responses can be impressive but also potentially misleading. Here’s why: Unless each client uses a core set of survey questions furnished by the vendor, the database likely will vary with respect to the number of companies contributing to each benchmark question. So, whereas your survey responses to one question may be compared with results from 13 companies and 34,000 respondents, responses to another question may only be compared with results from only 3 companies and 800 responses. So if your results differ from benchmark findings, it is difficult to determine whether the discrepancy is attributable to a difference between your company and the external environment or a difference between one benchmark and another. Additionally, there needs to be standardization and consistency in the order and method in which survey questions are presented, and in the scales that are used. Slight differences in these factors also can impact the integrity of the benchmark and the accuracy of the comparison (i.e., is the difference between your score and the benchmark a real difference or does it relate to how the questions may have been asked?). Client additive benchmarks have an additional limitation: your results are being compared with those of a single consulting firm’s client list. This list excludes many companies that may be doing their own survey on their own or who may have decided to hire a different firm.

2. Workforce Studies: This approach to survey benchmarks evolved as survey vendors realized the drawbacks in quality and the high expense of developing and maintaining quality client additive benchmarks. Workforce studies are similar to public opinion polls conducted by newspapers (e.g., the USA Today Poll) as well as research conducted by market research firms. A set of questions is administered to a sample of the workforce, typically segmented by industry and/or geography. Unlike the respondents in client additive benchmarks, these respondents are individuals who work in various organizations. These benchmarks do not reflect the responses of groups of employees who work for a single organization, but rather those of individual employees who work for individual companies. In other words, a workforce study database could have 1,000 respondents from 1,000 different companies. Although claims of statistical representation can be made, there is less control over what types of companies go into the database. Moreover, there is the question of who is actually completing the survey. Respondents usually are offered an incentive (such as a monetary reward, gift certificate, or entry into a raffle) to encourage participation in multiple surveys. Furthermore, comparing the results of your survey, which is sponsored by an employer, with those of a workforce-study survey, which is sponsored by a polling or market research firm, also may be questionable. Employees typically want to help their employers by sharing important feedback. Respondents to workforce surveys are not participating under the same assumptions, and thus comparisons also may be tenuous.

3. Consortium Benchmarks: The third type of benchmark is the consortium benchmark. In our opinion, this is the most valuable, accurate, and worthwhile external benchmark for comparing your survey results. These benchmarks are based on results from a collection of companies that share survey results and best practices in survey research. Membership is by application, and member companies must commit to the terms of membership, which include asking a minimum set of questions, attending meetings, and contributing best practices. The Mayflower Group (www.mayflowergroup.org) is one of the oldest survey consortia; it comprises large, multinational organizations with established survey programs. Other groups also have been formed, such as the Information Technology Survey Group (ITSG; www.itsg.org) and MIDAS, a similar group for the financial services.

One Final Note: We hope this primer has helped you understand the different types of survey benchmarks available today. Of course, there’s one survey benchmark we didn’t discuss – and that is your own survey data. Looking at a specific group within your organization relative to other groups, as well as in relation to your previous survey results, often can provide the most useful and relevant comparisons.

Corporate Social Responsibility (CSR) is becoming more prevalent in our global economy. The other night, a TV commercial informed viewers that a car company is operating a manufacturing plant that has achieved zero landfill status - nothing from its manufacturing efforts goes into a landfill. Going carbon-neutral is also gaining a lot of press. Several high tech companies are making a commitment to offsetting the carbon emissions they produce each year via their technology and/or employee commuting. They do this by supporting the planting of trees, investing in clean technologies, promoting the use of mass transit and car pooling or buying credits to offset the emissions from their business operations. Some organizations have also created foundations that are dedicated to promoting corporate social responsibility and philanthropy and are giving back to the community.

If your organization is thinking about embarking on or enhancing an existing corporate social responsibility program you may want to reach out to your employees prior to doing so. This will help you accomplish the following:

1. Communicate to employees that CSR is valued by your organization.
2. Solicit employees' ideas on ways they would like to contribute to the organization's CSR efforts.
3. Collect useful data on employees' willingness to do so, which will help your planning efforts as well as be a selling point to prospective employees you are trying to recruit.

Here are some tips to help you with your next CSR survey.

• Make sure to collect data via a web-based survey. Using a paper survey will send the wrong message (even if it's on recycled paper).
• Encourage participation by making a donation to a charity or community organization for every completed survey or if you achieve a certain participation level.
• Consider integrating these questions as part of a larger employee survey. By doing so, you can look at the relationship between CSR and other important outcomes like engagement.
• Make sure to include relevant demographic information like work location, job level and business unit.
• Be sure to communicate back to employees on the actions taken based on the survey - for example, if you decide to implement a community out reach program based on the survey, make sure to inform employees that the decision was made based in part on employee input.
  

Here are some survey items you may want to ask in your CSR survey.

1. I am aware of our company's CSR efforts. (Agree- Disagree scale)
2. I understand how our CSR efforts support our mission and vision. (Agree-Disagree scale)
3. Where should we focus our CSR efforts? (Open- ended)
4. How likely would you be to participate in a community day held during the work week? (Likelihood scale)
5. How likely would you be to participate in a community day held on a weekend day? (Likelihood scale)
6. In what ways can we help reduce our impact on the environment? (Open-ended)
7. I would be interested in participating in a mass- transit program (Agree-Disagree scale
8. I would be interested in participating in a car pool program (Agree-Disagree scale
9. I would be interested in participating in (list other programs you are considering) (Agree-Disagree scale)

Eight Tips for Making Your Customer (and employee) Surveys more Customer-Friendly

1. Tell your customers how long it will take to complete the survey. There is nothing more frustrating than having to go through a survey of an indefinite duration. So tell customers up front approximately how long it will take them to complete the survey. If your survey has a lot of branching and can vary in length depending on previous responses, then provide your customers with a time range.

2. Use a progress bar to make it possible for customers to see how far along they are in the survey. They may not know how many questions there are in total, but the progress bar will give them a good idea as to how close to completion they are. The progress bar is an important motivating factor-"5% more and I'm done!"-and promoter of peace of mind-"I'm progressing pretty quickly!"

3. Allow customers to go back and change their answers. Sometimes, a customer may answer a question incorrectly and want to go back and change his or her answer. So make sure to select a web- survey solution that allows respondents to navigate back a page or two.

4. Ask customers to give you qualitative feedback. Don't just ask for responses to closed- ended or multiple choice questions. Be sure to include a couple of open-ended questions to give your customers the option to share ideas, suggestions or additional feedback. Make sure that the questions are targeted and avoid using the generic words "Additional Comments." Also, do not include too many open-ended questions- ideally, use 2 to 3 and a maximum of 5.

5. Give them a reason to participate. Most people are busy nowadays and unlike employees who can expect to benefit from an employee survey, customers have less to gain. So make participation worth their while. The information they provide to you is extremely valuable so be willing to provide incentives. Consider holding a raffle, offering compensation, or giving a donation to a charitable organization for every completed survey.

6. Do not require an answer for every question. While you may think that every customer will have enough information/experience to respond to every survey question, they are likely to find this requirement very frustrating.

7. Let customers take a break from the survey and return to it later. Be sure to give customers this option to save the survey and resume it later if your survey takes more than 5 minutes to complete. Most web survey programs offer this capability and it can make things easier for busy customers who may find themselves pulled in different directions throughout the day.

8. Make sure to test run your web survey prior to its release to make sure that it works. This step is pretty simple but it is often overlooked. As a traveler, business owner and consumer, I am often asked to complete surveys. The most frustrating thing is participating in a survey that does not work.

So it has been awhile since my last post. We've been busy with a few interesting projects. In late April I attended the Society for Industrial Organizational Psychology conference in New York City. Every year the conference attracts more and more attendees. As a native New Yorker, I enjoyed the ability to attend the conference and still get home each night. I attended some very interesting sessions on employee engagement, a workshop on running a thriving consulting business and a few sessions on the current state of employee surveys.

Have you ever conducted a survey and struggled with what to do with the responses to open-ended questions? As someone who has been in the survey field for over 10 years, I have too struggled at times to more efficiently analyze these valuable pieces of information. Below, are a few approaches that can be used, some of which you may have heard before and others which may be new.

• Live Coding – you can train coders to read each comment and assign them to categories which are developed from reading a sample of comments.
• Key Word Search – Manual process where key words are used to identify comments. For example, you can conduct a key word search for all comments related to “benefits”.
• Self-code – Each respondent codes their own response directly on the survey (or through the web survey form). Coding often corresponds with the survey categories.
• Computational linguistics – a less labor intensive alternative that makes use of advanced linguistic theory and technology that pulls out key themes and then categorizes open ended responses.

Depending on your resources and the volume of comments you are analyze each approach has its benefits and drawbacks.

Live Coding: Given a fixed or limited resource pool and large volume of comments (i.e., thousands), live coding can be prohibitive.

Key Word Search: With limited resources and a basic spreadsheet or word processing program you can conduct key-word searches quite efficiently but will have less robust information (and will still need to spend time tabulating the results of the key word searches).

Self Coding: If your web survey technology allows you to do so, this can be an efficient means of coding open ended comments. Two drawbacks come to min 1) comments may span more than one category and self coding usually won't capture this 2) you will need to come up with a list of categories for your respondent to select. Unless you have a sense from previous efforts, your list may not capture all of the potential categories.

Computational linguistics: This approach is an efficient and robust method for analyzing open ended comments. It can work for questions with as few as a few hundred responses to tens of thousands. It still takes time to set up and conduct the analysis, so the cost-benefit analysis for smaller data sets is a potential issue. For larger data sets, the set up time can take a little more time (e.g., spell check, formatting, etc). There's also the investment in the software which depending on the provider can be expensive.

So, depending on your situation, each approach has its merits and limitations. However, if analyze responses to open ended questions helps you and your management team (or client if you are a consultant) act on the findings, then your investment is justified.

If you'd like to respond to this post, please feel free to do so my clicking on the link below. To see an example of computational linguistics, click on the link in our link section titled "Harnessing the Power of Open-Ended Comments".

Our recent newsletter features some tips on sharing survey feedback with employees from communication expert Liz Guthridge. To access these tips, click here. To learn more about Connect Consulting Group, visit http://www.connectconsultinggroup.com.

David Youssefnia will be presenting on Actionable Employee Surveys at the New York State SHRM conference at the Turning Stone Hotel and Casino on July 24, 2007. For more information about the conference, click here.