Jeffrey Saltzman, CEO of OrgVitality, has seen a lot in his experience as an I/O consultant. Here's one particularly interesting case:
My office phone rang, and a distraught employee from an organization in the middle of survey administration whispered, “I don’t know what to do. My boss is telling me how to fill out the survey”. When I asked why she was whispering she said she was hiding from her intimidating boss in a closet. I said, “the first thing you should do is get out of the closet”. I talked to her for a while and calmed her down. The boss was subsequently fired and the employee was free to fill out her survey as she liked. This was a pretty clear-cut case of someone trying to manipulate survey responses and create fraudulent data.
While it is extremely rare for survey responses to be manipulated like this, it does happen. Thankfully there are many indicators and analyses you can perform to help you determine if there is any fraud going on in your survey responses:
- The Response Rate
If you examine the response rate at the department level, and there’s an outlier for one specific department or business unit, it’s worth a follow-up. For example, if the organization overall sees a fairly consistent response rate across departments of roughly 65 or 70%, but one department has above 90%, it may be a sign that something is off. It doesn’t necessarily mean there is fraud, but it is worth a conversation.
- Data favorability
Similarly, if all departments are fairly consistent with their favorability scores but one department is significantly higher, that is also worthy of a follow-up.
- Response patterns
If you see survey responses with all 5’s or all 1’s on a 5-point scale that can be indicative of someone not reading the items and just checking the boxes down the line. At OrgVitality we try to guard against this by having automatic prompts that detect unusually high responses and nudging the user to consider whether their responses are accurate.
- Demographics
If the known demographics of a department don’t align with the demographics of the survey responses, it’s worth following up. For example, if you generally have a 50% split between men and women but the demographics of respondents reflect something much different, there may be fraud. This of course works only when the demographics are asked for on the survey and are not pre-loaded. Alternatively, this can mean people are trying to disguise themselves out of fear or are just concerned about confidentiality.
- Reverse scored items
Organizations often use reverse scored items for many reasons, but an additional benefit is the ability to detect if users are simply responding without reading the items; in this case, the response pattern doesn’t make sense.
- Lie scale
There are a few ways to build in lie detection. One way is to ask a few of the same questions twice, but one of the instances for each item is reverse scored. You then can check for consistency in responses. For instance, if someone is both positive and negative on advancement opportunities, you know something is up.
- Comments
Typically, comments are generally more negative than the data suggests they should be. That is because people tend to comment more when something is bothering them, rather than when everything is fine. Read all your survey comments; if they don’t seem aligned with the response pattern it is worthy of follow-up. If you’re overwhelmed with the sheer number of comments, tools like OrgVitality’s UQ can help by automatically sorting and filtering the comments.
- Survey timestamps
If you are tracking timestamps regarding how long it took to complete a survey you can examine the data for unrealistic completion times. We know that people in general will complete 4-5 items per minute. If a 50-item survey is completed in 2 minutes it would be all but impossible for someone to actually read the items.
- Asking
The most direct way to determine if there is fraud in your survey data is to ask. If you have concerns about whether someone else is filling out the survey for their employees, ask potential respondents if they completed a survey. Most people will tell the truth even if they have been instructed otherwise by a supervisor. The question then becomes who filled out the survey.
If you are concerned that there may be unsavory practices happening at your organization and want some help, contact us for more guidance.