As some of you may know, Dean Gayeski has asked us to consult on a questionnaire that will be distributed to a sample of media execs that attempts to get a read on how valued/important 'innovation' is in the communications field.
Below is a link to a pilot questionnaire that she distributed to Park School advisory board members and some high-profile alumni last year.
Our task is to take a look at the survey and to improve it so that it's ready to go into the field.
Please examine the questions and keep in mind what we've already talked about in terms of the "DOs and DON'Ts" of creating a questionnaire.
Our finished questionnaire should have the following:
- an introduction to the survey with a brief explanation of what we're trying to measure
- a few softball questions to ease them into answering
- transition statements from one group of questions to the next
- 5-point Likert scale items, instead of the 3-point questions used. A Likert scale is a five-point scale that ranges from Strongly Agree - Agree - Neither Agree nor Disagree - Disagree - Strongly Disagree.
- closed-ended questions wherever possible (open-ended questions make it extremely difficult to report summary data).
- controversial questions should come near the end of the questionnaire (if you ever want to make sure that NO ONE will take your survey, start by asking these three questions-- 1) How old are you? 2) How much money do you make? 3) What race are you?)
- all demographic questions should be at the end of the questionnaire
Not only should you be thinking of the above, but you should also be thinking of what you think appropriate questions to ask might be. Think of your own fields/positions and how the industry is constantly innovating. What questions are worthy of being asked? It's better to have too many that we pare down rather than too few.
I will be creating a Google Doc (I guess it's called Google Drive now) that we can all edit to help in the process. I'll post the info to it once it's all set.
Here's the link:
https://drive.google.com/file/d/0B2Dv4MQMdbN4TzdFNEY1UHFLb2s/view?usp=sharing
Feel free to post any questions in the comments section.
Below are some of my thoughts about the Pilot Questionnaire - when we're editing the Google Doc for the survey we're all doing together maybe we can all write in some color other than black to differentiate what was already there and what somebody has added.
ReplyDeleteGenerally speaking, I wonder how senior management will handle the term innovation and think some will likely embrace it as something they've done and people should do, and some will toss it aside as something that does not have a real impact on business. For that reason, perhaps there should be a couple questions early on stating things on the likert scale like "Innovation is an important part of my company." and "I consider myself an innovator." I'm not sure it works to assume these people are all innovators, despite their senior roles.
Question 2 - Is it possible to have age be a drop down menu with the ability to mine the survey based on age groups as well as individual ages?
Question 4 - add word "field" at the end
Question 5 - instead of skills/knowledge, maybe just say "which of the following are most important to be successful as a communication innovator?"
Question 5 - maybe add Strategic Planning, Technical knowledge/aptitude
Question 6 - instead of Good at Time Management, either Effective Time Management, or Efficient
Question 6 - maybe add Realistic, Supportive, Collaborative, Knowledge of Pop Culture, Visionary
Question 7 I think is a great question - but maybe should read divide that time period, instead of allowing for a day or week
Question 7 - maybe something about time not thinking about your idea
Question 8 - I might remove the "hiring a professional coach" line
I like question 11. I think question 9 may need to be re-written.
Great points Scott!
DeleteI especially agree with how senior executives are likely to respond to use of the word "innovation". I think it would be wise to offer a clear/concrete definition in the short survey intro. Purely from a collaboration best practice it would at least ensure every respondent is singing from the same song sheet.
I also agree with your assumption that if its not clearly explained, many people might translate innovation as business critical, and others may not (especially if they consider it to be just a buzzword)
I agree with you both, I write in my post below a reaction to that statement in particular as well. I also will chime in and agree that innovation has zero correlation with title or level. :)
DeleteI also had issues with the term "innovator" and "innovation". I think that it's left too vague and open-ended, which might make it hard to get actionable intel out of this. Asking people to define "success in innovation," for example, is hard if you don't have a working concept of what "innovation" is and if/why it is important.
ReplyDeleteHere are my comments:
1) Is name necessary? I know we want softballs up front but anonymity would help people open up.
2) I would definitely break this down into 5 or 10 year groups for selection
4) I would split this into 2 questions. First a closed-answer choice of sector. Second, a closed-answer choice of managerial level. That would be more helpful here in analyzing the data.
5) I think a transition between 4 and 5 would be helpful where the term “communication innovator” is given a broad definition. Also, it’s a small thing but I’m not sure why INNOVATOR is in all caps. That might confuse respondents and skew answers. Finally, the ordering mechanics of this exercise feel strange given the large number of answers. It would work with 5-6, but it’s hard to logically order such a large array of responses. One idea would be to change the exercise to “pick the 5 most important” from a list?
6) This question introduces “media innovator” as a term. That is confusing. Better to stick with one way or the other throughout.
7) I get what the question is trying to ascertain but the mechanics it uses to do so feel clunky. I’m wondering, how else could we get people to share insight about their behaviors during highly innovative times?
8) This question is out of place in the flow of the questionnaire. Should be moved just before 11.
9) This is a leading question which will result in a smaller range of answers.
11) This is a good line of inquiry but it’s not clear how the respondent is supposed to answer. It might be better to have a series of questions that let the respondent choose from predefined choices:
i.e.
Online◻︎ In-Person◻︎
Part-Time◻︎ Full Time◻︎
Taught by academics◻︎ Taught by industry professionals◻︎
I agree with just about all of your critiques, as well.
DeleteI'm going to start editing the document and will put all my edits in blue. All feel free to change/improve/remove my edits if you feel necessary.
I think it would also be helpful to list some notable communication innovations and have them rank them - I'll put those questions at the end for now, but it might help to put those early on to get them thinking about innovation.
Again, agree with you both. The idea to list innovations is my exact suggestion below! Certainly think specificity would be helpful here.
DeleteSome of the thoughts I had about the questions are rather similar to what Scott and Tim have already mentioned. So I apologize if they sound redundant. I think the definition/meaning of the words innovation and innovator should be laid out more clearly. Even if it is through an open ended question like ‘How do you define communications innovation?
ReplyDeleteQuestion 1: Name does not seem to add value to the data and like Tim mentioned, it might make the respondents more hesitant about what they put in
Question 2: Age does not have to be an open ended question
Question 4: Maybe close ended questions about sector and managerial level and then an open ended question to briefly describe the focus of your work
Question 5: It seems like too many choices and lots of words to put in an order. So maybe lesser choices with an ‘other’ option or a top five pick
Question 7: The ‘approximate total to 100%’ seems rather complicated. And the list does not seem very inclusive considering how many ‘other’ were added in the comments.
Question 8: I like this question but it feels like it belongs with question 11. So perhaps this could be question 10.
Question 9: 2-3 sentences gives little scope to answer both the ‘why the communication industry needs innovation now’ and ‘opportunities’ questions. So perhaps divide them into two questions
Question 11: The question about content seems a little vague and might bring in too many different responses. That could make data analysis difficult.
As to not repeat much of what is listed above, I agree in totality with the augmentations suggested above from a layout as well as context perspective. Some more thoughts on improvements below:
ReplyDeleteDemographic questions appear FAR too early. They should be at the end of the exercise and would be far less distracting and invasive feeling.
Question 5 has way too many choices, I think the users would suffer from the ‘let’s just finish this’ feeling we should all attempt at avoiding when sourcing participation in our research. There are so many choices I don’t think respondents could actually categorize and rank with true sincerity.
I don’t love the use of successful innovator, I feel like innovation is a process and philosophy and not something you can simply be successful at. I do understand the point, which is to reference innovators who have succeeded in implementing innovative solutions .
This survey could be hugely insightful, in its current state there remains a major miss on what kind of information we will actually derive.
Sammy you put it perfectly! 'let's just finish this' is the least desirable approach to the survey :)
DeleteYeah, the 'real' question deals with the conceptual definition of 'innovation.' We'll be sure to bring this up to the Dean before the document goes out. It needs to be explicitly clear what is meant by the term.
ReplyDelete