Products
Rob Brooks-Bilson's Blog
Tariq Ahmed's put together an interesting survey trying to understand what features people want to see in the next version of ColdFusion (ColdFusion 8). Tariq just released the results of the survey, along with the raw data from SurveyMonkey.
I was initially surprised at the results Tariq published on his blog, as what I saw there didn't match up with what I'd been reading in various blogs, mailing lists, or forums (for example, Image Manipulation came in at number 16, something I know to be one of the top requested CF featues for several versions now). Wondering what the raw data looked like, I downloaded Tariq's spreadsheet.
I tried posting my questions/findings directly on Tariq's blog, but after my first comment, the comment system there ate my next attempt (I'm guessing it was too long). In any case, apologies Tariq for making a separate post...
I don't claim to be a statistician by any stretch of the imagination, but I did have an undergrad class or two in statistics and survey design, and two things didn't seem quite right to me.
First, the question style used is somehting known as a Likert-type question. It's a question used to measure attitudes toward a particular construct:
1 - Strongly Disagree
2 - Disagree
3 - Neither Disagree or Agree
4 - Agree
5 - Strongly Agree
Technically, this is known as the Likert-scale, however I'm referring to it here as Likert-type because Likert-scale questions are always used with multiple related items, and the survey here deals with single, unrelated items (individual features).
In a survey using Likert-type questions like this, a response of neutral shouldn't be "scored" as meaning that someone wants a feature more than they don't. If you are assigning a value of 3 to a neutral response, and using that to calculate a mean score, you are making that response seem as if a respondent wants the feature more than they don't (which isn't the case). One of the big issues in using a Likert-type question in this type of survey is that the responses are ordinal because it can't be assumed that people filling out the survey will perceive the difference between adjacent options as being the same. In other words, the value assigned to them (1-5) is positional, not representative of an actual mathematical value.
One common way Likert-type questions are scored/analyzed is to combine all of the like/agree responses into one group and all dislike/don't agree answers into another, discarding all of the neutral answers. Since the question we're trying to answer here is what features are most important to people for CF 8, it seems to make sense to just combine the want/really want, and rank the items from there. This answers the question "what do people want to see" vs. what are people ambivalent about, or what aren't people interested in seeing (which you could extrapolate from the survey as well. If I order just based on what people voted that they want to see, here's the order or preference I ended up with. Out of 317 respondants, here's what people wanted (ordered by the number of people indicating "want it" or "really want it":
243 Image Manipulation
233 Improved Crash Prevention
232 IDE/Hooks into better interactive debugging
229 Access to Server Internals/Metrics
222 Add PDF Forms Support
216 Give us an IDE with CFC introspection
215 Improve CFDocument
211 AJAX Abilities
210 Control logging at the application level
209 Improvements to large file parsing
209 Create/Extract/Manage ZIP Files
202 Tools to debug remoting and gateways
196 Add native support for MS Office formats
190 Strong Integration w/Exchange (Mail/Calendar/Etc...)
189 Threading/Thread Management & Protection
186 Add native support for RSS
184 Add cache control at the app level
184 Manage many CF Servers from a single console
181 Enhanced flash forms using Flex 2
175 Improved Reg-Ex Support
173 Upload to a Variable instead of Disk
168 Add granular control of DB and content caches
156 Faster validation abilities (E.g < validator Flex's borrow or , validValues="x,y,z">
154 Integrated CF8+Flex FDS Install
153 CFAdmin Tools for creating Event Gateways
149 Add non-DOM based XML Parsing
149 Strong IMAP Integration
144 Make CFC Instances Serializable
143 Plug-In Architecture
141 CF Desktop Plugin to support drag and drop upload
137 CF to CF Communications via AMF (vs XML)
132 Add CSS/HTML pod layout support
128 OO Abilities: Constructors
119 Add refactoring to an IDE
116 Ability to deploy customized+scripted CF Installations
115 3rd Party Updates
114 Web Interface Hooks into JRUN Configuration
113 Static Methods in CFCs
113 Sandbox Security Enhancements
109 OO Abilities: Interfaces
107 Integration of Breeze Functionality
98 Tomcat & JBOSS Support
94 OO Abilities: Function Overloading
90 SNMP Gateway Service
77 Add accessibility 508 checking to an IDE
72 OO Abilities: Destructors
58 Mustang Support
56 Trend to be more Java like
55 E4X Support
If you really want to use the mean for each question, you should get away from a Likert-type question with a neutral value, and just use a scale of 1-5 with one being no interest and 5 being strong interest.
What all of this really illustrates is just how difficult it can be to design surveys and interpret their results. I really wish tools like SurveyMonkey would do more to help reduce ambiguity in surveys, as well as provide better analysis tools for the data generated.
Aura skin for Raymond Camden's BlogCFC inspired by Joe Rinehart & Steven Erat. This blog is running version 5.9.004.
Love your posting though, very interesting! Ya the funny thing about stats is you can crunch the numbers so many different ways.
I'm definitely the furthest thing from a math guy, the results in the raw data are literally a copy and paste from SurveyMonkey - so all I did was sort it. Though I hope everyone keeps in mind this was just for fun to get a sense of what the mood is.
The other survey style I like is the "if you had $x, how would you spend it..." Then sort by what features have the most amt of $ allocated towards it.
Cheers!
I think the survey wasa fantastic idea (and success, considering you got almost 400 people to participate).
I like the $$ allocation as well. I imagine that the cf team has to do something similar when trying to decide exactly what to put into a release. Every feature has a cost associated with it, and in their case, they need to get the maximum return for those $$ (which is why I don't think we'll be seeing any cfscript enhancements anytime soon).
I like your analysis, since I'm used to the Likert-scale and "wore that hat" when filling the survey in (ie hated the idea = 1).
the way you have it, my personal preferred feature comes in at #3 which is (my gut feeling) where it should be.
While I'm grateful that Tariq went to the effort, getting more raw data by ppl filling out the survey might be more representable...there's a lot of CF users out there.... getting, say, 31,700 users might show vastly different results....
The revised rankings are less of a shock result. I think this is a good example at how statistics can be presented in different ways to reveal completely different results. This is how politicians use statistics to back up their views. If you look at the raw data (if you can get your hands on it) sometimes you can find a different way of analysing it that contradict the official findings. Hence the classic Mark Twain quote "There are 3 kinds of lies: lies, damned lies, and statistics."
Tariq, thanks for running the survey. I mean no disrespect to your efforts, it's an easy mistake to make and flags up an important point about running surveys, how to analyse them, and how easy it is to accidentally or otherwise distort the results. I've been programming surveys for years and it takes a while to appreciate how complex even a simple set of results can be.
Interesting and accurate take on Likert scales. One approach I've always used with these sort of LIkert scales is to transform them - Neutral becomes 0, really like = 2 like = 1; really dislike -2 and dislike -1. That would allow for the results to more accurately reflect the relationship within the population as a whole.
regards,
larry
That's essentially what Dale did here on his blog:
http://dale.fraser.id.au
I don't totally agree with the method (due to bias introduced by the survey design), but the results are very close to mine regardless.
Thanks.
Side note: The "captcha" text is really hard to get right.
I'm looking at scaling back/removing the captcha. It's a real pain, I agree, as I often don't get it right either.
-Rob