Fun with Surveys

The “Game Stores Booming” survey that ICV2 published earlier this week raised a mild kerfluffle among game stores in a couple of online industry groups, unlike the comic retailer survey published just before that was greeted with a collective “Meh”.

The main concerns raised were 1) Who participated in the survey and 2) How was the data collected. Who participated in the survey is an especially valid question, given that the article gives no indication as to how many stores were contacted. Sample size, as I taught students in a statistics class many moons ago, is vitally important in determining the validity of a survey. I can take a survey of 10 stores, have 8 of them say they are doing great and voila “80% of stores contacted in my survey said they are doing great.” Very hard to generalize that to the industry as a whole, especially when we are looking at somewhere between 1500-5000 stores, depending on what criteria you use. Now if a larger number a number of stores were contacted, my old statistics textbooks said a minimum of 30 to come up with any generalizable numbers but the more the better, so if we are looking at 75 to 200 stores contacted then that over 80% number looks a lot more reasonable. Incidentally, this is why a lot of scientific studies testing nutritional supplements and assorted diets don’t hold up under close examination. Their results are based on studies of the effects of a diet or supplement on only a few dozen people, meaning it takes on a small number of positive responses to show a significant percentage of improvement.

Secondly, how was the data collected? Ideally and to made the data most generalizable to the overall population, ICV2 used a random sample. From a large pool of game store contacts, the surveyor uses some form of randomizer, random.com comes to mind, and contacts a group of stores selected at random. The fact that almost all game retailers still have land lines and have listings in the white pages makes this a much easier proposition that those poor pollsters who have to contact individuals for polls, since there is no central list for cell phones, as of yet, as there is for land lines. More than likely however, ICV2 used what is called a “convenience sample”. Whoever conducted the survey likely said, “Hand me that list of stores we have tucked away. We will call some number of them to see how they did in 2015.” The results are certainly reflective of stores within that pool. The problem is that the results are only reflective of that pool and must be used with caution if anyone tries to generalize the results to the overall populations.  That is why academic studies using a convenience sample always state so in the study limitations.

All that being said and being curious myself, I took a quick convenience self-selected sample, meaning that I posted a survey and other stores could respond or not. Of those stores responding, just over 90% indicated they were up at least 1% over 2014 and just over 75% indicated a sales increase of over 10% for the same time frame, correlating with ICV2’s results. Now, I didn’t break out length of time open, size of store, location of store, number of stores or if there had been any remodeling or expansion, all of which could affect sales increases. There is also the possibility that stores with sales declines didn’t want to respond. Still, the similar results give me hope for a good 2016. Please feel free to send your comments to ICV2.