 |
Terra Nova: Gender differences in MMOsDecember 17th, 2009 |
The latest results from the giant EQ2 data research project came out a few days ago, and this time they center on gender. I’ll just point straight to the summary from Terra Nova:
- Men are more driven to achieve within the game space, and women are more driven to socialize, although these differences are not as large as one might expect.
- Female players fall into two distinct categories: stereotypically feminine players, typically brought into the game by a partner, and very hard-core players.
- The hard-core women are more intense than their male counterparts: “The top 10% of male players played an average of 48.86 hours per week, while the top 10% of female players played an average of 56.64 hours per week.”
- Female players are healthier offline than the males. This is especially true among older players.
- When males and females play together within a romantic relationship, the males are less happy and the females more happy. When not playing in a romantic relationship, these outcomes are reversed: the females are less happy and the males more happy.
- There are a surprisingly large number of bisexual females playing, but not males. While male bisexual players stuck to the national average, females were about five times higher than the national baseline rate.
- Females under-report their playing time more than males.
The full paper can be gotten here (scroll down a bit) but you have to register or purchase it.
You can follow any responses to this entry through the RSS 2.0 feed. Responses are currently closed, but you can trackback from your own site.
Art
Game talk
Gamemaking
Mailbag
Misc
Music
Open thread
Reading
The Sunday Poem
Watching
Writing
I still have a problem with the participation percentage being unpublishable, but even if a vast majority responded to the survey, that doesn’t eliminate selection bias as much as one might think. It is common sense that higher response should lead to less bias, but studies have found no such correlation. It seems instead to depend on the methodology of the survey. There can even be an increase in bias when researchers attempt to increase participation.
Imagine if the players on the chosen servers heard that “phat loot” was being given away. So they log in and take the survey even if they had no intention to play. Even if their “main” is on a different server, they might load up a character they had once made to dabble in one of the alternate rules servers. In this situation, the researchers might think they are getting high participation from those logging in, when they are actually getting a high number of players logging in specifically to participate. This would greatly skew response in favor of a particular type of player that would desire the particular item.
The Public Opinion Quarterly did an entire issue on self-selection bias if anyone is interested in further reading.
Or bypass the thesis review committee and go directly to soliciting venture capital for a innovative online stop-smoking program.
Yukon Sam>the study indicating that lesbians and bisexuals have, on average, a higher incidence of smoking and ‘heavy’ drinking is interesting…
It’s not one I’ve any particular connection with, I just happened to know about it because I’m a game designer and I come across stuff like this from time to time in my research.
>but I have to wonder if that holds true for those whose social interaction is conducted largely online.
I’ve no idea. Anecdotally, it’s been my experience that in general adult players of MMOs seem to smoke more than the rest of the population, irrespective of their gender. I’ve no idea if that actually is the case, though. It could be that sexuality has nothing to do with the incidence of smoking and drinking in MMOs – I only mentioned that particular survey as an example and I happened to have come across it before.
>the gay and bisexual women I’ve met online seem to be an an elevated risk for veganism and pilates (again, no numbers, just my experience).
Or ninjaism and pirates. Until someone does a survey, we won’t know. If they do do a survey, and it matches what other surveys show, then that tells us that the online component is probably not a factor in whatever is being tested; if it has a difference, then either one or both surveys are flawed or the online component is a factor.
>I have to wonder if the same factors online that attract a disproportionate number of gay/bisexual women, hypothetically a less stigmatizing environment, also work to mitigate against smoking/drinking risk.
Well, there’s probably a PhD in that for someone willing to do the donkey work to find out.
Richard
When someone mentioned the staff, I said to myself, “Oh! That survey!” I responded to this one myself. A unique item is a unique item, regardless of whether you actually get any use of it, and most particularly if it has a highly distinctive visual appearance. Heck, the most prized item in Ultima Online for a long time was the ‘true black’ dye tub. The staff did seem to be much more desirable than its limited utility would suggest.
Richard, the study indicating that lesbians and bisexuals have, on average, a higher incidence of smoking and ‘heavy’ drinking is interesting… but I have to wonder if that holds true for those whose social interaction is conducted largely online. Clearly there’s an elevated risk of substance abuse in any group subjected to social stigma. On the other hand, the gay and bisexual women I’ve met online seem to be an an elevated risk for veganism and pilates (again, no numbers, just my experience). I have to wonder if the same factors online that attract a disproportionate number of gay/bisexual women, hypothetically a less stigmatizing environment, also work to mitigate against smoking/drinking risk.
Bernard Chen>has anyone actually purchased and read the paper?
You don’t have to purchase it. You can get it for free from Dmitri Williams’ site.
>I thought the self-selection criticism was entirely valid.
It is valid. It’s just not valid enough to discredit the work.
>The incentive given to players has the most value for players who are interested in cosmetic items and have or expect to have characters in the 1-10 level range.
Whatever, it seems to have worked. I brought up this very point myself in 2008 when the first paper appeared – see the comments on TerraNova, with Dmitri Williams’ reply.
>It seems very reasonable to assume that players who fit this description answered the survey and everyone else ignored it.
It does, but “everyone else” is a tiny minority in this particular case. Yes, even missing one person would skew the results ever-so-slightly, but at some stage you have to be able to say you have enough responses to be confident that they’re representative. I believe, from what they’ve said, that the researchers have exceeded that threshold. You may believe they haven’t. That’s fair enough; if you have a way of getting better information, I’m sure the researchers would love to hear from you.
>Richard, you sound like you have a personal relationship with the author(s) and are privy to some inside information that isn’t available to the rest of us.
I do know the authors, but that doesn’t mean I hold back any. I don’t have any inside information, however knowing the authors means that if they say the survey captured responses from the vast majority of people who played during the sample period, I’m confident that it did indeed do just that.
>Was the response rate 80%+?
They can’t say the exact response rate, because that would mean revealing how many people actually logged in that weekend; they do know this figure from the server logs, but SOE want to keep it confidential so they’re not allowed to reveal it.
>Was there any way to confirm that the non-respondents weren’t significantly different from the respondents?
No. However, even if they all held the same extreme views, the overwhelming response from those who did take part in the survey would still swamp them.
>(Were they able to compare non-voluntary/server-side player data?)
They were, they just can’t tell us actual numbers. They had a choice of saying how many people took the survey or what proportion of the people who could have taken it did take it; social science convention is to report the former, and they would never have got their paper published if they hadn’t provided it, so I guess that’s why they did.
Richard
> It seems very reasonable to assume that players who fit this description answered the survey and everyone else ignored it.
It seems reasonable…to assume a false dichotomy exists?
Aristotle just rolled over in his grave.
Now that a couple of days have passed, has anyone actually purchased and read the paper?
The conversation between Wukong and Richard was very educational as a reminder that experimental method potentially has a large impact on the quality of the data. I thought the self-selection criticism was entirely valid. The incentive given to players has the most value for players who are interested in cosmetic items and have or expect to have characters in the 1-10 level range. It seems very reasonable to assume that players who fit this description answered the survey and everyone else ignored it.
Richard, you sound like you have a personal relationship with the author(s) and are privy to some inside information that isn’t available to the rest of us. Was the response rate 80%+? Was there any way to confirm that the non-respondents weren’t significantly different from the respondents? (Were they able to compare non-voluntary/server-side player data?)
Bernard
Very interesting report although none of it very surprising.
Wukong>Self-selection is a problem with all surveys to a greater or lesser extent.
That seems more of a nuanced statement than your original assertion that the survey is “completely worthless”.
Yes, I agree, self-selection is a problem with all surveys to a greater or lesser extent. In the great scheme of things, in this case it seems to be to be at the “lesser” end, rather than the “greater” end, because of the number of respondents.
>In this case, there was no attempt to mitigate this problem. No attempt to randomize.
The attempt to mitigate was made by trying to net everyone. I personally would have liked to see a (smaller) calibration study that checked responses against actuality. It’s nigh impossible to check things like happiness, but gender, height and so on would be directly testable. The cost to get this kind of data would be huge, though, so you can see why they didn’t do it…
>No cover story for the survey.
There may have been IRB reasons for this. I don’t see it much of a problem that the players knew the general purpose of the survey, though; tricking them with a cover story could well have skewed results more.
>No attempt to screen for multiple accounts, which is always a problem online.
I believe the way that SOE handles accounts, it has some concept of “registered owner”, leastwise it did when I last looked at it (which was maybe 4 or 5 years ago, so things could change). Sure, some people will still use separate identities, but in minuscule numbers compared with those who don’t. I doubt this would make much difference to the data.
I agree, though, that if this wasn’t accounted for then people almost certainly would have been counted twice, because everyone who had more than one account would have logged on with their others just to get the reward for completing the survey – even if they wouldnt’ ordinarily have logged on with that account over the period.
>Maybe socially motivated women are more likely to participate in surveys promoted in-game. Maybe achievement oriented men will be more likely to participate in surveys that reward them in-game.
I raised this issue the first time the survey results were used. The response is that it doesn’t really matter because most players did the survey anyway.
>It’s problematic that the only attempt to shape the sample was by offering a very specific incentive. From playing EQ2 I know that staff would not be useful in combat, just a fashion or status item.
Scott Hartsman told me it was intended to be of some use to level 1-10 characters, or something like that.
I agree that selection from a range of items would have been better. It was a tough job to get just the one accepted by the SOE high-ups, though, so they went for something with fairly wide appeal but innocuous effect.
>I sympathize with his position, but science simply cannot work on undisclosed assurances.
It can’t work as well, but it’s not entirely broken. I don’t care if I have the exact proportion of users who responded, but “vast majority” means it’s definitely going to be over 80% and possibly over 90%. That being the case, I can give the data more credence than if it were 51%.
>Even if it could, the servers selected for the survey were not representative of the game’s general population as I note above.
In their earlier Who Plays paper, the researchers looked at differences between servers. As far as I’m aware, they always do this. If any server-specific differences had popped out, they would have mentioned them.
>When all we have is self-reported data, we have nothing but a house of cards. You seem very confident in the yet to be revealed foundation of that house. I’m not.
It’s always wise to be sceptical about surveys like this, even ones on this kind of scale. I’m confident that the research team is aware of the issues, but that doesn’t mean I take their figures on trust. I accept that my professional respect for those involved may not be shared by everyone who reads the paper, and that’s fair enough. However, there’s a difference between reading a paper with caution and dismissing it out of hand, which is what you seem to be advocating.
Richard
This is a great post Dmitry. I just had one of the ‘Doh!’ moments and ran back to correct my own site before publishing my comment. You see my own comment form did not match what I’m about to advise. I get less comments than you, so never noticed any problem. I’ve changed it now anyway so here goes.
IT developer
Why does the large percentage of bi-sexuals surprise you? Of the folks I know with that persuasion, they are typically adventure seekers. I’m not being stereotypical; it seems to be true. That they would be gamers seems wholely plausible.
@raph (OT): Colin Linden came to town to play tonight. I heartily suggest you buy every one of his CDs you can and go see him life. He is a Grand Master of the Axe and in styles I think are right for you.
I was looking forward to these papers because I thought that with SOE’s participation we would finally get some good hard data.
So unrealistic expectations led to disillusionment?
I just don’t think that “these are the best data we have” is an excuse for bad data.
It’s not an excuse. It’s a statement of fact. You’re brandishing ultimatums, none of which are relevant to anyone who understood the inherent limitations of the science.
It is not bad data. The data is the reports. It is not “people are happier”; it is “people report themselves as happier”. Then, when someone comes along and defines an absolute indicator of happiness and then does a study comparing reported levels and actual levels, we can draw some conclusions on the actual happiness of these EQ2 players.
I’m sure that will happen soon. And by soon, I mean never.
>Including the giant paragraph at the end that found the BMI reports to be highly disturbing and nearly inexplicable, right?
Right. Except it’s not inexplicable. I used it as an example because it was most easily explained by the methodology.
I agree this is a long standing problem in social science. I just don’t think that “these are the best data we have” is an excuse for bad data. I was looking forward to these papers because I thought that with SOE’s participation we would finally get some good hard data. So far all we got is that women probably lie about their weight and sexuality.
>What is it about self-selection that you regard as being fatal to surveys, and why does that apply in this example?
Self-selection is a problem with all surveys to a greater or lesser extent. There is always a problem with controlling for reasons for participation. In this case, there was no attempt to mitigate this problem. No attempt to randomize. No cover story for the survey. No attempt to screen for multiple accounts, which is always a problem online. For how this might have skewed results, let’s just at the first two hypotheses:
H1:Women will be more likely than men to express a social motivation for being in MMO games.
H2:Men will be more likely than women to express an achievement-oriented motivation for being in MMO games.
Maybe socially motivated women are more likely to participate in surveys promoted in-game. Maybe achievement oriented men will be more likely to participate in surveys that reward them in-game. It’s problematic that the only attempt to shape the sample was by offering a very specific incentive. From playing EQ2 I know that staff would not be useful in combat, just a fashion or status item. Maybe achievement oriented men seek status and socially motivated women seek fashion.
Richard, I have seen your exchange with Dmitri Williams on Terra Nova where he assures that the sample was a vast majority, but is unable to provide a hard percentage. I sympathize with his position, but science simply cannot work on undisclosed assurances. Even if it could, the servers selected for the survey were not representative of the game’s general population as I note above.
>>You should expect all of the self-reported data to be inaccurate.
>But perhaps in predictably inaccurate ways?
>>Without more objectively collected data, you have no way of knowing how inaccurate.
>But sometimes you do.
This is exactly my point. Where there is objective data, like in the case of playtime, we can evaluate hypotheses and even do follow-up studies of other populations, as you suggest. When all we have is self-reported data, we have nothing but a house of cards. You seem very confident in the yet to be revealed foundation of that house. I’m not.
Wukong: I hate to sound like I’m denigrating a field of research I’m actually very interested in. All I ask is that people actually read the paper.
Including the giant paragraph at the end that found the BMI reports to be highly disturbing and nearly inexplicable, right?
You’re not denigrating merely MMO research; you’re denigrating basically every field of social science, all of which largely depends on self-reporting. Which is fine; your complaints are noted and have been well-known since the inception of each field. But until a better and equally ethical method is devised, this is the most valuable set of data we’ve ever had.
No one is pretending the sample is representative, and no one is pretending that these interpretations are law. It barely even draws conclusions. I mean, average MMO player my foot: we know, and all the comments inside the paper make it clear the writers know, that this describes only EQ2 players. Not EVE players. Not MUD players. Not Mob Wars players.
This is data. And only data. Treat it as such.
Wukong>The whole survey was self-selected.
Yes, but such a large number of people responded that the figures are going to be good indicators even if they’re not exact. What is it about self-selection that you regard as being fatal to surveys, and why does that apply in this example?
I myself have reservations about the selection. I would have preferred to have seen a range of possible rewards, for example, rather than just a single snaky staff. However, if the vast majority of players over the sample period responded, it isn’t really going to make much difference.
>All the data was self-reported, except playtime was also independently measured. Reported playtime was systematically inaccurate.
Yes, I know. The researchers know. It was one of the first things they noticed. The fact that they know this would suggest that they are well aware of the dangers of self-reporting. We’ll have to wait until the paper on health before we find out how they corrected for this (unless one of them reads this and gives us a preview), but they could, for example, normalise against other self-selected surveys of similar groups which have been checked against actual physical measurements.
>You should expect all of the self-reported data to be inaccurate.
But perhaps in predictably inaccurate ways?
>Without more objectively collected data, you have no way of knowing how inaccurate.
But sometimes you do.
Example: from the EQ2 data we can see there is a fairly consistent under-reporting of playing time for each gender. If we were to undertake a study of WoW players, we could reasonably (not definitely, but reasonably) expect that they would follow the same pattern. If the number of hours they reported as playing per week were in line with those for the EQ2 players, then we could reasonably assume that the actual hours they player were in line with the EQ2 actual hours – even though we didn’t measure the actual hours of WoW play.
I’m not a social scientist, but I know people do surveys the whole time, and there are good data sets to which I would assume the researchers here have access. If they’ve used these results to normalise their figures, we’ll see that later.
I agree with you that if they just took the height and weight figures given in the survey as accurate, that would be a big mistake and we couldn’t trust the BMI. It’s not as if people are going to run off and weigh themselves when asked by a survey, they’re going to guess what they weigh. I’m a little dubious about using BMI to measure healthiness, too; I know, for example, that lesbian and bisexual women are generally less healthy than straight women as they smoke and drink more, but that wouldn’t necessarily be picked up their BMI.
That said, I can’t believe they wouldn’t have picked up on this. Those people are very experienced researchers in this area, and it would be stunning if they let something like that slip past without an explanation.
>That is what makes the study worthless, not my misogyny.
Just for the record, the misogyny line was to illustrate the ease with which your one-fault-all-wrong argument could be used against itself. I wasn’t trying to lay an actual claim of misogyny on you (you could be female yourself for all I know).
Richard
Raph, Collecting lots of bad data doesn’t make it good. Even if the response had been 100%, you can’t make the case that the sample was representative. For some reason they chose to survey only four servers: one PvP, one Roleplaying, one Exchange, and one Regular. 75% of those thousands of respondents were from alternate rule sets. What value would this data have with regards to the average MMO player even if it was accurate?
I hate to sound like I’m denigrating a field of research I’m actually very interested in. All I ask is that people actually read the paper.
Richard, I don’t have a problem with just one aspect of the survey. The whole survey was self-selected. All the data was self-reported, except playtime was also independently measured. Reported playtime was systematically inaccurate. You should expect all of the self-reported data to be inaccurate. Without more objectively collected data, you have no way of knowing how inaccurate. That is what makes the study worthless, not my misogyny.
Yukon, I never said underestimate, just not accurate. We are in anecdotal agreement. People are a poor source.
Morgan, yes, my desk is much, much smaller than that! I think your desk is larger than my kitchen! I am repulsed by the idea of food and crumbs dropping into my keyboard, followed by the tap, tap, tapping of greasy fingers, but perhaps if I had a full size diner for a desk I wouldn’t have that issue. Ha! =)
In my (purely anecdotal) experience, people in our society (especially but not exclusively women) tend to overestimate their weight rather than underestimate it, comparing themselves to photographic ideals that are impossible to achieve without the aid of PhotoShop, steroids, a personal trainer and/or major surgery.
If that were the case, both women and men would tend to be more fit in reality than their self-reported fitness would indicate.
If there is a counter-bias, I would expect it would come from the fact that people who are aware of their BMI would tend to be more fit than those who aren’t; in other words, a disproportionate number of less-fit people would be represented in the don’t know/didn’t answer count.
[...] This post was mentioned on Twitter by Raph Koster and Kay Kim (김 기웅), Emma Haley. Emma Haley said: this is actually really interesting http://www.raphkoster.com/2009/12/17/terra-nova-gender-differences-in-mmos/ by @raphkoster [...]
Wukong>Are you surprised a woman might not accurately report her weight?
Why, Wukong, I do believe you’re hinting at a wildly sexist stereotypical characterisation of the female psyche. Clearly, you have bias issues, and therefore we can entirely dismiss all your views, including your characterisation of the survey as being “completely worthless”.
See how easy it is to rubbish people?
This survey is by some distance one of the most comprehensive ever undertaken for MMO usage. The researchers are sufficiently aware of the dangers of self-reporting that they made that the subject of their first paper (comparing reported usage per week against actual server data; it turns out that both genders regularly under-estimate their playing time, but women do so by more than men – although women play for longer, too).
Now it could be that the figures for BMI are indeed subject to some misreporting. It is inconceivable, though, that the researchers haven’t thought of that. When they write their full paper on health (this one is on gender, not health), we’ll find out how confident they are of their figures and the extent to which their results are consistent with those of independent studies. Even if it turns out you are correct, and the BMI figures really are systematically under-reported in the same way that playing times are, that wouldn’t invalidate rest of the survey. You’d need to invoke a major conspiracy theory to explain away those bisexuality responses, for example.
Just because you have a problem with one aspect of a survey, that doesn’t mean you get to extrapolate it to the whole survey.
Richard
Hypothesis: this is because they have not yet mastered the fine art of fragging and consuming an entire pizza at the same time, without feeling any compulsion to clean the grease off the mouse.
Harumph. I clean the grease off my fingers before I touch the mouse again.
Adele: Repulsed? I think you need a bigger desk then. The desk that I use looks like the one in the following image, only bigger, fancier, and without the top compartments. I practically have two kitchen counters on either side of me! There’s space for a mini-fridge, too. ;p
http://images.hayneedle.com/mgen/master:MAI073.jpg
Entirely self-selected, but it was also thousands upon thousands of people — a large sample, covering a very large percentage of the EQ2 players who logged in over the course of several days. With sample sizes that large, especially correlated to the user data they have, this is well beyond “a glorified internet poll.”
Read the methodology of the study. It was entirely self-selected and mostly self-reported. It is completely worthless and no more than a gloried internet poll. For example, the “women are healthier” is based on self-reported BMI. Are you surprised a woman might not accurately report her weight?
“When males and females play together within a romantic relationship, the males are less happy and the females more happy. When not playing in a romantic relationship, these outcomes are reversed: the females are less happy and the males more happy.”
This was true for me, but largely because my wife was less interested in playing than in doing something together, which resulted in me making all the decisions of what we were doing, where we were going, what our goals were, what equipment was best, what tradeskills to pursue, etc etc were all left to me. I was playing while she was following and socializing.
Sam, you may be onto something. The only time I didn’t feel repulsed about eating at my computer, over my keyboard, was while I was pregnant, and well that just couldn’t be helped because I was almost always hungry it seemed! And it was either that, or stop the raids every 15 minutes, lol.
Female gamers are not just healthier than male gamers, they’re also healthier on average than the general population. Hypothesis: this is because they have not yet mastered the fine art of fragging and consuming an entire pizza at the same time, without feeling any compulsion to clean the grease off the mouse.
I don’t see anything too shocking there, except perhaps the bi-sexual thing. That is very curious. I wonder why that is? I will say I began playing MMOs before my husband. He was playing video games before me, and I was frustrated by it, but they were not MMOs.
I wanted to have something in common with him, and when I heard about SWG, and realized it was a game I didn’t have to fight in, I wasn’t a big fan of fighting, and I could talk to “real” people in, I wanted to try it out, and I got my husband to try it too. It was the gateway game for me, and of course I started out as a dancer. Overtime, I got tired of other players escorting me around, and learned to fight. And I transformed into the hardcore female player. At that point I surpassed my husband in play hours in every MMO, and so those results aren’t shocking either.
I would also say that every time we are playing an MMO, he is less happy than I am. I think because I am playing more, and I get more involved in the game. Usually to get my attention, he has to send me a /tell in the game, even though he is sitting next to me, because I will be in a group trying to pay attention, or raiding and be on Ventrillo, and he has never been a raider. And yes, I usually say I play less than I do too! Very interesting.
Thanks for the post Raph!