Gaming the Studies – Beginning in Earnest
This post is part of a series on the research studies on sexism cited in the FeministFrequency video series Tropes vs Women in Video Games. The first post in that series covered my views on the FeministFrequency videos and can be found here.
This post in the series is focused on background information about the relevant parts of the field of social psychology. While I present a table briefly summarising the studies that are the focus of this series, the purpose of the next two posts is to cover some background information and my approach to reviewing the studies. I plan on doing detailed explanation and analysis of each study in subsequent posts.
I feel I should point out that I am not a researcher in the field, nor do I possess qualifications that major in any form of psychology. I do have an ongoing interest in psychology, and I have studied a few psychology subjects during my time at university, and it is that experience and knowledge that I used as a springboard for the research done for this post. If anyone spots any errors in the way I’ve presented the information please let me know in the comments.
Studying the Studies
In order to understand what studies in social psychology can tell us about ourselves, it’s necessary to understand the basic process they typically follow. A sample population (“subjects”), usually undergrad students roped into it with promises of partial credits, are given a task where some are exposed to some stimulus material while others form a control group. The researchers then make observations of the way the subjects react to the stimulus in comparison to the control groups. Although the subjects are aware they are being experimented on, they are usually ignorant of the details of the experiment until a debrief session after the observations are made.
The most common way to make observations in social psychology is to get the subjects to fill out self-report questionnaires. These questionnaires involve subjective questions or prompts to which the subject response on a scale such as “strongly agree” to “strongly disagree” or “very likely” to “very unlikely”. These responses are numerically coded and then used to produce statistical results. This constrasts to experiments in cognitive psychology where observations are made with regards to the performance or capacity in undertaking a specific task. Tasks are often focused on word recognition, logic puzzles and/or reaction times.
To get a basic feel for the types of experiments that have been cited by FeministFrequency, I have listed the stimulus material along with the socially significant observations made. Notably, one of the studies (Fox & Tang) is an online study that seeks to examine correlations rather than explore causative effects through experimentation.
|Study||Stimulus (Priming) Material||Main Observation|
|Loughnan1||Images of men & women with varying levels of clothing||Self-reports of attitudes and beliefs about the individuals pictured|
|Rudman2||Sexist/sexual advertisements and a subsequent simulated job interview||Levels of sexual and dominating behaviour were independently observed|
|Dill3||Images of sexist stereotypes in video games||Self-reports of beliefs about women and sexual harassment|
|Yao4||Playing either Leisure Suit Larry, The Sims II or Pacman II||Self-reports of beliefs about women and likelihood to sexual harass|
|Fox5||Virtual Reality environment with avatar wearing revealing or conservative clothing||Self-reports of body-related thoughts and beliefs about rape|
|Fox & Tang6||This was study was an online survey that measured the correlation between demographics, gaming exposure, attitudes, beliefs and sexism in gaming.|
The next step is to look at the theories that social psychologists use to explain why they expect the given stimulus to provoke a measurable response.
The foundation for these studies is the cognitive psychology theory of priming. The basic premise of the theory is that exposure to certain stimuli activates the mental features related to that stimuli, and that as a result they become ‘primed’ and more readily accessible for subsequent use. The primary focus of the theory is on words and Wikipedia provides a useful example:
“For example, if a person reads a list of words including the word table, and is later asked to complete a word starting with tab, the probability that he or she will answer table is greater than if they are not primed.“
Experiments with priming need to involve two distinct stimuli. The first stimulus is used to prime some of the subjects. The experimenters then attempt to measure the reaction to a second stimulus and compare the differences between the primed subjects and the control subjects. This is the relationship between the two columns in the table above.
Importantly, this effect has been demonstrated even where the subjects are not consciously aware of its effect. A 1982 study lead by Endel Tulving7 asked subjects to study a given set of words. The subjects were then tested in 2 ways: conscious recognition of the words, and the ability to recognise the words when prompted only a few letters. For example, if two of the words the subjects studied were “ADENOID” and “EPITAPH” the subject may have later been asked (Yes or No) if they recognised the word “ADENOID” but then asked to complete the word “_PI__PH”. Half the words the subjects were tested on were not words they had studied to enable the relative effect of the priming to be measured.
When tested 1 hour after studying the word subjects were able to both conscoiusly identify which words they had studied and which they had not with reasonable accuracy, and more easily recognised the partial words that they had studied compare to words they had not. When retested 7 days later, the benefit to conscious identification had significantly diminished but the ability to more easily recognise partial words had not. This study provided evidence to support the theory that priming can work even when subjects are not consciously aware of its effect.
The priming effect has been demonstrated to go beyond simple word recall and extend to the priming of semantic concepts. One more widely known concept that is related to this is subliminal messages. The stroop effect also provides an accessible and practical example. Another relatable example of the basic concept is how memories are become much easier to recall when listening to familiar music, smelling familiar scents or simply reading the name of an old acquaintance.
Primed for Uncertainty
The studies I’ll be reviewing aim to extend these basic theories of priming both by extending the stimulus from words into multimedia, and extending the observations from cognitive performance into social beliefs and behaviours. While priming is a recognised cognitive theory, it is not without criticism and not necessarily a straight forward topic to study especially when extended to research the social and behavioural domains of psychology. Many of the studies in social psychology I’ve read in researching this post tie their social conclusions to the priming theory by including word recognition tests developed in cognitive experiments. They do this to establish that their priming stimulus has caused at least some cognitive effect, however they still face challenges in experimentally linking the priming to subsequent views and behaviours in the subjects.
In a 1996 John Bargh8 lead an experiment to study the effects of priming with words related to elderly stereotypes. Some subjects were primed with a scrambled sentence task that involved words such as “worried, Florida, old, lonely, grey, selfishly, etc”, while controls performed the task with neutral words. When the subjects left the room they were unknowingly timed by the experimenters using hidden stopwatches. The results showed that “Participants in the elderly priming condition (M = 8.28 s) had a slower walking speed compared to participants in the neutral priming condition (M= 7.30 s)”. In the subsequent discussion it was stated the results “suggest that exposing individuals to a series of words linked to a particular stereotype influences behavior nonconsciously.” They go on to note that the lack of time or speed related words indicates it’s the activation of stereotype that drives the differences in behaviour.
However, a subsequent study in 2012 lead by Stephane Doyen9 aimed to replicate the Bargh study and produced some striking results. In a first attempt to replicate the study, Doyen replicated the study increasing the number of subjects and using automated infra-red timing hardware. This effort failed to result any priming effect.
A second experiment in that study was conducted which was identical to the first with two exceptions. First the experimenters were divided into two groups: half were told the priming was expected to cause the subjects to walk slower, while the other half were told the priming was expecting to cause the subjects to walk faster. Second the experiments were asked to do manual timing with stopwatches under the false pretence that the infra-red equipment was faulty. The results of the automated timing showed that the subjects only slowed their movements when the experimenters were told to expect the prime to result in slower walking speed. Compared to the automated timing, the manual timing produced results correlated with experimenters expectations rather than consistent with the actual priming of the subjects.
This rather awkward result was not received terribly well by the original researcher. Perhaps not surprisingly, another failed attempt to replicated the “Elderly-related Words Prime Slow Walking” effect is (currently) the most read replication attempt on psychfiledrawer.org, a site designed to provide access to unpublished replication attempts. In a Science News article10 on the priming controversy, the researcher behind both that replication attempt and the psychfiledrawer.org site itself, Hal Pashler, was interviewed and voiced concern about the impact of publication bias on psychological research. It was also noted that there are four other replication studies in social psychology that Pashler has undertaken that have failed to reproduce the results of the original and that have not been published in journals.
It has become evident that priming is a cognitive process that is extremely sensitive to corruption by unintended factors. When studying the subtle response to a subtle stimulus it is important to ensure all other potentially confounding factors are appropriately excluded from the experiment. This presents an exceptional challenge as the research has shown that stimuli of which people (including the researchers themselves) are not consciously aware can still have a priming affect. Given the complex and opaque nature of the human mind, it seems studying the priming effect may require a process of elimination. A simple a demonstration of a causal effect in a particular combination of circumstances that happen to include the focus stimulus may not be sufficient.
For this reason it is important to ensure maximum capacity to examine and reproduce a particular experiment. The most important part of that is providing detail on the stimulus material, as this will allow review of that material against alternative theories and tested in alternative scenarios. During my research I found what I thought was a great example of how to provide information on stimulus material in a 1984 study into priming and the stroop effect. The study found a correlation between the level of conscious awareness of the prime and the magnitude of the priming effect, but this wasn’t the only thing I think is interesting. Note the level of detail cognitive psychologists Jim Cheeseman and Philip Merikle11 recorded their description of the stimulus material:
During the experimental trials, each stimulus display was composed of a letter-string prime and color-patch target. The color patches were centrally located rectangles of blue, green, yellow, and orange, which measured .95 cm (.6°) horizontal X .65 cm (.4°) vertical. They were enclosed by black borders that formed a larger rectangle measuring 1.25 cm (.8°) horizontal and .95 em (.6°) vertical. An identical rectangle drawn on a white card served as the fixation stimulus throughout the experiment.
The primes were letter strings that could appear either above or below the color patches. The primes included the four color words, BLUE, GREEN, YELLOW, and ORANGE, as well as a letter string consisting of five Xs. All primes were printed in uppercase 28-pt Helvetica Medium Outline Letraset (No. 2517). The dimensions of each letter were approximately .65 em (.40) horizontal X .80 ern (.5°) vertical, and the vertical distance from the center of a color patch to the center of a letter string was 1.45 em (.9°).
There were three different relationships between primes and targets. For congruent trials, the prime had the same name as the color patch, whereas on incongruent trials, the prime and the target had different color names. For control trials, the letter string of five Xs served as the prime.
This is an example of a description of a stimulus that ensures that competing theroies can be applied to the experiments to consider if they can provide an alternative or supplementary explanation for the impact of stimulus on the resulting measurements. It’s important to note that the stimulus is not the only part of the experiment, and also that it will obviously not be possible to document every single detail of all parts of an experiment. Existing theories can provide a guide as to which elements are important to record even if the experiment isn’t designed to exclude them as possibilities.
Moving on from controversies about the methods social psychologists study priming, some studies have looked at what factors can have an impact on the priming effect. A 2008 study lead by Dirk Smeesters12 looked at how the consistency of people’s views affected the capacity of priming to have effect. In one experiment the subjects were surveyed to measure how consistently they answered Social Value Orientation (SVO) questions which are designed to elicit either pro-self or pro-social responses. Subjects were then subconsciously primed with religious words (e.g.“priest”, “forgiveness”), business words (e.g. “manager”, “turn-over”), or non-word letter patterns (“XXXXX”). The subjects then participated in a ‘dictator game’ where they could decide how many chips they would keep and how many they would give to another unknown person.
The results showed that people who gave consistent pro-social answers were more giving in the game. The more interesting results were that people who gave more consistent SVO responses (whether consistently pro-self or consistently pro-social) were not affected by the prime while those who gave less consistent SVO responses were affected. The discusson noted “These results are in line with our theorizing that individuals with a highly accessible self-concept (such as high consistent SVO individuals) are less susceptible to priming influences than are individuals with a less accessible self-concept (such as low consistent SVO individuals).”
A second experiment duplicated the first, except that half the subjects were first primed though a translation task with words that would encourage a sense of self (“me”, “myself”) while the control were primed with locations (“city”, “village”). The subjects were then primed a second time as per the first experiment and participated in the same dictator game. The results showed that being pre-primed with a sense of self resulted in no effect from the religious or business word priming, while the others showed similar priming effect to the first experiment. The discussion talked about two boundary conditions for priming:
“A first, and important, boundary condition for priming effects is the chronic accessibility of behavioral responses… high consistent SVO individuals were impervious to primed religious and business constructs …
A second boundary condition is the activation of the self. … we found that primes only determined the behavior of low consistent SVO individuals when the self was not activated.”
A similar 2008 study by S. Christian Wheeler13 looked at how self-consciousness moderated the effects of priming. It found that a strong internal-state-awareness (e.g., “I am alert to changes in my mood”) reduced the magnitude of the priming effect while high self-reflectiveness (e.g., “I’m always trying to figure myself out”) resulted in increased priming effect. These studies show that priming can be as much about the individual subjects involved as it is about the priming stimulus.
Considering the qualities of the subjects brings us back to something I noted at the beginning of this post: the widespread use of psychology undergraduates as subjects for psychology experiments. One of the contempory concerns in researching psychology is how this makes results WEIRD. That is, the results only reflect people from Western Educated Industrialised Rich Democratic societies. Also important to note is that the subjects are typically in the 18-25 year old range. Given the recent change in views that adolescence extends throughout this range, it raises the question of how widely applicable the results of psychological studies are to even western societies. That is something that’s more closely linked to something I’ll look at in the next post.
In this post I’ve focused on the priming theory and how social psychologists have attempted to study its social and behavioural impacts. In the next post I’ll be moving on to focus on some of the principles behind the self-reporting of values and beliefs that dominate the measurements in the FeministFrequency studies.
1. Steve Loughnan, et al. (2010). Objectification leads to depersonalization: The denial of mind and moral concern to objectified others. Journal of Experimental Social Psychology, 40(5), 709–717.
2. Laurie A Rudman & Eugene Bordgia (1994). The Afterglow of Construct Accessibility – The Behavioral Consequences of Priming Men to View Women as Sexual Objects. Journal of Experimental Social Psychology, 31(6), 493–517.
3. Karen E. Dill, et al. (2008). Effects of exposure to sex-stereotyped video game characters on tolerance of sexual harassment. European Journal of Social Psychology, 44(5), 1402–1408.
4. Mike Z. Yao, et al. (2010). Sexual Priming, Gender Stereotyping, and Likelihood to Sexually Harass: Examining the Cognitive Effects of Playing a Sexually-Explicit Video Game. Sex Roles, 62(1), 77-88.
5. Jesse Fox, et al. (2013). The embodiment of sexualized virtual selves: The Proteus effect and experiences of self-objectification via avatar. Computers in Human Behaviour, 29(3), 930–938.
6. Jesse Fox & Wai Yen Tang (2014). Sexism in online video games: The role of conformity to masculine norms and social dominance orientation. Computers in Human Behaviour, 33, 314–320.
7. Endel Tulving, et al. (1982) Priming Effects in World-Fragment Completion Are Independent of Recognition Memory, Journal of Experimental Psychology 8(4), 336-342.
8. John A. Bargh, et al. (1996) Automaticity of Social Behavior: Direct Effects of Trait Construct and Stereotype Activation on Action, Journal of Personality and Social Psychology, 71(2), 230-244.
9. Stephane Doyen, et al. (2012) Behavioral Priming: It’s all in the Mind, but Whose Mind?, PLoS ONE, 7(1): e29081.
10. Bruce Bower (2012), The Hot and Cold of Priming, Science News, 181(10), 26-29.
11. Jim Cheesman & Philip M. Merikle (1984), Priming with and without Awareness, Perception & Psychophysics, 36 (4), 387-395.
12. Dirk Smeesters, et al. (2008) When do primes prime? The moderating role of the self-concept in individuals’ susceptibility to priming effects on social behavior, Journal of Experimental Social Psychology 45, 211-216.
13. S. Christian Wheeler, et al. (2008) Does self-consciousness increase or decrease priming effects? It depends (2008), Journal of Experimental Social Psychology, 44, 882–889.