As a group, we not only offer tools for people to engage with the issue of gender bias in book reviewing, but also seek to understand the bias being witnessed. Thus, we have been collecting data and performing studies involving both statistical analyses and qualitative surveys to better illustrate the points which need addressing to solve this issue.
We currently have three research projects in the works:
- the first is a statistical analysis of the themes associated to books reviewed in the New York Times Book Review between 2004-2014 according to book author gender;
- the second is a survey we conducted about the selection processes favoured by different publications through out the English reviewing world;
- the third is an analysis of the most distinguishing word features extracted through machine learning techniques associated to these reviews according to book author gender.
In order to understand the extent to which different genres are unequally represented among books that are reviewed and written by men and women, we conducted an initial study using data from the New York Times Book Review. We used Goodreads to match the books reviewed to a set of genres, which we used to create this analysis.
You may download the full report on this data analysis here.
Theme | Women (#) | Women(%) | Men(#) | Men(%) | P-Value | Skew |
---|---|---|---|---|---|---|
Gender | 166 | 64 | 92 | 36 | < 0.001 | F |
Romance | 266 | 47 | 304 | 53 | < 0.001 | F |
Government | 50 | 16 | 254 | 84 | < 0.001 | M |
Family | 79 | 59 | 54 | 41 | < 0.001 | F |
Economics | 15 | 13 | 102 | 87 | < 0.001 | M |
Science | 168 | 25 | 504 | 75 | < 0.001 | M |
Humanities | 23 | 16 | 120 | 84 | < 0.001 | M |
Youth | 236 | 42 | 325 | 58 | < 0.001 | F |
Magical Realism | 33 | 57 | 25 | 43 | < 0.001 | F |
Spirituality | 36 | 21 | 138 | 79 | < 0.001 | M |
Sports | 9 | 13 | 58 | 87 | < 0.001 | M |
War | 47 | 22 | 166 | 78 | < 0.001 | M |
Adventure | 58 | 23 | 192 | 77 | < 0.001 | M |
Business | 18 | 18 | 81 | 82 | 0.001 | M |
Sexuality | 37 | 47 | 41 | 53 | 0.011 | F |
History | 289 | 29 | 693 | 71 | 0.013 | M |
Crime/Mystery | 132 | 29 | 328 | 71 | 0.042 | M |
Arts | 88 | 29 | 224 | 72 | 0.071 | M |
Fantasy | 135 | 37 | 227 | 67 | 0.094 | F |
Comedy | 53 | 29 | 132 | 71 | 0.212 | M |
Environment | 57 | 38 | 93 | 62 | 0.225 | F |
American | 36 | 39 | 57 | 61 | 0.271 | F |
Social Science | 62 | 29 | 149 | 71 | 0.273 | M |
Military History | 8 | 24 | 26 | 76 | 0.277 | M |
Health/Body | 19 | 27 | 52 | 73 | 0.313 | M |
Education | 10 | 38 | 16 | 62 | 0.54 | F |
Leisure | 48 | 31 | 108 | 69 | 0.553 | M |
Self-Help | 13 | 37 | 22 | 63 | 0.595 | F |
Race | 8 | 28 | 21 | 72 | 0.694 | M |
Asian Literature | 9 | 28 | 23 | 72 | 0.708 | M |
European Literature | 63 | 33 | 129 | 67 | 1 | M |
Dark / Death | 41 | 33 | 84 | 67 | 1 | M |
Social Issues | 8 | 32 | 17 | 68 | 1 | M |
In order to understand how topic bias may be manifesting in book reviews, our research team felt it was important to first understand how publications select the books they review. Our hope was that having a better sense of this process would help us identify where topic bias may be creeping in and who might have the power to address it.
As such, we designed a survey that would give us insight into the book selection process. We sent the survey out to 97 publications and received 29 responses.
Here’s some of the key takeaways from the response data:
- Publications do care about diversity. Two thirds of publications value having a diverse range of authors represented in their reviews section and one quarter of publications actively look for diverse authors when selecting books to review. This means that at least a quarter of publications should be open to combatting topic bias.
- Section editors have the power to make change. Section editors (i.e. arts editors, books editors in newspapers/magazines), followed by reviewers, have the most influence over which books get reviewed. Advocacy around topic bias should then focus on these two targets.
- Publications already have useful book review data. While most publications do not have standardized selection processes – meaning biases can easily creep in – most publications (55.2%) do track the books they review, and some publications (38.3%) also track all the books they receive for review.
To gain further traction on the ways in which the content of books chosen to be reviewed differs between male and female authors, we also decided to examine the language within the reviews. To test if there were biases in the language of reviews towards men and women, we decided to train a classifier and extract words it found were most deterministic when deciding when a reviews was written about a female author or a male author's book. For those unfamiliar with the term classifier in machine learning, it is an algorithm whose task is to attach a label, in this case author gender, to an input, in this case the review, by determining to which previously trained on labelled group the review is most similar. In this case, our classifier was trained on 6738 (50/50 - male/female) randomly selected reviews from the New York Times book reviews from 2004-2014, and then tested for an accuracy score on 100 reviews.
We tried two different sets of word features, one containing the 1500 most common words and 2000 most common words having removed stop words (pronouns, conjuncts, determiners...). In the first case, the classifier reached an accuracy rate just shy of 70% and in the second 60%. Below are listed the top 25 most deterministic words for the first model:
The list suggests yet again that the problem may lie not in the reviewers’ bias, but in the themes of the books selected for review, with female authored books concentrating on family (boyfriend, grandmother, mothers, daughter) and the "near" (nearing) , while male authored books on society (entities, congress, commander, leadership) and the "far" (farthest).
We tried two different sets of word features, one containing the 1500 most common words and 2000 most common words having removed stop words (pronouns, conjuncts, determiners...). In the first case, the classifier reached an accuracy rate just shy of 70% and in the second 60%. Below are listed the top 25 most deterministic words for the first model:
The list suggests yet again that the problem may lie not in the reviewers’ bias, but in the themes of the books selected for review, with female authored books concentrating on family (boyfriend, grandmother, mothers, daughter) and the "near" (nearing) , while male authored books on society (entities, congress, commander, leadership) and the "far" (farthest).
Contains the word | Ratio |
immunity | male : female = 4.2 : 1.0 |
entities | male : female = 3.8 : 1.0 |
congress | male : female = 3.6 : 1.0 |
gist | male : female = 3.6 : 1.0 |
wires | male : female = 3.3 : 1.0 |
fabricated | male : female = 3.2 : 1.0 |
farthest | male : female = 3.0 : 1.0 |
geological | male : female = 3.0 : 1.0 |
forgery | male : female = 3.0 : 1.0 |
commander | male : female = 2.9 : 1.0 |
leadership | male : female = 2.8 : 1.0 |
herself | female : male = 5.3 : 1.0 |
cremated | female : male = 4.4 : 1.0 |
boyfriend | female : male = 4.3 : 1.0 |
halftruths | female : male = 3.7 : 1.0 |
discriminatory | female : male = 3.4 : 1.0 |
nearing | female : male = 3.4 : 1.0 |
grandmother | female : male = 3.3 : 1.0 |
northwest | female : male = 3.3 : 1.0 |
rumored | female : male = 3.2 : 1.0 |
boarded | female : male = 3.2 : 1.0 |
connectedness | female : male = 3.0 : 1.0 |
rentcontrolled | female : male = 3.0 : 1.0 |
mothers | female : male = 3.0 : 1.0 |
she | female : male = 2.9 : 1.0 |
daughter | female : male = 2.8 : 1.0 |