As a group, we not only offer tools for people to engage with the issue of gender bias in book reviewing, but also seek to understand the bias being witnessed. Thus, we have been collecting data and performing studies involving both statistical analyses and qualitative surveys to better illustrate the points which need addressing to solve this issue. We currently have three research projects in the works:
    • the first is a statistical analysis of the themes associated to books reviewed in the New York Times Book Review between 2004-2014 according to book author gender;
    • the second is a survey we conducted about the selection processes favoured by different publications through out the English reviewing world;
    • the third is an analysis of the most distinguishing word features extracted through machine learning techniques associated to these reviews according to book author gender.
These can be found in the tabs below.
In order to understand the extent to which different genres are unequally represented among books that are reviewed and written by men and women, we conducted an initial study using data from the New York Times Book Review. We used Goodreads to match the books reviewed to a set of genres, which we used to create this analysis. You may download the full report on this data analysis here.
ThemeWomen (#)Women(%)Men(#)Men(%)P-ValueSkew
Gender166649236< 0.001F
Romance2664730453< 0.001F
Government501625484< 0.001M
Family79595441< 0.001F
Economics151310287< 0.001M
Science1682550475< 0.001M
Humanities231612084< 0.001M
Youth2364232558< 0.001F
Magical Realism33572543< 0.001F
Spirituality362113879< 0.001M
Sports9135887< 0.001M
War472216678< 0.001M
Adventure582319277< 0.001M
Social Science6229149710.273M
Military History82426760.277M
Asian Literature92823720.708M
European Literature6333129671M
Dark / Death413384671M
Social Issues83217681M
In order to understand how topic bias may be manifesting in book reviews, our research team felt it was important to first understand how publications select the books they review. Our hope was that having a better sense of this process would help us identify where topic bias may be creeping in and who might have the power to address it. As such, we designed a survey that would give us insight into the book selection process. We sent the survey out to 97 publications and received 29 responses. Here’s some of the key takeaways from the response data:
  1. Publications do care about diversity. Two thirds of publications value having a diverse range of authors represented in their reviews section and one quarter of publications actively look for diverse authors when selecting books to review. This means that at least a quarter of publications should be open to combatting topic bias.
  1. Section editors have the power to make change. Section editors (i.e. arts editors, books editors in newspapers/magazines), followed by reviewers, have the most influence over which books get reviewed. Advocacy around topic bias should then focus on these two targets.
  1. Publications already have useful book review data. While most publications do not have standardized selection processes – meaning biases can easily creep in – most publications (55.2%) do track the books they review, and some publications (38.3%) also track all the books they receive for review.
Publications have the motivation, the means, and some of the data they need to make change. What they’re missing is actual knowledge of their own topic bias. This is where we come in: Just Review has developed a self-assessment tool that publications can use on the track data they already have from tracking their reviews to assess the level of topic bias in their publication. If section editors can determine where topic bias is occurring, they can take measures to stop it. For the full survey results, see here: Survey Data Summaries and Conclusions.
To gain further traction on the ways in which the content of books chosen to be reviewed differs between male and female authors, we also decided to examine the language within the reviews. To test if there were biases in the language of reviews towards men and women, we decided to train a classifier and extract words it found were most deterministic when deciding when a reviews was written about a female author or a male author's book. For those unfamiliar with the term classifier in machine learning, it is an algorithm whose task is to attach a label, in this case author gender, to an input, in this case the review, by determining to which previously trained on labelled group the review is most similar. In this case, our classifier was trained on 6738 (50/50 - male/female) randomly selected reviews from the New York Times book reviews from 2004-2014, and then tested for an accuracy score on 100 reviews.

We tried two different sets of word features, one containing the 1500 most common words and 2000 most common words having removed stop words (pronouns, conjuncts, determiners...). In the first case, the classifier reached an accuracy rate just shy of 70% and in the second 60%. Below are listed the top 25 most deterministic words for the first model:

The list suggests yet again that the problem may lie not in the reviewers’ bias, but in the themes of the books selected for review, with female authored books concentrating on family (boyfriend, grandmother, mothers, daughter) and the "near" (nearing) , while male authored books on society (entities, congress, commander, leadership) and the "far" (farthest).

Contains the word Ratio
immunity male : female = 4.2 : 1.0
entities male : female = 3.8 : 1.0
congress male : female = 3.6 : 1.0
gist male : female = 3.6 : 1.0
wires male : female = 3.3 : 1.0
fabricated male : female = 3.2 : 1.0
farthest male : female = 3.0 : 1.0
geological male : female = 3.0 : 1.0
forgery male : female = 3.0 : 1.0
commander male : female = 2.9 : 1.0
leadership male : female = 2.8 : 1.0
herself female : male = 5.3 : 1.0
cremated female : male = 4.4 : 1.0
boyfriend female : male = 4.3 : 1.0
halftruths female : male = 3.7 : 1.0
discriminatory female : male = 3.4 : 1.0
nearing female : male = 3.4 : 1.0
grandmother female : male = 3.3 : 1.0
northwest female : male = 3.3 : 1.0
rumored female : male = 3.2 : 1.0
boarded female : male = 3.2 : 1.0
connectedness female : male = 3.0 : 1.0
rentcontrolled female : male = 3.0 : 1.0
mothers female : male = 3.0 : 1.0
she female : male = 2.9 : 1.0
daughter female : male = 2.8 : 1.0