Anonymous Grading

Background

Higher education as a hostile environment

• Higher education is a hostile environment for various groups.
• Demographics (race, gender, class) are correlated with achievement (though which demographics varies depending on discipline – e.g., women much more under-represented among philosophy students than in some other disciplines (Dougherty, Baron, and Miller 2015).

Controversial Explanations

• Explanations why are controversial.
  • Is it a simple case of old-fashioned overt discrimination? – all too often it still is (Steele, James, and Barnett 2002).
  • Implicit bias? – likely have some impact, but likely that effect size is overstated (and I wouldn’t want to be betting that the relevant psych results replicate).
  • Gendered/etc. schemas (Haslanger 2008; Leslie, Cimpian, et al. 2015)? – likely, especially via formation of differential interests at pre-university stage (Baron, Dougherty, and Miller 2015).
  • ‘Differential interest’? – is there likely to be any distinct role for endogenous preference by group?
• Good quality (or even any) data is lacking to decide between these explanations.

Addressing Potential Disciplinary Hostility

• But the explanations are beside the point!

  The uncertainty regarding whether or not [a discipline] is hostile to [some group] should not prevent us from acting as if such discrimination were a real possibility…. (Benétreau-Dupin and Beaulac 2015: 69–70)

• We don’t have control over how our students were socialised pre-university, or what they’re exposed to outside of our courses.

• Unfortunately, we don’t have much control either over the selves we present to our students as role models.

  • Given current university priorities, if your department doesn’t have a staff member as a role model for potential students from a given group, you are unlikely to be able to hire one.

Improving Classroom Practice

• So if we are interested in addressing hostile environments, our most immediate impact is via classroom practice.
• Some hypotheses suggest that course content and/or teaching style is responsible for some of these effects:
  • e.g., that the ‘hyper-masculine’ argumentative style of philosophy disproportionately discourages women (Moulton 1989); or that the abstract and rarefied style of philosophical argument discourages those socialised to be more concerned with practical examples – women, low-SES background students (Dodds and Goddard 2013). There isn’t strong evidence for such effects having a big role (Baron, Dougherty, and Miller 2015: §4.2).
  • That said, you definitely need to diversify your reading lists – some evidence that overt challenges to embedded schemas, giving students non-token models that don’t fit their prior expectations, can disrupt them.

Anonymity

Anonymity and Fairness

• A different kind of classroom intervention is less focussed on redressing representation problems for some particular group, but avoiding exploitation of such problems.
• It is difficult to treat someone unfairly because of their group membership if you don’t suspect what groups they are a member of.
• So anonymity can be a useful tool in protecting students from overt discrimination, implicit bias, and the impact of gendered schemas on the interpretation of student performance.
• There are lots of places in classroom practice where anonymity could be at best awkward.
  • Anyone who’s had a zoom tutorial comprising a sea of black boxes can tell that anonymous small group teaching is not ideal.

Anonymous Assessment

• Unfair assessment has lasting impacts.
• Anonymous assessment is an easy way of avoiding those impacts.
  • Please call it anonymous, not blind!
• Anonymous assessment is the introduction of practices that attempt to enable the marking of student work without the marker being aware of the identity of the person whose work is being marked.
  • You can also anonymise your comments, but that’s not what I’m advocating.
• This is obviously not practical for some forms of assessment: class participation, presentations, and group work are not generally suitable for anonymous assessment.
• Essays (and exams) are the principal forms of easily anonymised assessment.

Why Adopt Anonymous Assessment?

Why Anonymous Assessment?

1. Fairness
   • Fairer for students, who are protected from conscious or unconscious bias;
     
   • Protects markers from accusations of favouritism or bias, and bolsters student confidence in assessment regime;
2. Accuracy
   • Improves validity of assessment regime;
3. Compliance
   • Encourages compliance with Assessment for Coursework Programs Policy

Fairness for students

• Prevents malicious or retaliatory treatment of individual students.
• Protects also against unfairness caused by biases in expectations.
  • I.e., no extra marks because ‘this was surprisingly good for X’, or penalties because ‘this work didn’t live up to Y’s usual high standards’, or ‘I’ll give this murky argument the benefit of the doubt because Z is on the ball in class discussion’.
• In line with university policy, one aim of assessment is to facilitate ‘judgements about student achievements’ – anonymous assessment focusses attention on the work embodying those achievements, rather than on the producer. All students treated equally on the basis of the specific assessment task, rather than on track record or ‘potential’.
• Addresses stereotype threat (Steele 1997), managing student concern about how our attitudes to them might affect their mark.

Protection for Staff

• If students know grading is anonymous, more difficult for them to accuse markers of bias or favouritism (Dorsey and Colliver 1995).
  • Even if bias isn’t a factor in practice, disgruntled students may sincerely believe (or insincerely allege) that staff are biased against them.
• Anonymous grading is a barrier to students invoking the grievance process over marks: it will not be possible to demonstrate bias, and there is reduced scope for marks to be based on factors outside the learning outcomes explicitly tested by the assessment task.
• Marking anonymously sends a signal to students that demonstrated achievement is what matters in your course – it can thus improve student’s trust in your assessment regime.
  • Example: when US law schools started using affirmative action in admissions, they also moved to anonymous grading, apparently to forestall complaints that affirmative action admittees were being graded more leniently (Carrington 1992: 565).

Improved Validity

• Validity of a measure is the ‘extent to which the scores from a measure represent the variable they are intended to’ (Price, Jhangiani, and Chiang 2015: 99).
• It is likely that anonymous grading improves validity of assessment conceived as a measurement of the quality of a piece of work, by eliminating a source of noise. Confounding factors are eliminated.
  • Not necessarily uncorrelated with quality (consider high-SES educational background which is correlated with quality), but screened off: the probability of a piece of work being good, given the evidence gained by reading it, is independent of demographic variables.
• Anonymity doesn’t control every source of noise – marker fatigue, reading order, spelling or other presentational errors all influence judgements of quality independently of the actual quality.
  • Can actually help with reading order though, since most implementations anonymise in a way that randomises order too.

Better Compliance

• University policy https://www.adelaide.edu.au/policies/700/ is that
  • ‘Marks reflect the actual achievement of the student(s)’;
  • ‘Assessment must enable robust and fair judgements about student performance’
  • ‘Assessment practices must be … undertaken … with honesty and integrity’;
  • ‘Gender, racial or cultural bias is avoided’;
  • ‘Grades are applied consistently’
• Anonymous assessment can foster all of these outcomes.

Objections

Objections

1. Unreliability
2. ‘I want to see student’s work in the context of their growth over the semester’; relatedly, ‘I want to assess students, not assignments’
3. ‘I think the idea of objective assignment “quality” is a construct, itself harming students. Anonymous grading reinforces it, and stops me from mitigating it’
4. ‘What about students with disability accommodations or other needs that should be recognised in assessment?’

Reliability

• The reliability of a measure is the extent to which the scores from a measure are consistently reproducible.
  • Sometimes called ‘test-retest reliability’ (Price, Jhangiani, and Chiang 2015: 96).
• Biased marking systems can be reliable for the wrong reasons: expectations of performance, based on bias or past experience, swamp actual evaluation of quality.
  • E.g., giving students the average of their previous marks on the next assignment is very reliable, but not at all valid.
• Reply: Anonymous marking is likely less reliable, because the steadying hand of expectation is absent. This is probably ok.

Assessing Students

• Some course coordinators want to reward students who improve over the course of semester, or penalise students who coast after good early assignments. They want a more ‘holistic’ kind of assessment of the student. Anonymous marking makes this difficult.
• Reply: I don’t think you should do this. But if you want to, make it a specific learning outcome, and add a component assignment targetting it.
  • E.g., give 10% of the course mark to the gradient of their mark trajectory.
• Focusing on the work means anonymous grading is less valid as a measure of student performance/ability.
  • Luckily, we already have measures of student performance: mean grade. Don’t try to use assessment of individual essays as a poor GPA proxy.
• There should be quite a lot of inconsistency from assignment to assignment. I suspect if we did an audit, we’d find less variability than we ought to, statistically speaking.

Objective Standards

• What if you think assessment quality, as constructed by University policy, is structurally unfair to certain groups, so that better attempts to measure it reinforce those biases?
  • For example: you might think our grade descriptors build in a structural bias against first in family, low-SES background, or international students. You might think coherence/clarity/clear expression are oppressive ideals, and that we need to recognise the way such ideals interact with sources of disadvantage among our students to assess them fairly.
• Reply: let’s grant, for the sake of argument, that there is structural bias in the grading standards. Two options:
  1. One response is to keep those standards, and then try to compensate students whose performance we expect might have been harmed by holding it to such standards, based on their apparent demographic presentation. This requires non-anonymous grading.
  2. Design better assessment tasks that don’t reinforce those bad standards!

Special Circumstances

• What if a student has revealed, through an access plan or extension request, some special circumstances that need to be taken into consideration in determining how well the submitted work reflects their grasp of the material?
  • This is the best objection to anonymous grading to my mind.
• Reply: you can often maintain anonymity.
  • E.g., you can defer marking until all extended assignments are in; if you use individual due dates in Canvas, you won’t know which assignments had an extension, and so you won’t be subject to any bias against students who request extensions, since you won’t know who they are.
• Second, you can adjust marks afterwards in such cases; you will at least be working from a robust baseline evaluation of the quality of the work submitted, and then you can decide how much the mark needs to increase to reflect the circumstantial impact on the student’s performance in the production of that work.

References