Grading Papers: Measuring Human Review, Testing Classifiers Series Part 3
Random sampling is a powerful eDiscovery tool that can provide you with reliable measurements of the efficacy and efficiency of searches, reviewers, or other classifiers
by Matthew Verga, JD, Xact Data Discovery
In “Pop Quiz: How Do You Test a Search?,” we discussed the application of sampling techniques to testing classifiers and introduced the concepts of recall and precision. In “Show Your Work: Contingency Tables and Error Margins,” we applied those concepts to testing a hypothetical search classifier. In this final Part, we apply them to human document review.
As we discussed previously, a classifier can be a search, a TAR process, or other things – including a human reviewer or a team of human reviewers. Just as a search or a TAR tool is making a series of binary classification decisions, so too are your human reviewers, and the quality of those reviewers’ decisions can be assessed in a similar manner to how you assessed the quality of a search classifier in the last Part. Depending on the scale of your review project, employing these assessment methods can be more efficient than a traditional multi-pass review approach, and in general, they are more precise and informative.
