It is well established that courts will support parties electing to use technology assisted review (TAR) to identify responsive documents in discovery. However, TAR methodologies and quality control (QC) measures are still very grey for many judges and parties alike. What technologies are most appropriate for identifying the scope of the data universe? How do you sufficiently train the system to properly classify documents? How do you demonstrate that the review method adequately found everything? A new order assists in bringing clarity to this grey space surrounding TAR.
On January 3, 2018, Magistrate Judge Jeffrey Gilbert and Special Master Maura Grossman released a detailed search protocol in In Re Broiler Chicken Antitrust Litigation, a complex, document intensive class action currently pending in the Northern District of Illinois. From deduplication and email threading to sampling and recall, the Order covers an array of topics. And rest assured, Grossman adds in plenty of references to transparency, cooperation, and the importance of regular meet and confers.
This new Order will quickly gain momentum outside of chicken industry lawsuits. In your next e-discovery matter, take heed of the Broiler Chicken search and QC parameters to avoid ending up with egg on your face.
Background of the Case
In September 2016, a food service distributor filed a class action suit on behalf of purchasers of broiler chickens from more than a dozen of the country’s top poultry producers. The action alleged producers conspired to manipulate the supply of chicken to keep prices artificially high, driving up profits.
In February 2017, the plaintiffs filed their first set of requests for production. As foreshadowed by Magistrate Judge Gilbert in a September 2017 discovery order, discovery in this case is likely to be complex and expensive, given 3 putative plaintiff classes, nearly 30 defendants, multiple theories of liability, and conduct covering nearly a decade in a $20-30 billion dollar industry. Perceiving potentially thorny ESI issues, in October 2017, Magistrate Judge Gilbert appointed Special Master Grossman to address and resolve disputes regarding e-discovery. Grossman wasted no time, issuing an Order in January 2018 outlining the document sourcing, search, and validation methods to be used by the parties.
Breakdown of the Order
Grossman begins the Order establishing that the parties will be reasonably transparent regarding the universe of documents collected for search, the search terms utilized, and the processes applied for TAR. Notably, throughout the Order, Grossman refers to this process as “TAR/CAL” (CAL signifying Continuous Active Learning) and never once uses the term “predictive coding.”
Next, the Order addresses culling technologies that will be applied prior to searching. Laying this foundation ensures that processing details are clear, so there are no surprises in identifying the scope of documents selected for searching/TAR. Specifically:
- De-Duplication: Data should be de-duplicated by hash value across all document custodians.
- Email Threading: A producing party may choose to only include inclusive emails from a thread, but must disclose this to the receiving party.
- Email Domains: If a party seeks to eliminate domains from a search, it will produce a list of those domain names to the opposing party.
- Targeted Collections: Documents that are not going to be searched should not be used for search term testing.
- Exception Reporting: The producing party must disclose processing exceptions.
- Other Culling: The producing party must disclose other culling parameters if used.
The second section of the Order addresses search methods, dissecting this topic into two areas: TAR/CAL searching and keyword searching. If using TAR, Grossman requires the producing party to disclose the name of the software and vendor, how training will work, what categories of documents will be included and excluded, and what quality control measures will be taken. Similarly, the producing party must disclose comparable details about selected keyword search terms, including stop words, case sensitivity, metadata field searching, synonym searching, and misspellings, to name a few. In developing the processes for TAR and keyword searching, Grossman expects the parties to cooperate and work together in good faith to resolve any differences.
The third section of the Order is devoted to a document review validation protocol, which applies no matter TAR or “exhaustive manual review” is used. In detail, a QC sample is created, consisting of 500 responsive documents, 500 non-responsive documents as coded by a human reviewer, and 2,000 unreviewed documents as the result of TAR. These documents are combined into a blind validation set to be reviewed by a human subject matter expert (SME) in the case. From there, a table of each document and its coding will be prepared and provided to the receiving party and Special Master Grossman. The producing party must also provide copy of each responsive, non-produced document found in the sample, along with recall statistics. At this point, parties must meet and confer about the recall estimate, and the quantity and nature of the responsive documents identified through the sampling process to determine if the review is substantially complete. Grossman strongly warned parties about turning to the recall estimate percentage as the sole indicator of an adequate review. Obviously, if there is a dispute, Special Master Grossman can intercede.
Implications for E-Discovery Professionals
If there were any questions about the complexity of e-discovery, this Order lays those doubts to rest. In the coming days and months, this Order will likely be examined at length across our community, but here are three brief takeaways.
- E-Discovery does not occur in the dark. The sheer number of times that Grossman mentions meet and confers, cooperation, and transparency is awakening. Make friends with your opposing counsel and their e-discovery team because you will be talking to them regularly.
- TAR and keyword searching work together in tandem. Nowhere in the Order did Grossman mention either searching or TAR. It’s almost assumed that parties are using both these tools to their fullest capacities. Moreover, Broiler Chicken is not an outlier in requiring parties to leverage keyword searching and TAR. Recently, Judge Katherine Parker, in Winfield v. City of New York, 2017 U.S. Dist. LEXIS 194413 (S.D.N.Y. Nov. 27, 2017), encouraged the parties to go beyond mere keyword searching, especially when the parties cannot agree on search terms, and also utilize TAR. As e-discovery aficionado, Ralph Losey, explains, “The Broiler Chicken protocol urges parties to use what I call a ‘multimodal’ approach where human intelligence is paired with multiple technologies in forming a legal search. Parties should not rely on a single method.” In sum, savvy parties will know how to harmonize all the e-discovery tools available at their fingertips.
- Get smart on sampling and recall. Perhaps the clearest theme from this Order is the importance of validation. If a myriad of technologies are used to identify responsive documents, then QC is key. Parties better have someone on their e-discovery team that understands math more than just a little and is prepared to have meaningful discussions about recall to demonstrate that the search and review protocol was adequate.
