Death, taxes and computer assisted review!

Introduction

Has the time come for a radical change in how eDiscovery is undertaken? When we consider all of the events that are taking place in courts the world over, perhaps computer assisted review (CAR), which includes predictive coding¹, should be adopted as best practice, especially in cases involving large amounts of electronically sourced information (ESI).

As we discussed in our Forensics Matters Publication No. 12-03 Is Predictive Coding the electronic discovery ‘Magic Bullet’?, the US Judiciary appears to be taking a proactive approach to adopting CAR.

In this article, Craig Macaulay and Harvinder Singh further discuss how the international eDiscovery landscape is moving towards the use of technology over traditional manual review and what this means in Australia.

Why should we use CAR in discovery?

• The desire of the judiciary to examine ways to reduce the enormous burden, both in time and money, of discovering documents.
• Increased speed as studies show that CAR allows recall of up to 70% of relevant documents, from an initial manual review of less than 1% of the total document set.
• Increased consistency (better risk management) as studies show that even under ideal conditions manual reviewers will only code documents the same way no more than 65% of the time.
• Increased proactiveness by the judiciary who are showing a willingness (in some cases eagerness) to explore a number of methods to increase efficiency and effectiveness in dispute resolution, such as expert conferences and concurrent evidence.
• Increased flexibility as the use of CAR does not mean that the matter is locked into that process. CAR lends itself to a varied approach, allowing reversion to manual review if required.

Craig Macaulay, Director, and Harvinder Singh, Senior Executive Analyst, from our Melbourne office, both specialise in Forensic Technology and eDiscovery.

The effectiveness of computer assisted review

The prevailing view has been that the use of CAR, whilst being a cost saving, was an inferior alternative to traditional manual review. However, research has indicated that CAR can produce superior results.

The Text Retrieval Conference (TREC) Legal Track Interactive Task Studies provide an insight into the use of and potential effectiveness of CAR. TREC is co-sponsored by the National Institute of Standards and Technology and the US Department of Defence. It was established to support research within the information retrieval community by providing the infrastructure necessary for large-scale evaluation of text retrieval methodologies. TREC have run a Legal Track interactive research exercise every year since 2006. Its specific goals include:

• Encouraging research in information retrieval based on large test collection.
• Increasing communication among industry, academia, and government by creating an open forum for the exchange of research ideas.
• Speeding the transfer of technology from research labs into commercial products by demonstrating substantial improvements in retrieval methodologies on real-world problems.
• Increasing the availability of appropriate evaluation techniques for use by industry and academia.

The TREC 2011 Legal Track interactive research exercise showed that 70% of the relevant documents could be identified from a seed set of less than 1% of the corpus using CAR processes.²

The TREC 2009 Legal Track interactive research exercise³ involved 11 teams, each using their own CAR processes to categorise 800,000 documents as being responsive or non-responsive to a series of simulated document requests. In addition, manual reviewers were set the same task for subsets of the 800,000 documents. The evaluation measures used were:

• Precision – a measure of how often documents that were marked as responsive were actually responsive, for example if 100 documents were marked as responsive but only 50 of those documents were actually responsive then precision would be 50%.
• Recall – a measure of how successful a team was at finding responsive documents, for example if 2000 documents in the document set were actually responsive but only 1000 of those documents were marked as responsive then recall would be 50%.
• An average of these two measures.

The results of the exercise were analysed by Grossman and Cormack in their 2011 article in the Richmond Journal of Law.⁴ Two of the top performing CAR teams were selected and their performance compared against that of the manual reviewers. The CAR teams outperformed the manual reviewers in four of the five selected simulated document requests. The Grossman and Cormack analysis indicated that CAR could yield more accurate results with less effort and cost than manual review.

The Grossman and Cormack research suggested that there is an upper bound on the maximum effectiveness of manual human review of 65% precision and 65% recall, as that is the level at which humans agree with one another. To put it another way, reviewers disagree at least as often as they agree that a document is relevant.⁵

Further cases supporting computer assisted review

Given the continuing research into the efficiency and cost effectiveness of CAR methodologies it is unsurprising that further cases have come to light which reflect the US Judiciary’s willingness to embrace CAR, sometimes against the wishes of the litigating parties themselves.

EORHB Inc. et al v. HOA Holdings LLC C.A. No. 7409-VCL (Del. Ch. Oct. 15, 2012)

Vice Chancellor J. Travis Laster in the Delaware Chancery Court made a Bench Order not only requiring that both parties use predictive coding but that they also use the same vendor. What is particularly interesting in this case is that neither party had requested predictive coding be used. Moreover, Vice Chancellor Laster’s order is contrary to Sedona principle 6 which states “ Responding parties are best situated to evaluate the procedures, methodologies, and techniques appropriate for preserving and producing their own electronically stored information .”⁶

The Delaware Chancery Court is arguably the biggest commercial court in the US. Because of Delaware’s advantageous incorporation laws, many firms incorporate in Delaware. Delaware Chancery Court decisions are highly regarded in other states, potentially making this a game changing decision.

Re: Actos (Pioglitazone) Products Liability Litigation

US Federal Magistrate Judge Hanna Doherty of the Western District of Louisiana entered a Case Management Order outlining the electronically stored information protocol the parties must follow during discovery. The order specifies a ‘Search Methodology Proof of Concept’ which includes predictive coding and advanced analytics sampling components. Essentially the ESI of four key custodians will be used to create a sample collection population against which Equivio’s Relevance predictive coding software will be employed as an alternative to traditional manual review. Whilst this is described as a ‘proof of concept’ the level of detail in the order suggests that the Court is more than acquainted with CAR and appears ready to adopt it over more traditional review methods should the trial yield positive results.

West African Gas Pipeline Company Ltd v Willbros Global Holdings Inc [2012] EWHC 396 (TCC)

The Court made adverse cost orders against the West African Gas Pipeline Company because of significant shortcomings in its eDiscovery process. These failures included inconsistent coding, de-duplication and redaction, inadequate gathering together of a complete set of electronic documents and the failure to properly review documents. This case demonstrates the potential consequences if eDiscovery is approached in an ad hoc or disorganised manner.

UK Perspective

Senior Master Whitaker of the Queen’s Bench, in the Epiq Systems Panel Debate on Judicial Attitudes to Technology Assisted Review made the comment that “Predictive coding is not a universal panacea but it gets the job done better and faster than humans can”. He further drew attention to the ability of predictive coding tools to decide on an acceptable margin of error and to measure output accordingly by stating “The error rate will almost certainly be better than with human review, to the extent that you can ever measure the shortcomings of human review”.⁷

Clearly there is a propensity from the international judiciary to encourage the use of CAR in the discovery process.

Developments in Australia

From a practical perspective the Australian legal environment is grappling with the issue of eDiscovery. Yet all the processes in place are based on the assumption that the parties can easily identify the categories of data within their electronic infrastructure. The reality is that parties do not possess this level of insight and very often are unable to identify where particular categories of documents are located.

Parties will still need to be able to search and triage large amounts of data in order to identify the categories of documents they hold. Until parties develop more robust document management and indexing systems, this has the propensity to be more difficult going forward as ‘big data’⁸ is utilised and greater quantities of data are stored in ‘the cloud’.⁹

Cases in the Australian Jurisdiction have tended to confine their discussions to the need for courts to be mindful that the vast amounts of ESI require a new approach to the discovery process.

In Matthews v SPI Electricity Pty Ltd & Ors [2011] VSC 401, Associate Justice Zammit stated that “The vast amount of data created and stored by electronic means poses a new challenge for the use of discovery as a Court procedure.” He further went on to say that “When intervening in the discovery process, the Court must try to achieve an optimum balance so that discovery is undertaken by each party in the most timely, efficient and cost-effective manner, while ensuring that discovery is proportionate to the size of the case and appropriate to the interests of justice.”

Similarly, the Australian Law Reform and Victorian Law Reform Commissions have both been critical of the increasing time and costs involved in the discovery process, particularly in cases involving significant amounts of ESI. Federal Court Practice Note CM5 reflects these concerns. It deals with eliminating or reducing the burden of discovery by requiring that “In determining whether to make any order for discovery, the Court will have regard to the issues in the case and the order in which they are likely to be resolved, the resources and circumstances of the parties, the likely benefit of discovery and the likely cost of discovery and whether that cost is proportionate to the nature and complexity of the proceeding.”¹⁰

The Victorian Supreme Court Civil Procedure Rules further act to limit discovery. Order 29 states that “In order to prevent unnecessary discovery, the Court may order that discovery by any party shall not be required or shall be limited to such documents or classes of document, or to such of the questions in the proceeding, as are specified in the order.” ¹¹

Victorian Supreme Court Practice Note 1 specifically outlines guidelines for the use of technology in civil litigation matters. These include that:

• Parties and their lawyers consider the use of technology prior to the commencement of any proceeding where it might lead to the more efficient conduct of litigation.
• Lawyers should give consideration to the ways in which the use of technology might lead to the more efficient conduct of the litigation.
• The parties retain an IT consultant to assist them in co-ordinating their implementation of the requirements of this Practice Note.
• The Court may order that a proceeding or certain steps in a proceeding be conducted using technology.¹²

Technology and ignorance is not bliss

In Australia, the inclination of the Courts is to take a dim view of claims of ignorance of technology. In BT (Australasia) Pty Ltd v State of New South Wales & Anor (No 9) [1998] 363 FCA (9 April 1998), Justice Sackville stated that “I do not think that technical sophistication is a prerequisite to a litigant or its advisors making inquiries to ascertain whether discoverable electronic communications or documents have been recorded and retained in a retrievable form.” This comment was directly related to Telstra’s failure to meet its eDiscovery obligations, specifically regarding the retrieval of potentially relevant ESI from backup tapes.

However, such judicial statements are not limited to matters involving sophisticated commercial parties. In Linfox Australia Pty Ltd v Glen Stutsel [2012] FWAFB 7097 , which was an appeal of a workplace dismissal claim to Fair Work Australia, the Full Bench stated that “The claim of ignorance on the part of an older worker, who has enthusiastically embraced the new social networking media but without fully understanding the implications of its use, might be viewed differently in the future.” Whilst not a comment on eDiscovery issues it nevertheless reflects the Court’s view on general ignorance of technology issues.

As knowledge and acceptance of technology increases in Australian Courts, they may in the future no longer view ignorance of CAR favourably.

A key advantage of using CAR tools is that they improve and increase the options available to legal teams:

• CAR leads to better quality and more consistent review outcomes, which not only provides a better outcome for clients, but also improves risk management for legal teams, reducing the likelihood of the issues encountered in the West African Gas Pipeline Company case by ensuring consistent coding of documents.
• Increased knowledge about the corpus of documents provides useful insights for legal teams allowing them to make better and earlier decisions on case assessment, allocation of resources and issues to be pleaded.
• At any time legal teams are free to revert to more traditional review protocols but even within that sub-set, CAR allows legal teams to understand where to prioritise their review to get the maximum early return on their investment in the discovery process.

Conclusion

In an effort to deal with ever increasing amounts of ESI, judiciaries across the globe are embracing CAR tools and methodologies in an effort to manage mega-litigations, reduce costs and improve efficiency.

Although the Australian Courts appear to be approaching the issue from the perspective of case management, specifically by placing limitations on the discovery process, the use of technology to manage ESI has begun to make inroads. Those in the legal community who fail to understand, embrace and make use of the tools available, may find themselves caught out and not able to provide best practice legal services to their clients.

It is only a matter of time before the efficient and effective use of CAR becomes a real competitive advantage used to attract clients who want cost effective legal solutions.

“The horse is here to stay but the automobile is only a novelty — a fad.”

The president of the Michigan Savings Bank advising Henry Ford’s lawyer, Horace Rackham, not to invest in the Ford Motor Co., 1903

Craig Macaulay and Harvinder Singh, KordaMentha

Footnotes

1.  1. CAR or predictive coding is an alternative to the traditional manual review of documents and involves:The manual review by senior members of a legal team of a sample ‘seed’ set of documents to identify whether they belong to certain categories (relevant, not relevant, privileged, etc)

   Computer analysis to apply the characteristics of the sample ‘seed’ set to the full population of documents to group them into the same categories

   The resulting reviewed document set is consistently categorised, using a process which is both auditable and repeatable.

2. Maura R. Grossman, et al, Overview of the TREC 2011 Legal Track http://trec.nist.gov/pubs/trec20/papers/LEGAL.OVERVIEW.2011.pdf
3. Bruce Hedin, et al, Overview of the TREC 2009 Legal Track http://trec.nist.gov/pubs/trec18/papers/LEGAL09.OVERVIEW.pdf
4. Maura R. Grossman & Gordon V. Cormack, Technology-Assisted Review in E-Discovery Can Be More Effective and More Efficient Than Exhaustive Manual Review, XVII RICH. J.L. & TECH. 11 (2011), http://jolt.richmond.edu/v17i3/article11.pdf
5. Ellen M. Voorhees, Variations in relevance judgments and the measurement of retrieval effectiveness, 36:5 Information Processing & Management 697, 701 (2000)
6. The Sedona Conference Best Practices Commentary on the Use of Search and Information Retrieval Methods in E-Discovery, 8 Sedona Conference J. 189, 193 (June 2007)
7. Epiq Systems, Epiq Panel Debate: Judicial Attitudes to Technology Assisted Review, held on 27 June 2012. Webcast available at http://www1.axisto.co.uk/webcasting/investis/epiq/epiq-panel-debate/
8. A term used to describe the exponential growth, availability and use of information, both structured and unstructured. Big data usually includes data sets with sizes beyond the ability of commonly used software tools to capture, curate, manage, and process the data within a tolerable elapsed time. Big data sizes are a constantly moving target, as of 2012 ranging from a few dozen terabytes to many petabytes of data in a single data set. Examples include social networks, social data, document and data archives.
9. A computing paradigm in which tasks are assigned to a combination of connections, software and services accessed over a network. This network of servers and connections is collectively known as ‘the cloud’. Computing at the scale of the cloud allows users to access supercomputer-level power. Using a thin client or other access point, like an iPhone, BlackBerry or laptop, users can reach into the cloud for resources as they need them. For this reason, cloud computing has also been described as ‘on-demand computing’.
10. Federal Court of Australia Practice Note CM5
11. Supreme Court (General Civil Procedure) Rules 2005
12. Supreme Court of Victoria Practice Note 1 of 2007

To view the full range of KordaMentha publications visit:

 www.kordamentha.com/publications

This publication, and the information contained therein, is prepared by KordaMentha Forensic Partners and staff. It is of a general nature and is not intended to address the circumstances of any particular individual or entity. It does not constitute advice, legal or otherwise, and should not be relied on as such. Professional advice should be sought prior to actions being taken on any of the information. The authors note that much of the material presented was originally prepared by others and this publication provides a summary of that material and the personal opinions of the authors. Limited liability under a scheme approved under Professional Standards Legislation.