Cyborg Law and the Professional Network

Daniel Katz has argued recently in a very interesting piece that a contraction in legal jobs, caused by the recession, is also structural: that is underlying changes in the way legal services are purchased and supplied is limiting the number of legal jobs that will be created in the future.  Price competition for lay users of legal services and via, “sophisticated general counsel applying informatics techniques to lower their company’s legal bill” is part of this picture.  For me and, for Katz, a more interesting question is where competition may drive delivery of legal service and the education of lawyer.  Katz suggests the rise of “computation / automation / “soft” artificial intelligence…[to] automate or semi-automate tasks.”  Legal service providers and the lawyers who work within them may look radically different.  The skills of lawyers may have to change and, probably most importantly, the way in which lawyers think about the law and their role within the legal system may also have to change.

For Katz, this process is being driven by increasing computing power and data availability (through reduction in the costs of storage and computing power).  Outsiders with access to data, such as TyMetrix (“Division of the legal informatics conglomerate Wolters Kluwer”) advise, “corporate counsels and other sophisticated clients” on how to get the best value out of lawyers they instruct through the real rate report.

For educationalists and law firms perhaps the most important prediction is that,

“Informatics, computing and technology are going to change both what it means to practice law and to “think like a lawyer.” When it comes to the application of the leading ideas in computation, informatics and other allied disciplines, the market for legal services lags many other industries. In other words, yesterday’s fast is today’s slow and this is only the beginning.  Aided by growing access to large bodies of semi-structured legal information, the most disruptive of all possible displacing technologies -quantitative legal prediction (QLP) -now stands on the horizon. Although different variants of QLP exist, the march toward quantitative legal prediction will define much of the coming innovation in the legal services industry (and it will occur whether you like it or not).”

As the article proceeds it becomes clear Katz is talking of a profound but in some ways modest change, at least for now.  That is, how data can be harnessed to improve, rather than replace, lawyer judgments and in particular predictions about the cost, risk and outcome of cases.

The key to this process appears to be access to data and the ability to analyse human interactions with that data.  Thus just as Google has combined spelling algorithms with analysis of user search data, so perhaps can lawyers, law firms pioneer new approaches develop more automated approaches to legal tasks.  More likely is the scenario whereby outsiders with the scale, investment, data analysis capacity and lack of investment in current business models drive change.  A trouble with many of the examples that are used to illustrate this possibility is that they operate to predict behaviour in fairly stable systems: impressively complicated as a self-driving car is for instance, it operates within the boundaries of physical rules (but also the human unpredictability of those also on the road).  The development of IBM’s Watson, however, suggests greater potential for soft AI:

“From Yorktown Heights, New York -This is Jeopardy! – The IBM Challenge.”   On February 14, 2011 famed announcer Johnny Gilbert stepped to the microphone and unveiled the greatest example to date of performance computing that threatens the core of typical white collar work. The IBM Challenge pitted IBM Watson versus Brad Rutter and Ken Jennings, the two most successful Jeopardy champions in history. After the  multiday challenge there was a clear winner – Machines 2, Humans. Watson made it  look easy. On the edge of facing defeat, Jennings the 74 time consecutive Jeopardy champion stated: “I, for one, welcome our new computer overlords.”

“It is hard to understate just how difficult of a problem it is for a machine to compete in a game such as Jeopardy. Topics are wide ranging and include detailed questions in domains such as history, literature, politics, arts and entertainment, and science. Contestants often confront clues that “involve analyzing subtle meaning, irony, riddles, and other complexities in which humans excel and computers traditionally do not.” Finally, answers typically much be given very quickly – often in 2-3 seconds.

“Watson accomplishes its task without access to the internet and instead uses large  bodies of structured and semi-structured data as it interprets text and refines its answers.  Watson applies a mixture of technologies including Natural Language Processing (NLP),  Information Retrieval (IR), Knowledge Representation and Reasoning, and Machine  Learning (ML).

To be circumspect, Watson has the capacity to make what to human eyes look like baffling mistakes.  However, “It is [also] a working  computer system that is actively being applied to a variety of professional domains notably the field of medicine (i.e. data driven medicine) where individual doctors are called upon to analyze large amounts of information and rapidly execute the best possible judgment.” [e.g. including cancer care]

A crucial point is that, “Quantitative legal prediction [QLP] based technologies are designed to remedy and/or supplement the shortcomings of human reasoners.”   QLP has the potential to extend the data that a practitioner can take account of; counter biases in cognitive judgment (such as the influence of spectacular recent successes or failures); and thus facilitate professional judgment.  If it works.  If this sounds like science fiction it is worth noting that there was a sentencing support system in use in Scotland for some time (I think it may have fallen into disuse, but I stand to be corrected).  As Katz puts it, “the age of quantitative legal prediction is about a mixture of human + machines working together to outperform either working in isolation.”  Move over Allie McBeal or Rumpole; it is time for Cyborg Law?

Well probably something a bit gentler and less threatening, but very interesting nonetheless.  It is worth dwelling for a moment on the areas where QLP is being currently used.  Cost analystics to predict, manage and reduce costs are an area where data became plentiful and is easily analysable and ‘outsiders’ existed with a business opportunity and an increasingly willing client group (General Counsel).  “TyMetrix leveraged its existing relationships as providers of backend billing and payment software to various law departments.” As legal services become increasingly provided within networks of outsourcing as well as firms, the capacity and appetite for innovation may increase.  “Understanding large scale data aggregated across multiple clients was the key to garnering some deep insights, TyMetrix convinced it respective clients to pool and aggregate anonymized billing information for purposes of better understanding the contours of the respective legal marketplace. Using this and other associated metadata.” (See also http://www.datacert.com/ and http://www.skyanalytics.com/).

The disruptive power of greater price competition is still building across the legal services sector.  With question of price come doubts about quality.  It will be interesting to see how far metrics about cost diversify into quality.  Some UK firms have monitored case outcomes of fee earners in quantitative ways where outcomes are easily quantified and often repeated (personal injury cases being one example); the Legal Aid Board flirted with outcome monitoring of criminal practitioner (something which has been adopted in Chile where aquittal rates of defence lawyers are monitored to keep them up [HT Roger Smith]) and there is a history of formulating quality measurement in legal aid work more generally which proved valuable but labour intensive (peer review in particular).  Large scale legal service firms might develop and refine these approaches under pressure from institutional clients (or possibly regulators).

Katz is right to suggest that clients will begin to look more closely and critically at whether judgments about the quality of the lawyers they instruct are correct.  Law firm selection might also be more informed by a metrics based approach, pointing to Lawyer Metrics, “a company devoted to developing data driven and scientifically informed forecasting models that predict the future success of individual lawyers (particularly at or near the entry level) in law firms and other related legal enterprises. Their approach is designed to debias both the hiring decision and the subsequent employee evaluation process. In other words, Lawyer Metrics value proposition is linked to law firm efficiency and “huge gains to be made by focusing on traits or attributes that are actually correlated with performance.” (see their website here)  One of the biases they claim to have weeded out is the tendency to overrate candidates simply because they have the same educational background as those making the hiring decisions.  Ultimately, this may have more of an influence on diversifying the legal profession than (say) in vogue mentoring schemes, but they are not without limitations.

There are other examples of QLP.  Most famously The prediction of (US) Supreme Court decisions was done better by a decision tree than by a body of experts: “While the experts correctly forecast outcomes in 59.1% of cases, the machine got a full 75% right.” And, there is also crowd-sourced prediction, see for example, http://www.fantasyscotus.net/. If machines can predict supreme court cases it does not seem far fetched to suggest machines can assist in predicting optimal offers to settle in personal injury cases for instance.

The limits to QLP are threefold.  One is the availability of data.  Current solutions have been generated where large volumes of data are available.  The second is complexity.  Prediction works better in simple systems but significantly less well in more complex systems.  Nate Silver’s ruminations on baseball show how data richness was important to the Moneyball phenomenon.  Weather prediction shows the limits of prediction with complex systems even where those systems follow basic rules of physics).  Law is getting a bit more like baseball, and not just in the league tables and transfer rounds; an interesting question is how complex is it really and whether the Jocks (or is it the nerds?) can overcome the Arts majors.  This leads me to the third limitation, which I will call the resilience of the professional network but which is really about ideology.  The provision of law takes place within a network of institutions and rules until recently largely governed by professions.  That network has become increasingly less stable of late.  For a number of years, large funders of legal services have exercised their powers to question professional domination of the network.  Government both as a funder and a regulator of the terms of exchange within the network (I am thinking here of reforms to civil litigation funding in particular) is similarly reconfiguring relations within that network.  And furthermore, regulators are increasingly questioning the traditional rules that have organised network relations.  If QLP provides a better understanding of how the network works than rule-based narratives of more traditional models then the network can reconfigure itself; but the judiciary and the professions still retain considerable influence on the shape such change. It still takes an analytic rather than probabilistic approach to legal problems.  A probabilistic approach is in its infancy and has political legitimacy problems to attend to – problems which courts and regulators can emphasise through professional negligence and practice of law/reserved activity rules.  Then again so does mediation and other partial solutions to the travails of developed nations’ legal systems.