One of the interesting counters to discussions about AI predicting legal outcomes (aside from it not always working that well yet), and whether legal robots are really robots* is that a practice at the heart of lawyering is the understanding and giving of reasons. Computers can’t do that, so the story currently goes, and so lawyers are safe. But an interesting study has just been published in Artificial Intelligence and Law called Recognizing cited facts and principles in legal judgements by
Olga Shulayeva, Advaith Siddharthan and Adam Wyner which gives us some inkling into how near or far we might be from machine driven analysis of law’s reasoning.
The aim of the piece was to test whether it is possible to generate accurate summaries of the facts and reasons for decisions in case law. Think of it as a significant step towards automatic headnote generation. Given the overwhelming volume of case data, this is not a trivial advance and it is a technique which suggests the possibility of shaping reasons from unpredictably structured narratives.
What they find is this: they test the human annotation of cases (an expert trains a lay person to identify text in the cases which contain cited facts and relevant principles) and they compare the accuracy of so doing it with an algorithmic approach. They find, “human annotators can achieve reasonable agreement on which sentences in legal judgements contain cited facts and principles (respectively, K = 0:65 and K= 0:95 for inter- and intra-annotator agreement)”. This human comparison forms their ‘gold standard’, the best achievable human classification of relevant facts and law on the cases.
“We further demonstrate that it is feasible to automatically annotate sentences containing such legal facts and principles in a supervised machine learning framework based on linguistic features, reporting per category precision [facts or principles identified as relevant were correct] and recall figures [not missing relevant facts or principles] of between 0.79 and 0.89 for classifying sentences in legal judgements as cited facts, principles or neither using a Bayesian classifier, with an overall j of 0.72 with the human-annotated gold standard.” It looks pretty accurate, in other words; not perfect but good. Like humans.
The test was performed on 50 common law reports taken from BAILII on civil matters, mainly contract, trust and property law cases.
As a result, they conclude, “it is feasible to automatically annotate sentences containing such legal facts and principles to a high standard. The reported studies lay the basis for further applications, including creation of meta-data for search and retrieval purposes, compilation of automated case treatment tables containing summaries about legal principles and material facts of cases, and automated analysis of reasoning patterns and consistency applied in legal argumentation.”
In this way if we see, “determining the authority of a case required a strong grasp of precedent and legal analysis” and finding precedential information as, “‘buried in the sea of irrelevant information” where , “…appellate court decisions are rife with disagreements between the judges on what the law is.” Then, inspite of these concerns, machines can be programmed to learn the task well, even before we consider the efficiency gains of the machines having the potential to do this on large volumes of case law. How do they do it?
They seek to tie, “specific statements of legal principles and evidence to citations within decisions” through what they call, “argumentation zoning” to isolate, “the argumentation used by the author of that paper or judgement.” These arguments can often, they think, be identified linguistically; “a legal principle can, for instance, be indicated by deontic modality, e.g. expressions of must for obligation, must not for..” Principles are, we might be unsurprised to learn, typically stated in the present tense, and facts in the past tense. Certain verbs and verb tenses were potential signifiers; as were, “word pairs that are grammatically linked”; length of the sentence; the position of certain words in the text; and there was a citation in the sentence. Collectively, these factors contributed to accurately being able to identify the facts and principles which (it appears) made up the core of the judgment.
One final comment from the author on its utility:
“This functionality could, for example, allow a legal practitioner to not only search, say in Google, for citations mentioned in a case, but also the associated legal principles and facts, providing deep access to and insight into the development of the law. It would also offer the opportunity to access the law directly rather than via the edited and structured materials made available by legal service providers.
I should add some notes of caution. The author’s acknowledge there are limitations and weaknesses to the approach. For example the system may systematically miss legal principles which are ”active” in reasoning, but need to be, “inferred from the text.” It seemed to me that the amount of text that makes its way into the relevant facts and principles is a reasonably sizeable proportion of the case (from the article it looks like the relevant facts and laws might make up about a third of the case judgment – as was the case for the human annotators). The system does not order or explain the judgment it simply extracts (most of) the relevant information. It’s a step towards being able to extract the reasoning from cases.
——————————————
* Yep, I know the natural response is who cares? But if you like a bit of hype bashing read this and make up your own mind – to my mind it does not matter if something can be called a robot for the titlation of our legal press, it matters whether the product better meets a need. The blog post linked to contains interesting thoughts on that too.
Lawyer Watch 