Quantitative legal prediction and the wisdom of crowds

Professor Dan Katz gave a great talk at UCL’s Centre for Ethics and Law last night .  You can see the slide deck here.  It ranged from his Fantasy SCOTUS predictor to the power of random forests of decision trees to aid concrete legal decision-making.  It turns out the best human predictor of US Supreme Court decisions is probably not some hot shot Washington lawyer but an actuary called Jacob, who can’t explain in legally coherent ways why he is so good at this (let’s face it, we’ve all been there, eh boys?). And that, in Dan’s view, “We have too many decisions in law made by single experts”: a point increasingly recognised by law firm risk committees, litigation funders and (naturally enough) insurers. His essential point was that ensembles of experts, crowds and algorithms are very likely to be the best predictors of legal outcomes.

A point of some note for those interested in how this all really works is Dan’s pioneering courses of Quantitative methods for lawyers and legal analytics.  You can see his courses online and teach yourself.

Posted in Uncategorized | Leave a comment

My response to the SRA’s proposals for an SQE

This response to the SRA’s Consultation on training tomorrow’s solicitors is a personal one.  I am afraid this will not make a great deal of sense to those of you not familiar with the proposals, my apologies in advance.

My response concentrates on the main issue of concern that I have with the SRA’s proposals and in particular the relationship between the proposals, undergraduate legal education and hopes for an innovative, adaptable, high quality legal profession.  That issue can be stated in this way: when the SRA now approaches all its regulatory policy with an emphasis on minimal/proportionate regulation as a necessary precursor to the benefits of innovation, why is the SRA making proposals which will have the effect of dramatically overregulating legal undergraduate education, with a particular likelihood of inhibiting, rather than promoting, innovation?

Put more simply again, my main problem is with the extent of the knowledge requirements, the way they will be assessed and the time at which they will be assessed.  We have traveled from a modest ambition to define the skills and competencies of solicitors for 2020, to a knowledge framework which looks familiar to students of the Law Society Finals and a skills framework familiar from the LPC.  In other words, we have a solicitor’s framework defined, in broad terms, by the Law Society in the 1990s, with the unfortunate language of Part I and Part II thrown in for good measure.

My assumption is that the defining of knowledge across 13 categories and the freeing of the assessment of these categories from the LPC will broadly lead to two types of undergraduate legal education:

  • Group A – a selection of the Russell Group/Golden Triangle law schools that will seek to protect traditional approaches to legal education; and,
  • Group B – the remainder that will, with greater or lesser willingness, be driven by competition for students to teach as many of the knowledge categories in the SQE as possible.

As a result, unless the SQE assessment proves spectacularly successful at upsetting perceptions of law school hierarchy amongst law firms, then the SQE is likely to enforce a segmentation of the legal education market which will reinforce and exacerbate existing problems within the recruitment market for trainees.  As such there is a significant likelihood that the proposals will weaken not strengthen diversity within the legal profession.

The final concern about such an extensive, and backward looking, definition of knowledge requirements is that, depending on the true nature and extent of these, there is a severe risk that the SRA will define out space for substantive innovation in undergraduate legal curricula (save perhaps in those law schools least interested in thinking about how such curricula might better service students and the professions, potentially further cementing the status of Group A).

It also seems likely, on our current state of knowledge about the assessment mechanisms and the approach, that the SQE will drive a prioritisation of a limited kind of knowledge (and a particularly legal kind of knowledge) over the crucial elements that a high quality legal education could provide: understanding law in context; critical thinking; and innovation in thought about law.  In particular, the more the SRA defines, the less space for innovation there is.  This is a plea, and to be honest I am sufficiently worried about the SRA’s proposals to be genuinely pleading, for a dramatically more proportionate approach.

I make some more specific points and develop some of the themes below.

When should knowledge be tested?

The SRA’s focus is – as stated – on, “assuring consistent and comparable high quality standards at the point of admission across all pathways to qualification”.  In other words, the concern is to be assured of what lawyers know once qualified, not at some point prior to that.  The SQE would permit knowledge tests a significant, possibly limitless, time period prior to qualification.  They will show that a solicitor candidate did, at one time, have relevant knowledge of one or more of the SRA’s key areas.[1]  It will not test whether the solicitor has knowledge of the area within which they will actually be practising, unless their areas of practice map squarely onto the SRA’s knowledge area.  Nor will it test the extent to which their historic knowledge has degraded by the time of the qualification.  With such a poor fit between what needs to be assessed for competence, and what and when something is to be assessed, there ought to be a major rethink and slimming down of the general knowledge components required of a solicitor before the point of qualification.  After all, the key thing about being a professional is that one has the knowledge relevant to one’s area of practice and keeps that knowledge up to date.  The knowledge competences are strangely divorced from this key requirement.

Relatedly, the knowledge elements are very wide, and often only tangentially or minimally related to the practice areas of many solicitors.  The list of subjects can fairly be characterised as some fantasy of a generalist solicitor or a backward looking shopping list of subjects that are have generally been taught at degree or LPC level. The knowledge requirements of most practitioners are almost certainly much more specialised than this list.

Similarly superficial and irrelevant generalism may be less rewarding for students too.   We need innovation in the content of law degrees as much as we need innovation in delivery and assessment.  I am not seeing innovatory possibility in a system which requires every would-be solicitor to study wills and probate, and taxation, and the two litigations, for instance.  Most solicitors function reasonably well having forgotten (I’d wager) most of the property and trusts law they were taught even a few years ago. And to those that say, ah yes but you never know when a trusts point will come up (even though the answer will often be, never), it is just as easy to think of other subjects where this is as or more likely.  Intellectual property and employment law are increasingly central to a great deal of business activity insofar as it involves lawyers and may thus more generally relevant than several of the current crop of requirements.  Sensibly defining knowledge requirements on a subject by subject basis as the SRA has done is a fool’s errand: it is way too over and under-inclusive to be useful.

With the impossibility of sensibly defining a wide knowledge base of this kind for solicitors, comes the freedom to demand a narrower more sensible base.  It might also be observed that it concentrate on what is best assessed early, and which might be seen as core to the ability of young lawyers to develop are more translatable knowledge and skills  – this might include some substantive building blocks (such a proper understanding of precedent and the legal system, EU/UK/HR law, and professional ethics) but concentrate more on the critical reasoning, research and communication skills which enable a young lawyer to research and work with law with confidence and skill.  An SRA concerned about inconsistency[2] can similarly investigate those concerns with much smaller and less inhibiting influence on legal education.

A related problem will be how will an SRA defined curricula develop?  A top-down, knowledge dominated curriculum, drafted in committee is not obviously likely to drive up standards and adapt quickly, or with imagination, to an increasingly global, technology influenced legal services market.

Can the SQE assist with diversity?

Reduced cost is one way in which an SQE may improve diversity.  The potential costs of the SQE itself, and the secondary market for crammers which will quickly evolve, may be offset by potential reductions in the costs of an LPC type course or courses.  It is difficult to predict, therefore, whether reduced costs will really come to pass.

The potential benefits, from a diversity and quality perspective, of better benchmarking of university and candidate quality candidates through national testing is also interesting.  A key to the success of this is preventing the segmentation of the education and recruitment market referred to above.   For diversity to be significantly improved, firms would need to have an early (end of first year undergraduate) and robust indication of a candidates aptitude for law which was a stronger signal of their abilities than, in particular, A-level grades or ‘getting into a good university’.  Or, over time, firms would have to be able to see that identifiably good students from the elite wing were in fact not better than identifiably good students from the non-elite wing.  If a firm could see that, say, a non-Russell Group University had a good crop of high performers on the SQE, they might be more inclined to shift their recruitment from their traditional universities towards that non-Russell Group University if the SQE enables them to make this distinction with confidence.

It is an appealing idea but it is doubtful that this can be achieved.  This is partly because, on my reading of the evidence in your consultation paper, the difficulties of extending the SQE beyond a pass fail distinction mean that the test will be unlikely to distinguish – let alone distinguish robustly – between the merely good student and the good or excellent student that early recruiters of trainees are looking for.  It may also not do so early enough (and of course the earlier it is the more it divorces knowledge requirements further the point of qualification and from graduateness).

It is my judgment that early recruiters of trainee solicitors (which is the area where the biggest diversity problems are) are not going to be swayed towards recruiting from a more diverse set of candidates by the SQE.  Even were a league table of pass rates to develop, the more traditionally prestigious law schools would quickly adapt to the SQE and ensure sufficient uniformity of achievement to not meaningfully advance on diversity grounds.  With a relatively flat range of performance in league tables, firms will be very likely to maintain their traditional recruitment patterns.

The inconsistency problem

A second way in which the SRA needs to think about proportionality is in relation to the way it responds to concerns about different pass levels on the LPC, the potential differences in assessment regimes at the undergraduate level, and a lack of knowledge about the quality of the ‘signing off’ system for qualification (where there is no discussion of the historic arrangements for training contract monitoring).

In so far as the SRA has evidence of competence problems, the SRA is much more hazy about the educational deficits that cause them. In fact it has no robust evidence, at least that I have seen, of the ways in which educational and training process is linked to or can be improved sufficiently to improve competence.  This is a strange basis for deregulating the LPC and super-regulating LLBs.  Testing a wide variety of sometimes irrelevant legal knowledge three years or more before practice is not an obviously strong candidate for significantly improving the competence of lawyers.  Still less is it proportionate.  The acid test of the need for SRA reforms is what is likely to work?  The SRA does not know and so relies instead on solving another problem (consistency and comparability)  which they do not in fact have the evidence to establish and diagnose with precision.

Consistency and comparability is very important, but if legal education is to flourish and innovate there has to be room for variation in what is taught.  The SQE poses a major threat to this, and this is doubly true without the SRA being able to diagnose a) whether, let alone how, the extent and nature of a competence problem is related to any inconsistency in assessment standards; and b) how differently regulated systems of legal education and qualification will be likely to fix the undiagnosed problem.   Moreover the SRA puts enormous faith in the quality of, as yet, undeveloped tests to deliver baseline indicators of competence. In this way the SRA has landed on the idea that it can and should prescribe and regulate the substance but should deregulate the process of what is taught in the potentially folorn hope that this will reduce costs and without being sure yet that their competence tests will be anywhere near good enough.  For undergraduates this substance concentrates on a computer assessed test of knowledge, when skills and competencies might be more core and more appropriate for a graduate level qualification.

With evidence of the linkages between competence and training so absent, as the SRA’s consultants acknowledge, [3] I would have thought the place to start is with assessing competence at the point of qualification and work backwards, establishing what is going wrong and then tracing the causes back to targeted solutions.    Similarly, the SQE would be much more wisely brought in incrementally, allowing all the stakeholders, but particularly the SRA, to develop its understanding and improve its implementation.  It follows that there is a great deal of uncertainty and vulnerability in the proposals which would benefit from a significantly more measured approach.

The reserved services problem

As I understand it, one of the reasons for the rather wide set of statements of knowledge which solicitors would be required to have relates to reserved activities.  This seems to be another area where a more proportionate approach can be imagined and one that fits more with the increased flexibility of the CPD regime.  It seems to me much more reasonable for the SRA to specify knowledge requirements that a solicitor requires to be able to practice in a reserved area and requires that, prior to practising in that reserved area, the solicitor has to pass the relevant knowledge and – if relevant – skills tests. If a solicitor does not practice in a reserved area they do not need the knowledge and skills to do so. If they do practice, they need to pass the relevant assessments, perhaps with the ability to work under supervision until they reach that point as any paralegal would be able to do.  This would significantly reduce the extent of the SQE knowledge requirements, reduce costs and make (some modest) diversity gains more likely.  It would be more flexible, and permit more innovation. And it would ensure training in the reserved areas was relevant and proximate to practice.  It need not even be necessary to have a reserved services licensing scheme – it would be relatively easy the SRA, LeO and Insurers to pick up complaints where a solicitor is practising in an area where they need a set of reserved legal services knowledge and skills assessments.  Ex post regulation here is probably sufficient.

Further inflexibilities

There is a suggestion that skills assessments be mainly confined to reserved areas, and that contexts should include transactional and litigation contexts.  This is another unnecessary inflexibility.  Why focus on this distinction? What is the bewitching power of transaction vs litigation over the profession’s consciousness? Why not, for example, require that lawyers have to have worked for claimants and defendants – a more illuminating difference than transactional vs litigation work in my opinion – or businesses and individuals?[4] The transaction litigation divide seems to me to be focusing on surface detail rather than looking carefully at the underlying skills or competences that are really necessary to practice as a solicitor. If one follows the logic of the SRA position and asks, what are the uniquely transactional competences required in reserved activities, the importance of transactional skills starts to look rather suspiciously small.


I am an academic (and former solicitor) who has spent the entirety of my career working at the interface between practice and academia.  I believe that legal education should change to deepen its relationship with practice, but also that the profession needs to take more seriously the need for a vigorous, independent and research active academy able to engage with, evaluate and generate ideas on the future of legal practice.  The often stated tension between a liberal law degree and the education of neophyte practitioners is a false dichotomy but it is a false dichotomy sponsored by practice and the academy to protect traditional ways of working.

That said, the prescription of thirteen knowledge areas represents a wholly and foolishly disproportionate attempt to break down this dichotomy.  Not only are the knowledge requirements over-inclusive, but they are also under-inclusive.  For most solicitors, the main bodies of legal knowledge they would draw upon every day to be competent would be of a different nature and order.  As a scheme to deliver an assurance of competence it misses its key target – the knowledge that practitioners actually use on qualification to deliver on their client’s needs.

Quite aside from any question about the harmful impact of such a knowledge-dominated curriculum, assessed in the way the SRA envisages, on the quality of legal education there is a major question-mark over the adaptability and responsiveness of such a curriculum. The SRA has championed the need for regulators to step back from regulating unless absolutely necessary, and more pertinently, has also lauded the need for innovation and adaptability in the legal service market and yet is proposing a top-down, highly prescriptive framework which it will have to manage and update with a high degree of centralization and (one must predict) a low degree of fleetness of foot.  The potential for sclerosis is obvious and will in my view be enormously harmful to the long term health of the curriculum and the profession.

[1] Depending on how the knowledge areas are clustered together.

[2] Although it should be noted the SRA is not able to link competence problems to evidence of inconsistency in degree and LPC awards.

[3] The SRA’s economic impact assessment states, for example, “comprehensive evidence does not exist that links low quality advice to the qualification requirements.”

[4] I am not genuinely suggesting these things as requirements, merely that alternative distinctions might be more powerful in training and competence terms than the transaction/litigation one.

Posted in Uncategorized | 7 Comments

Calling on in-house lawers…

I have posted this on the Ethical Leadership for In-House Lawyers blog, so apologies if you have seen this already, but I wanted to ensure maximum exposure for an initiative we are very excited about…

Be part of the conversation – A roundtable discussion with Paul Gilbert of LBC Wise Counsel facilitating

In 2015 a major new project was launched to research and engage in-house lawyers on the role of the General Counsel in the context of Ethical Leadership.

The project is led by Professor Richard Moorhead, Professor Stephen Mayson both of UCL Centre for Ethics and Law (CELs), Paul Gilbert of LBC Wise Counsel and Steven Vaughan of the Centre for Professional Legal Education and Research (CEPLER).

The project has attracted the interest of the SRA and we believe it is the first time this important subject has been systematically researched with in-house lawyers at the heart of the conversation.

In 2015 the first phase of the project was to undertake an online survey of in-house counsel. The full results of the survey will be published this year along with the commentary of in-house lawyers in conversation with the project team.

This therefore is your invitation to attend a roundtable “town hall” conversation and to be part of the debate. The meetings will be held in March, April, May and June. Emerging themes from the survey will be shared, but crucially we want your views to be heard. Each conversation will be under the Chatham House rule so we can all speak freely without attribution. It is a unique study and your insights are vital for the research.

There is no fee to attend any roundtable meeting, we simply want you to come. It does not matter your sector, the size of your team or your seniority. Your views matter as an in-house lawyer today, so please be heard. Meetings are schedule for just two hours and hopefully at times that are not disruptive for the working day.

Dates and venues:

  1. 7 March 8am to 10am Sheffield at the offices of Irwin Mitchell
  2. 7 March 5pm to 7pm Birmingham at the offices of Irwin Mitchell
  3. 9 March 5pm to 7pm Manchester at the offices of Irwin Mitchell
  4. 10 March 8am to 10am London at the offices of Irwin Mitchell
  5. 10 March 5pm to 7pm London at the offices of Irwin Mitchell
  6. 20 April 5pm to 7pm London at the offices of Bevan Brittan
  7. 21 April 12.30 – 2.30 Leeds at the offices of Bevan Brittan
  8. 26 April 8am to 10am Bristol at the offices of Bevan Brittan
  9. 26 April 5pm to 7pm Bristol at the offices of Bevan Brittan
  10. 11 May 430pm to 630pm Birmingham at the offices of Bevan Brittan
  11. 2 June 6pm to 8pm London at the offices of Norton Rose Fulbright

In order to secure your place at the table, please reply to Tina Harris th@lbcwisecounsel.com or Paul Gilbert pg@lbcwisecounsel.com indicating which meeting you wish to attend. They will then write to confirm your place and full venue details.

You will not be added to a mailing list by registering your interest.

More on the project:

There is growing evidence, research and some anecdotal commentary to suggest that the role of GC is under increasing pressure and that the professional ethical boundaries are not as elegantly drawn as may be helpful for our increasingly sophisticated world of work. While it seems fair to assume that no one relishes more regulation, and more regulation of in-house lawyers/lawyers may be inappropriate anyway, it is interesting to note that there are no qualifications needed to be a General Counsel, no commonly accepted guiding principles for the role and no requirements on business to create an environment in which it is appropriate to employ in-house lawyers. In addition the types of GC role vary dramatically from strategic executive consigliere to transactional specialist. As a result the expectations of business of the GC role vary widely as well.

This initiative therefore is not a call for new or more regulation. It is however a call for in-house lawyers to come together (sometimes with other stakeholders) to debate the role of the General Counsel, its ethical framework and then to suggest the principles that could guide the role. We are certain that in-house lawyers know best what the tough issues are and how they manage them. This is therefore a facilitated forum for debate, a place to offer and share insight, to share resources and help shape the needs of the General Counsel as well as serving the interests of the profession and of business.

Please be involved. Please pass the invitation on to colleagues, friends and contacts who are in-house lawyers as well. Please be part of the debate.

Posted in Uncategorized | Leave a comment

What to make of the SRA’s Independence Survey?

The SRA’s polling data which – I would say impressionistically – has garnered unusually high levels of media interest outside of the legal press – is worth a quick look. I’m not sure how relevant it is to the current debate about extent of the SRA’s independence from the Law Society. As the questions are phrased and asked, the survey respondents may have got the impression that profession was self-regulated and independence was being considered.   The truth is much more nuanced; in fact, it’s not really like that at all. What the SRA and the Law Society are really fighting about is the difference between substantial independence and total independence from the profession’s representative body.  Oh, and money.  It’s a fair and important fight but I’m not sure the consumer poll gives us much insight into either question.  And I say that as an instinctive supporter of a more independent model.  Indeed, to prove the point, I look forward to the Law Society conducting a futile counter-poll which says something like, Do you agree that the experts should have a say in regulation of legal services or do you think that faceless* bureaucrats who will pander to government and not protect your interests should build their own empire.

Even so, there is something of interest in the data.  That is the extent to which the public appear to mistrust self-regulation.  Only 6% though that solicitors should be self-regulated. No matter how much one wonders what the public understood by self-regulation, or whether they understood that solicitors aren’t currently self-regulated, that is a very low figure indeed.  The second  thing of interest is the very high proportion of the public felt that solicitors needed to be regulated: 86%.  I wonder how much that figure will get quoted when the SRA are proposing further deregulation.  We shall see.


* They’re not really faceless. And I don’t really think the Law Society would conduct such a poll, although some regulators have been almost that dumb in the past.

Posted in Uncategorized | 2 Comments

Background statistics

The background of trainee solicitors in the larger City firms is under scrutiny again this week.  The Times (£) reports a piece of research done by Chambers suggesting on their sampling (which it is difficult to scrutinise from the story):

69 per cent of the magic circle firms’ trainees had gone to private schools, compared with 31 per cent from state schools.

The position in the City firms generally was more even at 48 per cent/52 per cent, while London firms outside the City and those outside London reported about two thirds of their trainees were from state schools.

Nine of the 18 firms taking part in the City Solicitors Horizons’ initiative to combat the “poshness bias” in the legal profession responded to the Chambers survey. At seven of those firms, between 52 per cent and 75 per cent of their trainees had been privately educated, with two reversing the trend – at Macfarlanes, 65 per cent had been to state schools and at Pinsent Masons 67 per cent.

If the figures are close to correct, even the ‘better’ profiles are alarmingly bad. The story’s also interesting because it hints that the US firms take a more egalitarian approach by recruiting from a wider range of Universities. Perhaps they have looked beyond the polish and found the spit?

Meanwhile Alex Novarese takes some time to remind the readership of Legal Business that law firms may have gone backwards on this:

You may have noticed [he tells them, that] a decent chunk of your veteran rainmakers aren’t that posh. There’s a reason for that.

Posted in Uncategorized | 1 Comment

Stripping down to improper advice?

Bloomberg published an interesting story reporting that Sullivan & Cromwell LLP and Slaughter and May’s are of interest to U.S. prosecutors, “asking whether [the] two law firms gave Standard Chartered Plc improper advice as they steered the bank through a sanctions-violations investigation.”  The prosecutors (who may be the source for the story) have not, though, “accused the law firms of wrongdoing.”

I have written previously about this role of SCB’s in-house lawyers in relation to what was known as wire-stripping, but this story appears to relate to the subsequent investigation into SCB which led to substantial fines for breaching US sanctions law. The bank claimed to have acknowledged past criminal conduct but has been/is being re-investigated to, “determine whether the bank had withheld evidence of Iran sanctions violations.”  BNP Parbibas and some of its outside lawyers have also found themselves in hot water in similar circumstances.

What we are told prosecutors want to know now is whether the law firms improperly advised the London-based lender on the submission of certain information during the investigation that led to the 2012 settlement.

Another of SCB’s professional/independent advisers (not its lawyers) have previously been criticised for “softening” its reports to regulators, “by request of the bank or its counsel.” (emphasis added, but it is not clear from the story whether this request emanated from either in-house or outside counsel).  This kind of management of independent advisers is coming under increasing scrutiny and beginning to draw professional advisers closer towards scandal.  The story ends with this statement:

Rarely are lawyers and law firms taken to task for their opinion letters or advice. Even when a corporate decision is found to be at fault, the company may be able to minimize the penalty with the advice of counsel as a shield.

Rarely, but sometimes. It remains to be seen whether this US investigation goes anywhere.

Posted in Uncategorized | Leave a comment

2016, Artificial Intelligence and the Year of Circumspection

A fascinating story in today’s FT by Richard Waters asking whether Watson can save IBM (it is pay-walled but well worth a read). IBM’s Watson is the much known and increasingly heavily marketed IBM artificial intelligence phenomena. It beat real humans in a complex general knowledge game (Jeopardy) and its apparent ability to help diagnose cancer has been much promoted, with this example being particularly famous:

A demonstration showed how quickly Watson is able to diagnose illnesses, and provided a real life case that took doctors and nurses six days to diagnose, and only ended with the correct diagnosis because a nurse had seen the disease before. Based on symptoms input, Watson was able to correctly diagnose in minutes. The demonstration took place at IBM Watson’s New York City, New York office on May 27, 2015.

The story reports on IBM’s struggles to apply Watson. The ‘we will go to the moon’ style statements of Watson’s potential have been tempered by a great deal more pragmatism.  They have broken down Watson into 40+ different products. These products are quite specific or even described as basic, although I’d describe them as more prosaic. Nevertheless IBM report these are selling quickly. They include things like sentiment and character analysis from things like tweets (God help us all, well  me anyway) and intelligent data analysis.

Watson’s famous victory in Jeopardy inflated expectations that great leaps could be made in terms of computers understanding complex, unpredictable information and answering complex questions. The promise has not yet been fulfilled.  The FT story suggests that the well trailed example of a Texas hospital trying to use Watson to help diagnose cancer has proved difficult, with the report suggesting this has not (yet) been successful – despite the demonstration above.  The hospital’s head of innovation says, “Turning a word game-playing computer into an expert on oncology overnight is as unlikely as it sounds” but retains her hope that it can get to something like the hoped for system.

The claim seems to be that Watson is being used to brand more pedestrian applications than the original conception of Watson as an  adaptive artificial intelligence. Another way of putting this is it is an early stage technology rather than an mature one: a (still rather impressive) ZX Spectrum to the MacBook of 2020.  Pragmatism sometimes yields results though. An Australian energy group is reported as using Watson’s natural language processing facilities to investigate its database of documents and mine intelligence from it: the example given is working out from 30 years of projects the best calculation is for pipeline pressure. According to IBM, the application still involve “high-end ” AI and quick processing of unstructured data .

A further problem that I was pleased to see recognised is that of trust. Where an artificial intelligence system makes human like predictions, it does not ordinarily offer human like explanations as to how it reaches its decisions. It is a problem likely to be important in law, where reasoning can be characterised as part of, not just preceding, the result.  IBM are reported to be trying to overcome this, so far without – according to Waters – significant success but at least they are trying.

For lawyers, the position may be tantalising. Some of Watson’s products – and the Australian example – promise real applications.  Yet law’s data may be peculiarly complex and nuanced.  Success in current AI seems to be most likely in areas of work with already high levels of predictability and structure.  That is the claim made in a very interesting paper by Remus and Levy which seeks to draw some boundaries around the near term possibilities of robot lawyers.  The paper is ably summarised here by Caroline Hill.  Remus and Levy sometimes under-estimate or miss current potential amongst existing providers of AI-like services, but they also make a valuable attempt to remind  us that there is a big gap between what can currently be done and what is claimed as possible. There needs to be a much stronger focus on the currently possible, what works and what does not work, why it does not work and why, and the normative implications of greater artificiality, automation and probabilisation of law. That debate has been masked by a phoney war between futurists and Luddites and is greatly hindered by an absence of real evidence.

A final lesson from the FT story is this: a key driver of the advance in artificial intelligence has been the availability of large amounts of data. Watson’s apparent ability to cope with unstructured data may encourage Law firms and legal departments with the ability and willingness to access repositories of such data to look for their equivalents of pipe pressures through the knowledge mining above.  What could be learned by intelligent search through case files, opinion letters, advice letters, deal bibles, contracts and the like?  This would require investment of time and money that the large accountancy firms may have more of a culture and appetite for than law firms and may lead to incremental rather than transformational change: more of Team GB/David Brailsford’s 1% increases in effectiveness than a Steve Job’s like seizing of the market, perhaps. Equally, the holy grail of better identifying what works requires tying lawyer input (advice given, documents drafted, litigation and trial strategies) to outcomes in terms of actual behaviour, value to the client and so on. One sometimes sees and hears of small advances in the making of such linkages but lawyers generally have relied on a folksy, experiential approach to linking the quality of what they do to the impact of their work on clients and the broader world. Better making that linkage is a key challenge for legal practitioners with pretensions for seizing high volume and high value impact.

So whilst the FT and the Remus and Levy pieces can be seen as pricking the bubble of law tech evangelism somewhat, I prefer to see in it a healthy maturity. More real work is being done to apply technology. More failures are coming faster. After the unknown comes the hype and after the hype comes the disappointments and progress of reality.  We can probably expect advances, but advances that are modest and worthwhile (though not as worthwhile as they are hyped to be), rather than transformational. And we will tend not to know much about them yet.  At least in the near term.

Posted in Uncategorized | Leave a comment