Having already blogged on RCTs in legal services evaluation recently, the Yale Law Journal content alerted me to the publication of a much discussed and controversial paper by Greiner and Pattanyak. Here randomisation was used to assess the offer of representation from a student clinic at Harvard Law School dealing with unemployment benefit cases (a little like unfair dismissal cases here). The Yale paper is here and in case that is paywalled an earlier version of it is here. It is a great paper both for understanding the benefits of RCTs and for starting to understand the controversies that arise in their application. It also pulls no punches. Most importantly, it is the first in a series of papers (see below) which explores the impact of representation programmes in a number of different contexts.
The study is controversial because the results are somewhat counter-intuitive and, for lawyers, counter-cultural. An offer of representation (which was then very frequently accompanied by actual representation, 90%ish) delays the outcome of the case significantly (causing the clients some detriment because they mostly won their cases anyway so tended to win later) but did not significantly improve the likelihood that that got their unemployment benefit. Those who did not get offered representation by the Clinic got similar outcomes but more quickly (and only 39% of those had gone on to find representation).
The authors of the study reject the idea that the results are explained by a lack of quality of representation on the basis that the programme is well supervised, and entry is very competitive. They do this a little too easily for me, postgraduate students at Harvard however well supervised will not have repeat player experience of the adjudicators they appear before (which Kritzer suggests is critical to good outcomes) but they make a number of points in favour of the quality of the service which make out a case. They also reject the idea that the results are explained because clients turned down get representation less. Their view here is highly plausible: as noted already 90% get representation through the clinic’s offer; whereas only 39% of those turned down do. They have a number of other suggestions as to why representation for this group of clients in this forum may not make any difference:
1. The clients seeking advice may, in fact be those clients most able to help themselves:
“This suggests to us either that the claimants who initiated contact with HLAB possessed personal characteristics making them more likely to win their cases or that the underlying facts in their cases were unusually strong. Either is possible; the former strikes us as particularly plausible given that the HLAB unemployment intake system depended on a telephone contact initiated by the claimant. In other words, perhaps HLAB and the other attorneys who assisted clients who had contacted HLAB were helping only those who did not need the help.” (p. 2173)
2. The adjudicator “had used and trained its staff effectively to create a system that was accessible to pro se [self-represented] litigants”.
3. Whilst the assistance given to represented clients was significant, the cases might have been relatively straightforward (they did not, for example, involve expert evidence) and so representation would not have had much effect.
There are alternative explanations and it is important to emphasise that this study does not prove that representation does not work. As the authors note the impact of representation may be highly contextual: it may depend on how clients are selected; how the courts they are to be represented in work; and so on. Indeed, they report on two other studies where beneficial outcomes for clients were seen in RCTs in the field of housing and juvenile cases. Greiner has also led other studies, with more apparently in the pipeline. We can get a sense of what sorts of factors influence the impact of representation from these two housing court studies.
Potential clients were defendants in housing eviction proceedings, and both programs were oversubscribed in the “District Court Study”. After receiving limited assistance, clients were randomly offered (or not offered) a traditional attorney-client relationship from a staff attorney. Here there was an effect that benefited clients:
“At least for the clientele involved in this District Court Study, a clientele recruited and chosen by the service provider’s proactive, timely, specific, and selective outreach and intake system, an offer of full representation mattered. Approximately two thirds of defendants in the treated group, versus about one-third of defendants in the control group, retained possession of their units at the end of litigation. Using a highly conservative proxy for financial consequences, treated group defendants received payments or rent waivers worth a net of 9.4 months of rent per case, versus 1.9 months of rent per case in the control group. Both results were statistically significant. Meanwhile, although treated cases did take longer to reach judgment, the offer of representation caused no increase in court burden as measured by other, more salient metrics.”
“…even though control group defendants received substantial (but limited) legal assistance, and even though the adjudicatory process included certain measures designed to promote access to justice, such as mediation and some judge-initiated questioning. Thus, the adjudicatory system did not provide full access to justice despite the best efforts of personnel within it.”
Yet in a further study the “Housing Court Study” clients were similarly receive randomised offers of a traditional attorney-client service having received some limited (“unbundled”) assistance in housing eviction cases. The alternative offer to the traditional service was that:
“all study-eligible potential clients could (and most did) receive assistance in filling out answers and discovery requests. In addition, occupants not offered a traditional attorney-client relationship from the provider’s staff attorneys.. …received a referral to that provider’s lawyer for the day program. The lawyer for the day program provided same-day-only representation in hallway settlement negotiations and mediation sessions but not in court appearances or in filing motions.”
Here the offer of traditional representation is broadly being compared with unbundling plus duty assistance of a sort. Representation did not have an apparent, “effect on the likelihood that the occupant would retain possession, on the financial consequences of the case, on judicial involvement in or attention to cases, or on any other litigation-related outcome of substantive import.” Nor was there evidence that for these clients in this type of case the lawyers kept cases out of courts (although the evidence here was more limited).”
The work is tremendously interesting and sometimes unsettling, but it is also very important. An interesting but constructive critique can be found here but to give you flavour from the abstract:
“The Greiner and Pattanayak findings challenge our intuition, experience and deeply-held professional belief that lawyer representation of indigent clients in civil matters is fundamental to the pursuit of justice. Our first reaction is that the study must have fatal conceptual or methodological flaws – the researchers studied the wrong thing in the wrong way. Even when we learn that the study is credible and well designed, we doubt that this kind of research is a worthwhile use of our time or money relative to serving needy clients. Finally, and perhaps most importantly, we worry that the published results will only serve as fodder for the decades-long political assault on legal services for the poor.
If replicated across venues, however, studies like Greiner and Pattanayak’s can tell us a great deal about individual representation, program design and systemic access to justice questions. In fact, we cannot make genuine progress in any of these areas – much less marshal the case for more robust legal aid investments and the right to counsel in some civil cases – without better evidence of when, where and for whom representation makes a difference. Fortunately, developments in law schools, the professions and a growing demand for evidence-driven policymaking provide support, infrastructure and incentive for such research. For these reasons, we urge legal services lawyers and clinical law professors to collaborate in an expansive, empirical research agenda.”
The temptation will be either (for the lawyers) to reject the findings out of hand or (the funders) to say this shows that we do not need representation. Neither position is the correct position. Done properly, and sufficiently often, in enough contexts, such work can assist in the rational design of hard pressed legal assistance services. The question is will anyone take the trouble?