Bar Barometers, Aptitude Tests and THOSE numbers

With the Bar Standards Board releasing its (second) annual statistical report or Barometer as they more elegantly describe it, and the legal journos picking up on the increase in those applying to do the BPTC, it occurred to me now would be a good time to pick up a few points on aptitude test I’ve been meaning to address (been a bit busy, no one seems to have noticed I’ve been away (sniff)).

As is now well known, the Bar’s response to the numbers issue (and other issues) has been to seek to put a barrier between the BPTC course and applicants: the aptitude test. The full report of the evaluation is here. It is well worth a read and a response, if you have time between now and the end of February. For what they are worth here are my thoughts (I’ve blogged previously on this as the story has unfolded):

  1. The report does the best job it can of advocating an aptitude test (almost, see last para.). I have, as most people have, a good deal of sympathy with the concern that too many students do the BPTC (although one has to ask, how do we decide how many is too many and who should decide?). They also run an interesting argument that good students are dragged down by being taught alongside the less able. Mixed ability classes, it is argued, are a problem for Bar Students. See more below.
  2. The report rather overplays its hand in my opinion. One reason is that most of the predictive power of aptitude tests (the ability to predict who will pass the BPTC) is already available in A-level and degree results. There is a rather modest increase in predictive power by using an aptitude test (that is the added value of an aptitude test is modest and weakened further by the need to be generous in how the test is applied and further weakened, probably, by proposals to allow multiple (infinite?) resits). How many candidates that modest increase of predictive power would improve things for is not clear (they talk of percentage variance which is not the same thing as actual percentages of students and the actual modelling of numbers of students affected by a test does not look at the issue in this way. This is a serious flaw in my view). In other words, the vast majority of the work in selecting candidates who can pass the BPTC is already done, or capable of being done, by A-level and degree scores. Aptitude tests add not very much to the equation.
  3. What is also very clear is that some very good candidates would, in theory, be excluded by the aptitude test (unless they resat until they passed) and some very poor students would get through. It was ever thus. Consider debating whether to exclude students with a 2:2 and you have the same problem. All entry tests are rough justice. I can’t say I am too worried about that, but it is an issue worth considering in the light of my “2:2” students example below.
  4. The report and consultation appears to me to consider only one option for dealing with the problems that it identifies. It is really aptitude tests or nothing. No alternative or more proportionate responses are considered. This seems to be because a Bar Committee which met several years ago concluded that an aptitude test was a good idea so other options are now off the table. It leads me to wonder whether the BSB are overly fettering their discretion in considering the issue in these terms.
  5. Relatedly, and very importantly, there is a failure to consider whether there are more proportionate ways of dealing with the problems identified in para 1 above. Is there an information based approach that would work better (for instance the BSB could monitor and publish BPTC and pupillage success rates by University and degree classification)? As for the problem of mixed ability classes, would streaming of small groups by ability address that problem?
  6. The costs of an aptitude test are seriously underestimated. The BSB only seems to quantify the bare cost of the test. The cost of coaching is not really considered.

These are the problems with the consultation paper. There is a more fundamental problem which is why stop people doing the course if they want to do it and are fully informed? One reason, which bubbles up in some conversations, is the desire to trim the profits of law schools and the salaries of certain Chief Executives in particular. Not the most solid of justifications, but understandable. The second is that students are wasting their time or need saving from themselves. The profession’s interests are protected by failing students who are not good enough, not by stopping students who are likely to fail from getting in (some of whom it would transpire are in fact very good). The problem with the saving students from themselves argument was brought home to me recently by my consultations with a number of students who I did not think would succeed in getting a pupillage. In the light of recent debates, I was highly aware of the need advise these students very firmly of the risks and my opinion on their prospects for success. I laid it on thick. They all did pretty much the same thing: smiled and said they had always wanted to be a barrister and were going to try anyway (each time I see this, I think of this post). Given that aptitude tests, or any entry test, is an imperfect indicator of the ability of students to make a good lawyer one might form the judgment that, ultimately, leaving it up to the student might be the most appropriate response.

There are some remaining arguments against that. The only solid one I can think of is this. This large group of ‘unlikelies’ willing to take on the BPTC and risk not getting a pupillage is going to be skewed towards a particular group of people. Those who can somehow afford it. I did not see that the evidence from the aptitude test assisted with this argument, either though. In all likelihood, the group is already pretty much skewed that way. The Sutton Trust after all tried aptitude tests as a means of improving access into Universities. They concluded, and it is worth emphasising they did so against all their inclinations, that aptitude tests failed to do so. It’s a shame, but there it is.

3 thoughts on “Bar Barometers, Aptitude Tests and THOSE numbers

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s