by Peter Frankl.
Reminiscent of IBM’s Big Blue beating chess champion Gary Kasparov, a computer has scored a higher degree of accuracy in predicting case outcomes than a group of London-based commercial lawyers. From the results announcement by CaseCrunch:
CaseCrunch is proud to announce the results of the lawyer challenge. CaseCruncher Alpha scored an accuracy of 86.6%. The lawyers scored an accuracy of 62.3%.
Over 100 commercial London lawyers signed up for the competition and made over 750 predictions over the course of a week in an unsupervised environment.
The problems were real complaints about PPI mis-selling decided and published by the Financial Ombudsman Service under the FOIA.
The main reason for the large winning margin seems to be that the network had a better grasp of the importance of non-legal factors than lawyers. We will publish a research paper explaining the approach we took and the broader significance for legal theory soon.
Evaluating these results is tricky. These results do not mean that machines are generally better at predicting outcomes than human lawyers. These results show that if the question is defined precisely (such as – was this complaint about PPI mis-selling upheld or rejected by the FOS?), machines are able to compete with and sometimes outperform human lawyers.
The CaseCrunch team seems to be downplaying the significance of the results. They may be right about that but for different reasons than suggested above.
At this point we hardly know anything about the lawyers involved in the test such as their degree of expertise in this field and how much time they spent on the cases.
Even in the worst case scenario for lawyers, if computers are better by a margin of almost 25%, there is still no reason to panic. My $5 calculator beats me at mental arithmetic almost 100% of the time. My smartphone knows a lot more about the world than I do when I press the right buttons and enter the correct search terms.
Computers are tools. They can empower people. Tools such as CaseCrunch can empower lawyers and consumers alike. It can start the problem-solving, such as for a legal matter, at a higher level.
What my calculator and smartphone lack, as much as I am attached to them, are the human qualities of empathy, ethics, trust and compassion. A relationship with a computer might be rated at 5% compared to a human advisor at 95%. Even if this is the case, trusted advisors inevitably will be using tools such as CaseCrunch to an ever-increasing degree.
© 2017 Legal Practice Intelligence