7 questions to ask your AI contract review vendor

‍Introduction and key summary

Automated Contract Review (ACR) (sometimes call AI contract review) software is proliferating and becoming increasingly accurate and powerful.  The market is becoming more mature, but usage lags behind use of contract drafting tools and contract and matter management tools.

ACR technologies come in many shapes and sizes and use different types of artificial intelligence (normally some combination of natural language processing, machine learning and deep learning algorithms and often a combination of supervised and unsupervised learning).  

Adoption rates are increasing because of higher accuracy rates, better user interface and a more mature approach to selling and adoption by both buyers and sellers.  There is less overhyping and instead more focus on simple but powerful use cases.

The benefits are clear.  According to Syke, a Legal Technology Consultancy,  studies have shown direct comparisons and benchmarking where AI has achieved 94% accuracy (in seconds) where lawyers achieved 85% (in 90 minutes).

As a customer, we believe that there are 7 key questions to ask your AI vendor.

1. Does it really handle ‘third party paper’ well - if so, can I bring my own blind contract to the demo? 
2. How does it work with my existing systems and processes? 
3. Does it just extract key contract terms or perform a risk analysis? 
4. Does it properly understand the roles of the parties in a contract?
5. How does it handle definitions, exceptions and overlapping provisions? 
6. Can I set-up and train the system?  
7. How is it priced?  

Coming out of the ‘trough of disillusionment’?

Aside from a growing cohort of legal operations professionals, who understand the systems and have tested them, users have often lacked the knowledge and experience to understand the capabilities and limitations of the tools and how to compare them.

Vendors and tech evangelists have likewise often oversold the capabilities of the tools and undersold the practical constraints such as how much data is required to train an AI model.  

The sector has arguably gone through the trough of disillusionment following inflated expectations and is entering the slope of enlightenment.

This paper explains what is behind the bonnet of the tools, how they work and offers advice on how to cut through the AI hype and choose a tool that may work for you.

What is ACR?

ACR often contains a number of key technological components:

• An AI engine - often combining Natural Language Processing and Machine Learning - built to understand different formats and texts in documents.  This can use supervised or unsupervised machine learning techniques or a combination of both.  This Kira blog post has a good explanation of the differences.  ACRs may themselves use widely available foundation language AI models such as BERT or GPT which their own proprietary AI models then refine and apply to the contract world.    
• Logic and rule based algorithms - (sometimes called an ‘expert system’) undertaking further analysis so that data in the contract can be properly summarised, grouped and compared to the user’s own policies.
• Traditional software which provide workflow features such as collaboration tools, clause libraries, editing features and visualisation of data.    

Tiro from Lexical Labs combines all of the above, notably rules and logic based algorithms which perform a risk analysis as well as extracting key contract metadata.   This means it is particularly useful for reviewing inflight contracts in the negotiation phase.  

Key use cases of ACR

‍

• Contract intake and triage - New draft contracts are ingested (often automatically from a contract intake system or ‘legal front door’) and risk reviewed.  This initial review forms part of the intake triage process so that contracts are assigned to the right reviewer, for example assigning simple contracts to business teams or junior paralegals and more complex and riskier contracts to senior lawyers and contract managers.  With some skilful set-up, it is possible to automate the triage review using the ACR software.  

• Contract negotiation - Once assigned, ACR software can help the business or legal reviewer finalise the contract.  As well as its automated review and risk flagging features, ACR allows the reviewer to respond on the contract by editing or redlining or creating a list of requested changes.  Clause libraries, fallback wording and other playbook features assist this process. 

• Contract approval - ACR software can act as a final pair of eyes before contract approval.  The contract can be scored against the enterprise’s policies and special exemptions obtained for contract risks which go beyond the normal tolerance.  The ACR can also show the risk improvement throughout the negotiation process and the time in negotiations.  This type of data is invaluable in assessing the effectiveness and efficiency of legal and contracting teams as well as the appropriateness of an enterprise’s policies.

• Contract execution - once signed by e-signature, ACR software can automatically extract and summarise key terms and risks.  This information can be pushed via an API into a contract management system or displayed on dashboards or in reports.  A summary of the key risks and responsibilities can be invaluable for project managers managing complex contracts, as in the construction industry.

• Contract management after signing -  ACR allows key contract terms to be searched and contract milestones (such as renewal or reporting dates) to be accessed.  Advanced ACR systems allow for a risk or strategic review of an enterprise’s executed contract or the ability to ask questions of an executed contract using natural language text.  For example, which of my contracts have transfers of IP.  ACR systems can also do a review of a portfolio of executed contracts as part of due diligence process during an M&A transaction or e-discover process in a litigation.

Key questions to ask your AI contract review vendor

‍

The obvious initial question is to ask for a simple explanation of how the ACR works, what AI techniques it uses and what types of contracts it is trained on.  Are these relevant to my business?  Don’t fall for jargon or overhyping.  If you don’t understand the answer, ask someone else or decline teh demo.

After that we see 7 key questions to ask your vendor.  

1.  Does it really handle ‘third party paper’ well - if so, can I bring my own blind contract to the demo? If you deal with a lot of inbound contracts on third party paper, it's vital that the ACR can handle this.  Some ACR systems can’t because they are trained to do a deviation analysis from a defined starting point.  
   ‍Third party paper is inherently messy.  Every contract has different language, structures, clauses, definitions and schedules.  Some have tables which an ACR may struggle to read.  Ask the vendor to bring one of your own contracts to the demo (anonymise if necessary) so you can do a ‘blind tasting’.   You’ll get a good feel for the system’s strong points and limitations.  Also, the demo will be much more interesting!
2. ‍How does it work with my existing systems and processes? Most ACR systems focus on contract review and don’t provide an end-to-end contracting solution.  So it must work seamlessly with your existing systems and processes.  You will want to minimise process changes when introducing ACR technology.  So ask can it automatically launch from your contract intake system, ‘legal front door’ or other contracting systems?  Can I use the ACR in my preferred document viewer and text editor - such as Microsoft Word via a plugin or must I view in the vendor’s own web portal?  Can it push contracts and data into a contract management system after signing?  
3. ‍Does it just extract key contract terms or perform a risk analysis?  This is a key distinction between different ACR systems.  Some will just extract key data.  Others are providing a more advanced risk analysis.  What is more important to you? 
4. ‍Does it properly understand the roles of the parties in a contract?  An ACR can only do a credible risk analysis if it understands who is the (i) supplier / service provider and the customer and (ii) what are the responsibilities of each party.  So does the cap on liability apply to one party or both?  Who retains or receives the IP rights.
   ‍An ACR system might be trained to understand references to ‘Supplier’ and ‘Customer’ in a contract.  But what if the contract refers to each party by their name or ‘we’ and ‘you’.  In those circumstances, would the system understand who is the customer and the supplier?  
5. ‍How does it handle definitions, exceptions and overlapping provisions?  Contracts are intricately sewn together using definitions, clauses, sentences and schedules.  Some clauses override other ones.  So a cap on liability may not apply in certain circumstances such as data breaches.  Does the system read each provision separately or the whole contract together?  Ask and test this.
6. ‍Can I set-up and train the system, or am I completely reliant on the vendor?  Different customers will have different preferences on this.  How much flexibility and responsibility do you wish to take?
7. ‍How is it priced?  By seat or by volume of contracts (either a per upload price or a subscription price based on annual volumes)?  Or some combination?  Can the vendor give you a per seat price or a per contract price?  Ask for pricing in the first discussion.

‍

Ps. How do you use or intend to use GPT or generative AI tech?  This will be the subject of a separate paper.  

‍