7 questions to ask your AI contract review vendor

Unlocking the Potential: Discover the 7 Essential Questions to Ask Your AI Contract Review Vendor and Gain Insight into the Future of Contract Management.

June 26, 2023

Introduction and key summary

Automated Contract Review (ACR) (sometimes call AI contract review) software is proliferating and becoming increasingly accurate and powerful.  The market is becoming more mature, but usage lags behind use of contract drafting tools and contract and matter management tools.

ACR technologies come in many shapes and sizes and use different types of artificial intelligence (normally some combination of natural language processing, machine learning and deep learning algorithms and often a combination of supervised and unsupervised learning).  

Adoption rates are increasing because of higher accuracy rates, better user interface and a more mature approach to selling and adoption by both buyers and sellers.  There is less overhyping and instead more focus on simple but powerful use cases.

The benefits are clear.  According to Syke, a Legal Technology Consultancy,  studies have shown direct comparisons and benchmarking where AI has achieved 94% accuracy (in seconds) where lawyers achieved 85% (in 90 minutes).

As a customer, we believe that there are 7 key questions to ask your AI vendor.

  1. Does it really handle ‘third party paper’ well - if so, can I bring my own blind contract to the demo? 
  2. How does it work with my existing systems and processes? 
  3. Does it just extract key contract terms or perform a risk analysis? 
  4. Does it properly understand the roles of the parties in a contract?
  5. How does it handle definitions, exceptions and overlapping provisions? 
  6. Can I set-up and train the system?  
  7. How is it priced?  

Coming out of the ‘trough of disillusionment’?

Aside from a growing cohort of legal operations professionals, who understand the systems and have tested them, users have often lacked the knowledge and experience to understand the capabilities and limitations of the tools and how to compare them.

Vendors and tech evangelists have likewise often oversold the capabilities of the tools and undersold the practical constraints such as how much data is required to train an AI model.  

The sector has arguably gone through the trough of disillusionment following inflated expectations and is entering the slope of enlightenment.

This paper explains what is behind the bonnet of the tools, how they work and offers advice on how to cut through the AI hype and choose a tool that may work for you.

What is ACR?

ACR often contains a number of key technological components:

  • An AI engine - often combining Natural Language Processing and Machine Learning - built to understand different formats and texts in documents.  This can use supervised or unsupervised machine learning techniques or a combination of both.  This Kira blog post has a good explanation of the differences.  ACRs may themselves use widely available foundation language AI models such as BERT or GPT which their own proprietary AI models then refine and apply to the contract world.    
  • Logic and rule based algorithms - (sometimes called an ‘expert system’) undertaking further analysis so that data in the contract can be properly summarised, grouped and compared to the user’s own policies.
  • Traditional software which provide workflow features such as collaboration tools, clause libraries, editing features and visualisation of data.    

Tiro from Lexical Labs combines all of the above, notably rules and logic based algorithms which perform a risk analysis as well as extracting key contract metadata.   This means it is particularly useful for reviewing inflight contracts in the negotiation phase.  

Key use cases of ACR

  • Contract intake and triage - New draft contracts are ingested (often automatically from a contract intake system or ‘legal front door’) and risk reviewed.  This initial review forms part of the intake triage process so that contracts are assigned to the right reviewer, for example assigning simple contracts to business teams or junior paralegals and more complex and riskier contracts to senior lawyers and contract managers.  With some skilful set-up, it is possible to automate the triage review using the ACR software.  
  • Contract negotiation - Once assigned, ACR software can help the business or legal reviewer finalise the contract.  As well as its automated review and risk flagging features, ACR allows the reviewer to respond on the contract by editing or redlining or creating a list of requested changes.  Clause libraries, fallback wording and other playbook features assist this process. 
  • Contract approval - ACR software can act as a final pair of eyes before contract approval.  The contract can be scored against the enterprise’s policies and special exemptions obtained for contract risks which go beyond the normal tolerance.  The ACR can also show the risk improvement throughout the negotiation process and the time in negotiations.  This type of data is invaluable in assessing the effectiveness and efficiency of legal and contracting teams as well as the appropriateness of an enterprise’s policies.
  • Contract execution - once signed by e-signature, ACR software can automatically extract and summarise key terms and risks.  This information can be pushed via an API into a contract management system or displayed on dashboards or in reports.  A summary of the key risks and responsibilities can be invaluable for project managers managing complex contracts, as in the construction industry.
  • Contract management after signing -  ACR allows key contract terms to be searched and contract milestones (such as renewal or reporting dates) to be accessed.  Advanced ACR systems allow for a risk or strategic review of an enterprise’s executed contract or the ability to ask questions of an executed contract using natural language text.  For example, which of my contracts have transfers of IP.  ACR systems can also do a review of a portfolio of executed contracts as part of due diligence process during an M&A transaction or e-discover process in a litigation.

Key questions to ask your AI contract review vendor

The obvious initial question is to ask for a simple explanation of how the ACR works, what AI techniques it uses and what types of contracts it is trained on.  Are these relevant to my business?  Don’t fall for jargon or overhyping.  If you don’t understand the answer, ask someone else or decline teh demo.

After that we see 7 key questions to ask your vendor.  

  1.  Does it really handle ‘third party paper’ well - if so, can I bring my own blind contract to the demo? If you deal with a lot of inbound contracts on third party paper, it's vital that the ACR can handle this.  Some ACR systems can’t because they are trained to do a deviation analysis from a defined starting point.  
    Third party paper is inherently messy.  Every contract has different language, structures, clauses, definitions and schedules.  Some have tables which an ACR may struggle to read.  Ask the vendor to bring one of your own contracts to the demo (anonymise if necessary) so you can do a ‘blind tasting’.   You’ll get a good feel for the system’s strong points and limitations.  Also, the demo will be much more interesting!
  2. How does it work with my existing systems and processes? Most ACR systems focus on contract review and don’t provide an end-to-end contracting solution.  So it must work seamlessly with your existing systems and processes.  You will want to minimise process changes when introducing ACR technology.  So ask can it automatically launch from your contract intake system, ‘legal front door’ or other contracting systems?  Can I use the ACR in my preferred document viewer and text editor - such as Microsoft Word via a plugin or must I view in the vendor’s own web portal?  Can it push contracts and data into a contract management system after signing?  
  3. Does it just extract key contract terms or perform a risk analysis?  This is a key distinction between different ACR systems.  Some will just extract key data.  Others are providing a more advanced risk analysis.  What is more important to you? 
  4. Does it properly understand the roles of the parties in a contract?  An ACR can only do a credible risk analysis if it understands who is the (i) supplier / service provider and the customer and (ii) what are the responsibilities of each party.  So does the cap on liability apply to one party or both?  Who retains or receives the IP rights.
    An ACR system might be trained to understand references to ‘Supplier’ and ‘Customer’ in a contract.  But what if the contract refers to each party by their name or ‘we’ and ‘you’.  In those circumstances, would the system understand who is the customer and the supplier?  
  5. How does it handle definitions, exceptions and overlapping provisions?  Contracts are intricately sewn together using definitions, clauses, sentences and schedules.  Some clauses override other ones.  So a cap on liability may not apply in certain circumstances such as data breaches.  Does the system read each provision separately or the whole contract together?  Ask and test this.
  6. Can I set-up and train the system, or am I completely reliant on the vendor?  Different customers will have different preferences on this.  How much flexibility and responsibility do you wish to take?
  7. How is it priced?  By seat or by volume of contracts (either a per upload price or a subscription price based on annual volumes)?  Or some combination?  Can the vendor give you a per seat price or a per contract price?  Ask for pricing in the first discussion.

Ps. How do you use or intend to use GPT or generative AI tech?  This will be the subject of a separate paper.  

No items found.
Looking for some more value?

Other Posts

Browse some related whitepapers

Initial views on Recent Developments at OpenAI

OpenAi’s recent Developer Day and the launch of GPT-4 Turbo, the new Assistants API and the ability to create custom versions of ChatGPT for specific purposes (GPTs) presents the contracting industry with both a massive opportunity and a potential bombshell.
Read more

Automated Contract intake and triage

Automating your contract intake and triage might be the most valuable legal operations change you make in the next year. Here's why:
Read more

Tendering - Pass the Parcel - Who's in charge of the contract?

Revamp your tendering process, mitigate risks, and secure winning deals by harnessing AI contract review technology.
Read more

Why procurement contracting often seems harder than sales - but shouldn’t be!

When it comes to negotiating a contract for buying goods or services, the seller has it easy!
Read more

Why contract playbooks are essential (and easier than ever to set-up)

Unleash the Power of Contract Playbooks: Discover the essential steps to create and optimise playbooks for streamlined contract negotiation, improved outcomes, and significant time and cost savings in less than 3 hours.
Read more

Structured data in contracts and open AI algorithms

Explore the future of contracts with structured data and open AI algorithms, driving efficiency, risk management, and innovation in the legal industry.
Read more

Reimagining Law - The automated M&A deal?

Redesigning M&A deals with cutting-edge technology and streamlined processes, empowering companies to reshape the future of legal transactions.
Read more

How Fast Growing Digital Businesses can Harness the Power of AI Contract Review

Unlock the Power of AI Contract Review for Fast-Growing Digital Businesses and Streamline Your Contracting Process with Efficiency and Precision
Read more

Lexical Labs automatically risk scores your contracts

Tiro, from Lexical Labs, is the most advanced AI based automated contract review system available. Tiro reviews contracts - using both deep learning and logic based algorithms - just like a lawyer or specialist contract reviewer would.
Read more

Contract triage and pre-screening using Automated Contract Review

Unleash the power of Automated Contract Review (ACR) technologies and revolutionize your contract triage process, saving time, reducing costs, and empowering your business teams for faster and more efficient contract management.
Read more

A Contract Management Blueprint for Agile Businesses

How do you keep up with the contract workload that comes with a rapidly scaling agile business? This guide explores how to develop a best-in-class contract management function.
Read more

Automated review of Customer Contracts – Stantec Case Study

Experience faster contract reviews, improved risk outcomes, and enhanced efficiency with Tiro, the AI-powered automated contract review software - Stantec's case study reveals groundbreaking results.
Read more

7 Trends Transforming Contracting and Legal Professionals in the Engineering, Construction and Advanced Manufacturing Sectors

Discover the remarkable transformation taking place in engineering, construction, and advanced manufacturing sectors as contracting and legal professionals embrace innovation and adapt to the demands of the future.
Read more

An AI Engineer’s Personal Journey Into Law

Discover the striking parallels between the worlds of law and software engineering as an AI engineer embarks on a personal journey, unraveling the hidden connections and potential for automation in contract review and negotiation.
Read more

Are you feeling the pain of Third-Party Paper?

Reviewing draft contracts from the other party in the negotiation (so called ‘third-party paper) is like playing away in sport – home advantage counts.
Read more

Sign up for our newsletter

No spam, we promise! Just the occasional email with product updates and value from our whitepapers.

By clicking Sign Up you're confirming that you agree with our Terms of Service.
Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.