Lantyer Dossier
How to evaluate a legal AI tool before using, hiring or recommending it
A practical framework for law firms, legal departments, public institutions and researchers evaluating legal AI with attention to LGPD, confidentiality, hallucination risk, source validation and governance.
Executive summary
The test is not whether the tool looks intelligent. The test is whether the institution can govern it.
Legal AI adoption fails when procurement is guided only by demos, speed or generic productivity promises. A responsible decision must examine data flows, legal accuracy, confidentiality, human review, vendor obligations and institutional readiness.
Legal quality
A tool must be tested against local legal questions, source validation and professional review.
Data governance
The team must know what enters the system, where it goes, how long it stays and whether it trains models.
Operational control
Adoption requires policy, accountability, logging, incident reporting and exit planning.
Central thesis
Legal AI is a governance decision before it is a software decision.
For international readers, the Brazilian context is important because LGPD, professional secrecy, civil-law reasoning and public-sector accountability make adoption more than a technology purchase. The same tool may be low risk in a public training example and high risk in a confidential litigation file.
A legal team should not ask only whether a system can summarize documents, draft clauses or answer questions. It should ask whether the institution can explain, control and audit the use of that system. If the answer is no, the tool may create legal risk faster than it creates efficiency.
This dossier turns evaluation into a practical sequence. First, classify the data and legal work involved. Second, test the tool with local and difficult questions. Third, review vendor obligations. Fourth, define internal use rules. Fifth, monitor errors after adoption.
Common errors
Six mistakes that weaken legal AI adoption
Buying before mapping data
A legal AI tool is not only a productivity feature. It creates a data flow. Before adoption, the team needs to know which documents, client information, personal data, trade secrets and litigation materials will move through the system.
Treating hallucination as a minor defect
Invented citations, false summaries and distorted legal arguments are not small UX problems in legal work. They can contaminate petitions, opinions, contracts, internal reports and strategic decisions.
Ignoring jurisdictional fit
A tool trained or designed around another legal system may be impressive in English demos and still weak for Brazilian statutes, courts, procedural vocabulary and civil-law reasoning.
Confusing confidentiality with a marketing promise
Security language in a landing page is not enough. Legal teams need contractual clauses, retention rules, access controls, audit rights and deletion procedures.
Skipping a local test set
Procurement should include a test with local legal questions, sensitive edge cases and representative documents. Generic benchmark claims do not answer how the tool performs for your actual use cases.
Adopting without internal policy
Even a good tool can become risky if professionals use it without rules about prompts, uploads, review, source validation, client confidentiality and approval workflows.
Concepts
Glossary for evaluating legal AI
Legal AI tool
A system that supports legal research, drafting, review, summarization, document management, contracts, compliance or institutional legal operations.
Generative AI
Models that generate text, images, code or other outputs from prompts and context. In legal work, their persuasive fluency can hide factual or legal errors.
Hallucination
A generated assertion that looks plausible but is false, unsupported or distorted. In law, this includes invented cases, wrong statutes and fabricated citations.
LGPD
Brazil's General Data Protection Law. It structures the processing of personal data and matters whenever legal tools handle client, employee or citizen data.
Professional secrecy
A core duty of lawyers and legal institutions. It cannot be reduced to general privacy language because legal files often contain strategic and sensitive information.
Vendor lock-in
Dependency created when data, workflows, templates or knowledge become difficult to export or replace without operational loss.
Human review
A mandatory legal-quality layer. AI outputs should be checked by a responsible professional before they affect clients, courts, contracts or public decisions.
Source validation
The process of checking whether cases, statutes, authorities and quotations exist, are current and support the legal proposition stated.
Lantyer Matrix
Thirty criteria before using or hiring legal AI
The matrix is intentionally broader than privacy. A serious decision also tests source quality, workflow, human responsibility, contract allocation, vendor dependency and institutional maturity.
Data, confidentiality and governance
Data categories
Identify whether the tool receives public information, internal know-how, client documents, personal data, sensitive data or privileged material.
Purpose limitation
Define exactly what the tool may be used for and what uses remain prohibited.
Retention rules
Check whether prompts, documents, outputs and logs are stored, for how long and under whose control.
Training use
Confirm whether uploaded data can be used to train, fine-tune or improve vendor models.
Access controls
Require role-based access, audit trails and clear rules for vendor personnel access.
Deletion and export
Verify whether the team can delete data and export work product without dependency.
Legal quality and source reliability
Citation accuracy
Test whether legal citations exist and whether they support the proposition generated.
Jurisdictional coverage
Measure performance on Brazilian statutes, courts, agencies and legal vocabulary when Brazil is the target context.
Update frequency
Understand how often the legal database or retrieval layer is updated.
Explainability
Prefer systems that show sources, reasoning path and confidence limits instead of unsupported conclusions.
Adversarial testing
Ask difficult local questions and compare the output with trusted legal sources.
Human approval
Require a workflow that keeps professional review before final use.
Security, procurement and operational fit
Information security
Review encryption, authentication, logging, incident response and administrative controls.
Contractual allocation of risk
Check liability, indemnity, service levels, audit rights and breach notification.
Integration safety
Assess integrations with document repositories, email, case systems and contract platforms.
Operational continuity
Plan what happens if the service is unavailable, discontinued or materially changed.
Cost model
Look beyond subscription price to usage limits, scaling costs, training and change management.
Portability
Avoid tools that trap legal knowledge or templates in a closed environment without export.
Ethics, people and institutional readiness
Professional responsibility
Define who is accountable for using, checking and approving AI-assisted work.
Bias and discrimination
Evaluate whether outputs can reinforce unfair treatment in employment, consumer, criminal or public-sector contexts.
Transparency to stakeholders
Decide when clients, courts, employees or citizens should know that AI assisted a process.
Training plan
Train users on safe prompts, prohibited uploads, review routines and escalation paths.
Internal policy
Publish simple rules before broad deployment, then revise them after tests.
Governance owner
Assign a person or committee responsible for approvals, exceptions, incidents and periodic review.
Evidence, audit and continuous improvement
Test records
Document test prompts, outputs, errors and corrections before procurement decisions.
Auditability
Keep enough logs to understand what was generated, who reviewed it and what was finally used.
Incident reporting
Create a channel for hallucinations, data exposure, bias and misuse.
Review cadence
Reassess the tool after major model, contract, feature or legal changes.
Exit plan
Define how the institution will stop using the tool without losing records or workflows.
Evidence preservation
For litigation or public-sector use, preserve sources and review trails when AI materially supports an output.
Risk semaphore
When use is low, controlled or restricted
Low-risk experimentation
Use with public materials, synthetic examples, generic training tasks or internal brainstorming that does not include client secrets, personal data or final legal conclusions.
Controlled professional use
Use with internal documents or legal drafts only after policy approval, data mapping, contractual safeguards, source validation and human review.
Restricted or prohibited use
Avoid uploading confidential case files, sensitive personal data, strategic client material, sealed records or documents whose disclosure would create legal, ethical or institutional harm.
Profile checklists
Different roles need different questions
Law firm partner
- Approve use cases before tools enter practice
- Separate marketing demos from legal-quality evidence
- Demand confidentiality and source-validation rules
- Assign responsibility for final outputs
Legal department
- Map business data and contract repositories
- Review vendor terms with privacy and security teams
- Pilot with low-risk documents first
- Define escalation for errors and incidents
DPO or privacy lead
- Identify lawful bases and data categories
- Review retention, transfer and training clauses
- Test anonymization and access controls
- Document LGPD governance decisions
Public legal office
- Check procurement rules and transparency duties
- Avoid automated decisions without accountability
- Preserve audit trails and public-interest justifications
- Evaluate sovereignty and vendor dependency
Professor or researcher
- Clarify authorship and research-integrity rules
- Check citations manually
- Avoid uploading unpublished or restricted material
- Teach students the limits of AI-generated legal text
Supplier questions
Questions to ask before procurement
If a vendor cannot answer these questions clearly, the legal team does not yet have enough evidence to approve the tool for sensitive or professional use.
Can our prompts, files or outputs be used to train or improve your models?
Where is data processed and stored, and which international transfers may occur?
How long do you retain prompts, uploaded documents, generated outputs and logs?
Who can access our data inside your organization or subcontractor network?
Can we delete, export and audit our data at the end of the contract?
Which legal sources does the tool use, and how often are they updated?
How does the system reduce hallucinated cases, statutes and citations?
Does the tool provide source links or traceable evidence for legal assertions?
What incident-notification commitments are included in the contract?
Does the product support role-based access and usage logs?
Which security certifications or independent audits can be reviewed?
What happens if a generated output causes professional or institutional harm?
Can the tool be tested with our local Brazilian legal questions before purchase?
What changes require prior notice, such as model replacement or terms updates?
Validation protocol
How to validate legal sources generated by AI
The review should begin before the text looks final. Fluent legal writing can create false confidence, so teams need a repeatable protocol.
Start with the generated legal proposition, not with the polished paragraph.
Check every statute, article, case, agency act and quotation in an official or trusted source.
Verify whether the authority is current, applicable and not reversed, superseded or jurisdictionally irrelevant.
Compare the AI answer with at least one independent legal source or professional review.
Mark unsupported statements as drafts until evidence is found.
Keep a review trail for outputs used in legal opinions, contracts, petitions or institutional decisions.
Do not let the AI summarize confidential facts unless the tool has been approved for that data class.
Require final sign-off by a responsible legal professional.
LGPD and professional secrecy
Data protection is only one layer of the legal-risk analysis
International readers often map every Brazilian privacy question to a GDPR-style checklist. That is useful, but incomplete. In legal work, a tool may satisfy a formal privacy requirement and still be inappropriate because the uploaded material is protected by professional secrecy, litigation strategy, client privilege expectations or public-sector confidentiality rules.
The Brazilian General Data Protection Law, known as LGPD, requires a legal basis, transparency, proportionality, security and governance for personal-data processing. A legal AI evaluation must identify whether prompts, documents, embeddings, logs, metadata and outputs contain personal data, sensitive data or information capable of identifying parties, employees, witnesses, clients or public servants.
For law firms and legal departments, the harder question is not only whether a vendor says that data will not train a public model. The institution must understand where the data is hosted, who can access support logs, whether the model provider or integrator has subprocessors, how deletion works, whether prompts are retained, and whether confidential legal material can be reconstructed from histories, exports or analytics dashboards.
A responsible internal policy should therefore separate low-risk educational prompts from client documents, draft pleadings, contracts under negotiation, due-diligence materials and evidence. The more sensitive the file, the more the institution should require contractual safeguards, access controls, logging, deletion rights, incident procedures and explicit human authorization before use.
Data map
Document what enters the system, where it is processed, who can access it, how long it remains available and whether it can be used for training or evaluation.
Secrecy layer
Treat professional secrecy and litigation strategy as independent constraints, even when a privacy assessment appears manageable.
Exit evidence
Ask vendors to explain deletion, export, suspension and migration procedures before the tool becomes operationally indispensable.
Responsibility
Human review cannot be symbolic
In legal practice, a human-in-the-loop label is not enough. The relevant question is whether the reviewer has time, authority, expertise and evidence to reject the output.
Many AI policies fail because they say that a lawyer or analyst must review the result, but do not define what review means. Reading a polished answer is different from checking sources, testing counterarguments, confirming dates, identifying missing exceptions and deciding whether the output is suitable for the specific client, case or institution.
A serious workflow assigns responsibility before the prompt is written. Who may use the tool? Which matters require supervisor approval? What evidence must be saved? When must the use of AI be disclosed internally? Which outputs are prohibited from direct filing, signature, publication or client delivery? These questions convert human review from a slogan into a control.
The institution should also avoid blaming the individual professional for every failure created by poor procurement. If the tool is inaccurate, opaque, badly configured or integrated into a rushed workflow, governance must address design and management choices, not only final-user conduct. Responsibility is shared across procurement, information security, legal leadership, training and day-to-day practice.
| Review layer | Minimum expectation | Evidence |
|---|---|---|
| Legal source review | Authorities must exist, be current and support the proposition stated. | Source list, screenshots, database links or reviewer notes. |
| Matter suitability | Use must match the sensitivity, jurisdiction, deadline and professional role involved. | Use-case classification and approval record. |
| Output approval | Final legal content must be accepted by a responsible professional, not by the model. | Named reviewer, date, version and decision notes. |
Shadow AI
The biggest risk may already be inside the institution
Legal AI governance should not assume that adoption begins on the procurement date. In many organizations, professionals are already testing public tools informally before leadership approves a policy.
Shadow AI emerges when teams use external systems without approved accounts, logging, training or data-classification rules. The motivation is often practical: a lawyer wants to summarize a document, compare arguments, translate clauses or prepare a first draft under deadline pressure. The risk is that confidential material may leave the institution before anyone has assessed the provider, contractual terms or retention practices.
A purely prohibitive response rarely works. If the institution only bans every tool, usage may move further out of sight. A better approach is to define permitted experiments, prohibited data categories, approved tools, escalation channels and safe examples for training. Governance should make responsible behavior easier than improvised workarounds.
The evaluation process should include interviews with real users. Ask which tasks they already automate, where they feel pressure, what kind of output they trust, which mistakes they have seen and what rules they consider realistic. This does not weaken governance; it makes the policy fit the actual workflow.
Policy before punishment
Start with clear rules, approved examples and safe channels for doubts. Discipline alone does not create an adoption system.
Pilot with constraints
Run small pilots with anonymized material, controlled accounts, source-validation routines and documented feedback before sensitive deployment.
Training by scenario
Teach professionals with examples from contracts, litigation, public-sector work, research and publication, instead of generic AI slogans.
Continuous monitoring
Review logs, incidents, user questions and output failures after launch. The first approval is not the end of governance.
Scoring
A practical scoring matrix for decision meetings
| Criterion | Question | Evidence to request |
|---|---|---|
| Data risk | What type of data enters the tool? | Public, internal, confidential, personal, sensitive or privileged |
| Legal accuracy | Does it cite and apply law correctly? | Local statutes, cases, doctrine and procedural context |
| Source transparency | Can the user inspect the basis for answers? | Links, citations, excerpts and retrieval logs |
| Contract strength | Are vendor duties enforceable? | Retention, training, breach, audit and liability clauses |
| Security | Can the tool protect legal material? | Authentication, encryption, access logs and incident response |
| Human workflow | Who reviews and approves output? | Review policy, role assignment and escalation |
| Operational fit | Will the tool work with existing routines? | Integrations, training, cost and support |
| Exit risk | Can the institution leave? | Portability, deletion and continuity plan |
| Public accountability | Is transparency required? | Public-sector use, citizen impact and auditability |
| Maturity | Is the organization ready? | Internal policy, training, governance owner and review cadence |
Internal policy
Minimum rules before broad adoption
The first internal policy does not need to be long. It needs to be clear enough to stop unsafe uploads, unsupported citations and unreviewed legal outputs.
No confidential client or citizen data in unapproved tools.
No legal citation may be used without source verification.
Every AI-assisted legal output must have a responsible reviewer.
Users must separate brainstorming, drafting and final professional work.
Sensitive data requires prior privacy and security assessment.
Vendors must be reviewed before procurement or broad use.
Incidents, hallucinations and unsafe outputs must be reported.
The policy must be reviewed after pilots, model changes and legal updates.
Implementation
A 30-day plan for safer adoption
Days 1-7
Map current use
Identify who is already using AI, which tools are being used and what data may be entering those systems.
Days 8-14
Classify risks
Separate public experimentation from confidential, personal-data and professional-output use cases.
Days 15-21
Test and document
Run local legal tests, register hallucinations, review vendor answers and create a minimum evidence file.
Days 22-30
Publish rules
Approve a short internal policy, name a governance owner and define the next procurement or training step.
Ecosystem
How this dossier connects to Lantyer
This dossier bridges public reading and institutional application. Articles explain the legal debates, the Radar monitors new signals, the Observatory maps tools and risks, and institutional training turns the framework into policy and practice.
Sources and reference layers
What informs this framework
FAQ
Frequently asked questions
Does this dossier recommend a specific legal AI vendor?
No. The dossier is an educational and editorial evaluation framework. It does not sell, rank, certify or endorse software vendors.
Why does a Brazilian dossier matter to international readers?
Brazilian legal teams face the same adoption pressure as global teams, but with LGPD, civil-law reasoning, professional secrecy and public-sector constraints. The framework is useful precisely because it makes those local variables explicit.
Can this replace legal, privacy or security advice?
No. It is a structured educational resource. Concrete procurement, privacy, security and professional-responsibility decisions require specific analysis.
Are the downloadable PDFs in English?
The current public PDFs are the Portuguese materials generated for the original dossier. The English page provides an adapted static version for international readers.
Need to train a legal team?
Lantyer can turn this framework into an institutional lecture, applied workshop or internal policy session for law firms, legal departments, public bodies and faculties.
Request institutional training