Data Scientist - Agentic AI (US)
Draftwise
About Us
Founded in 2021, Draftwise is a contract drafting, review, and negotiation AI powered by your legal knowledge. Draftwise executes complex drafting, review, and search workflows directly within Microsoft Word, drawing from your organization's best guidance, precedent, and language.
Draftwise serves top law firms and legal departments globally, including over half the Vault 10, dozens of Am Law 100 firms, and Fortune 500 organizations. The company is headquartered in New York and has offices in London.
Join Our Mission
Draftwise is building AI that makes contract work faster and more precise for the world's leading law firms. Our platform unlocks institutional knowledge from decades of legal work, helping lawyers draft and negotiate with greater efficiency and accuracy.
You'll work alongside experienced engineers from companies like Palantir and Google, plus legal professionals from top global firms, building technology that elite lawyers use daily.
Why Join Draftwise
Proven Product-Market Fit
Our core technology serves the legal industry's most demanding clients. You'll build solutions that lawyers at Vault 10, Magic Circle, and AmLaw 100 firms rely on for their most important work.
Strong Foundation
Y Combinator alumni with $20M Series A funding from Index Ventures and Bek Ventures. Work with the latest generative AI and machine learning technology while having the resources to execute ambitious projects.
Meaningful Impact
Your work directly improves how legal professionals serve their clients. Build features that users actively love and depend on, with immediate feedback on real-world impact.
Global Offices & Remote Culture
Offices in New York, London, and San Francisco with the possibility of remote work. Work with a global team while maintaining the flexibility that works best for you.
What We Value
Strong communication skills in an open environment.
The ability to work independently and make informed decisions with minimal supervision.
Interest in working in a dynamic environment with dynamic objectives.
A commitment to autonomy, ownership, and delivering high-quality solutions.
Openness to giving and receiving constructive feedback.
About the Role
We are seeking a Data Scientist focused on Agentic AI to push the frontier of agentic AI applied to highly complex legal tasks. You will design data‑driven experiments on multi‑step agent workflows, define success and failure with legal SMEs, curate gold‑standard evaluations, and ship targeted fine‑tunes that improve accuracy, latency, and reliability in production. You’ll collaborate closely with our NLP/ML engineers and legal experts to translate messy, real‑world contract workflows into robust, measurable systems.
Key Responsibilities
Agentic Experimentation & Evaluation
Run rigorous, data‑driven experiments on complex agentic tasks (tool‑use, retrieval, drafting, review, redlining, summarization, citation).
Operationalize success by working with legal SMEs to define task‑level objectives, guardrails, and failure taxonomies.
Build and maintain evaluation suites (labeled datasets, prompts, harnesses, and regression tests) that demonstrate consistent, statistically significant improvement over time.
Production Analytics & Reliability
Instrument and analyze production metrics to diagnose agent behavior (e.g., tool‑selection errors, hallucination modes, slow paths, non‑determinism) and triage issues by impact.
Propose and test mitigations (workflow redesign, routing, guardrails, retrieval changes, prompt refactors, function‑calling updates).
Targeted Fine‑Tuning & Optimization
Identify high‑value candidate components for fine‑tuning (e.g., routing models, classification/refusal heads, RAG re‑rankers, drafting subtasks).
Train and evaluate fine‑tuned models (e.g., SFT/RFT/DPO) to improve task quality and reduce latency/cost.
Own offline → online rollout plans, A/Bs, canarying, and performance SLAs.
Data Management at Scale
Curate high‑quality labeled datasets with SMEs; design annotation guidelines and QA loops to ensure reliability and reproducibility.
Build scalable data pipelines for collection, cleaning, transformation, and versioning across sensitive legal corpora.
Cross‑Functional Integration
Partner with product and engineering to integrate models and eval gates into production workflows.
Share best practices on metrics, experimentation, and statistical rigor with adjacent teams.
Innovation & Thought Leadership
Track emerging methods in agent frameworks, tool‑use, retrieval, and evaluation; pilot promising techniques and productionize those that deliver measurable wins.
About You
Requirements
1+ years in applied NLP/ML or data science (or equivalent), including experience with LLM‑driven agents and traditional ML.
Demonstrated ability to design experiments, define metrics, and make statistically sound decisions (A/B testing, power analysis, regression testing, error analysis).
Hands‑on with evaluation dataset design and labeling workflows; strong instincts for dataset quality and drift.
Fluency with Python, modern ML/LLM tooling, data pipelines, and production metrics analysis.
Clear written and verbal communication with both technical and non‑technical partners.
Availability to work in Eastern US timezone
Nice‑to‑haves
Background in information retrieval or ranking for RAG.
Experience training and shipping targeted fine‑tunes (SFT, RFT) and measuring their latency/cost/quality trade‑offs.
Experience with function calling, structured tool‑use, routing, and guardrails.
Prior exposure to legal, compliance, or other high‑stakes text domains.
Mentorship experience and/or technical leadership in experiment design and measurement.
What We Offer
In-office or remote with offices in NYC and London
Meaningful equity and competitive compensation
Fully covered medical insurance
Equipment & workspace stipend (new laptop + home office budget)
Unlimited PTO & sick leave
401k, FSA, HSA
A chance to shape a category‑defining product and make measurable impact for the world’s most influential law firms
