Description du poste
About the role We are seeking a contractor who combines legal reasoning with hands-on AI engineering to implement several data development and data analysis projects. The work spans the full process: defining the legal questions, building the workflow that answers them, evaluating its accuracy, and improving it when the analysis fails. This is not a position focused on using chatbots to draft legal work. You will build and test systems that analyze legal records at scale. You should be comfortable reading an appellate opinion closely and building the system that makes that analysis repeatable. About Scrutinize Scrutinize uses data and AI to make state courts transparent and accountable. We are the only organization publishing data-driven records of individual judges at this scale, with a current focus on New York State. Since our founding in 2023, we have published profiles of more than 1,200 New York trial judges, each grounded in rigorous data and legal analysis. Our newest project, Decoded , is a public, plain-language record of how the New York Court of Appeals and each of its judges have ruled, sorted by the issues that shape life in New York. Decoded, and the projects our contractor will work on, use AI at all stages, from engineering unstructured data to running legal analyses at scale. Our work is impactful: it is used by decision-makers, advocates, journalists, and voters. It has been recognized by The David Prize and Ironclad’s Legal AI Innovators Award, and backed by Fast Forward, Harvard Law School, Tyler Cowen’s Emergent Ventures, and others. Our collaborators include the New York Civil Liberties Union; clinics and centers at Harvard, NYU, and Cornell law schools; and leading watchdog groups like Reinvent Albany. Why work with us Your work is published and used . What you build goes onto a public site that journalists, advocates, and voters consult, not into an internal tool. You will have real ownership . We are a very small team. You work directly with the founder, and your design decisions will shape the product and its impact. The problem is hard and new . The challenge is to use AI to analyze legal issues and legal datasets accurately and reliably enough to publish judge-by-judge records. How the work is done Our pipelines are built by directing AI coding agents (such as Claude Code and Codex), not by writing code by hand. You design the system, specify the work, and verify the result. We expect you to already work this way but to have experience with traditional coding and data pipeline designs. What you will do Specify the legal analysis. Translate legal questions into clear instructions, categories, decision rules, and structured outputs. Distinguish holdings from dicta, majority reasoning from separate opinions, and genuine legal disagreement from differences in wording. Build AI workflows. Design and maintain multi-step workflows using language-model APIs, prompting, document processing, and structured extraction. Integrate them with existing datasets and pipelines. Evaluate and improve accuracy. Build test sets and validation criteria; compare results against source documents and expert judgments; investigate error patterns; and revise prompts, inputs, code, or methodology. Design controlled experiments that separate the real effect of a prompt or model change from run-to-run variation. Define when a result should be withheld for human review. Make the work reproducible. Maintain source references, traceable outputs, versioned code, and documentation. Improve reliability, efficiency, and scalability without sacrificing legal accuracy. Work across disciplines. Collaborate with Scrutinize’s partners to identify useful analyses and communicate both findings and limitations plainly. Who should apply We welcome applicants who developed their expertise through either law or engineering. A JD and appellate experience are preferred, but neither is required. What matters is demonstrated competence in both legal analysis and technical implementation. Legal-first candidates: A JD and substantial experience interpreting appellate opinions or other complex legal materials, together with a demonstrated record of building working AI systems. Technical-first candidates: Strong software and applied AI experience, together with meaningful experience working with judicial opinions, legal research, or other legal documents that require substantive interpretation. Skills we need Engineering using AI agents: Design. Enough software background to decide what a pipeline should look like before any code exists: its stages, what each stage receives and returns, what gets saved, and how failures are recorded. Specification. Ability to give coding agents precise instructions and standing rules, and to break a large change into pieces an agent can complete correctly. Verification. Ability to establish that agent-written code does what was intended without taking the agent’s word for it: tests, adversarial audits, inspection of saved outputs, reading code, and controlled before-and-after comparisons. Scale. Experience running long, multi-stage jobs over thousands of documents, including resuming after failures, handling rate limits, and controlling cost. AI development: Experience creating, testing, and refining multi-step LLM workflows, including writing prompts and running systematic evaluations. Practical experience matters more than familiarity with a particular model. Legal judgment: Ability to identify legally significant distinctions, assess whether an interpretation is supported by a source, and recognize uncertainty rather than force a confident answer. Working style: Careful, independent problem-solving; clear writing; and the willingness to revisit an approach when evidence shows it is wrong. Engagement Independent contractor, $75-$100 per hour depending on experience. Up to 20 hours per week for six months. Remote. You should be available during working hours in a continental US time zone. How to apply Fill out our application form and upload a resume. The form asks you to sign in with a Google account to upload it. The hiring process will include multiple interviews. Scrutinize selects and engages contractors and employees without regard to race, religion, color, national origin, sex, sexual orientation, disability, age, veteran status, ancestry, citizenship, marital status, gender identity, or any other status protected by applicable law. We consider qualified applicants regardless of criminal histories, consistent with legal requirements. If you need an accommodation during the application process, please let us know in your application.