Machine Learning Engineer

Role Summary:
- As our Machine Learning Engineer - Evals, you will take full ownership of evaluating Honcho end-to-end.
- Build the core benchmarking harness that informs our research-driven team whether our entity representations are truly improving.
- Plastic Labs develops the core memory and identity layer powering agentic systems, and our product is already live in production.
- Because this involves complex multi-agent architectures with emergent black-box dynamics, your primary challenge is establishing rigorous standards for what constitutes quality in evolving representations without fixed ground truth, then architecting the underlying measurement infrastructure.
- If you thrive on constructing evaluation metrics from scratch, analyzing deep execution traces to uncover root-cause issues, and directly engineering production-grade solutions, this team is an ideal fit.

Responsibilities:
- Architecting custom eval suites: establish objective scoring mechanisms and benchmark definitions for evolving entity representations, continually adapting them as system capabilities mature.
- Building robust evaluation infrastructure: construct scalable pipelines, automated labeling pipelines, LLM-as-a-judge frameworks, and versioned regression runs to ensure rapid response cycles.
- Extracting actionable insights: inspect execution traces and benchmark outputs to isolate systemic failures, delivering high-conviction system design updates that directly enhance output fidelity.
- Scaling agent simulation environments: expand continuous multi-agent testing capabilities to unlock higher throughput for concurrent entity evaluation.
- Driving complete execution loops: lead problem definition, data pipeline construction, automated benchmarking, and technical writeups with total autonomy and minimal administrative overhead.

Requirements:
- Must have: Track record of architecting, implementing, and running LLM evaluation frameworks end-to-end in real-world software environments.
- Must have: Production-level Python expertise with emphasis on automated test suites, maintainability, and release engineering.
- Required: 3+ years of experience designing and owning LLM evaluation platforms.
- Required: Deep experience with PyTorch, Hugging Face ecosystem, and Weights & Biases; familiarity with data pipeline tools like Hydra, Kafka, or SQL is a strong plus.
- Nice to have: Hands-on exposure to agentic evaluation, memory systems, user identity models, or personalization infrastructure.
- Nice to have: Primary or co-authorship on peer-reviewed research or equivalent high-caliber open-source contributions.
- Nice to have: Degree in Computer Science, Math, Stats, or relevant quantitative field from a recognized university.

Location: New York, United States

Skills required for this job:

• Agentic systems
• Data pipelines
• Hugging Face
• LLM evaluation
• PyTorch
• Python