Machine Learning Infrastructure Engineer
Posted Updated
Description
You will design and improve an in-house workflow orchestration framework and build the scientific preprocessing pipelines that run on it. You will optimize workflow performance, implement caching and lazy execution, improve developer experience, and translate research workflows into scalable, reproducible systems.
Responsibilities
- Design and evolve the workflow orchestration framework
- Build and optimize preprocessing pipelines for protein, chemical, simulation, and ML data
- Profile workflows and eliminate performance bottlenecks
- Develop lazy execution, incremental computation, caching, and artifact reuse
- Translate scientific workflows into scalable computational pipelines
- Improve pipeline authoring, debugging, monitoring, and extensibility
- Make architecture decisions for execution, storage, scheduling, lineage, and reproducibility
- Improve the orchestration platform using lessons from scientific pipelines
Requirements
- DAG
- Workflow orchestration
- Distributed systems
- Storage
- Serialization
- Concurrency
- Resource scheduling
- Performance optimization
- Data pipeline
- Scientific workflow
Benefits
- Equity
- Medical insurance
- Dental insurance
- Vision insurance
- 401(k) plan
- Unlimited PTO
- Free office lunches and dinners
- Paid family leave
- Life insurance
- Long-term disability insurance
- Short-term disability insurance