Skip to content

ServiceNow

NewBe an early applicant

Software Engineer (Agent Evaluation Platform)

Posted Updated

Description

  • Moveworks’ AI agents plan, call tools, and change real state in enterprise systems. This role owns the measurement signal that scores agent behavior precisely enough to help agents improve. You’ll build the judgement layer of the agent evaluation platform: the rubrics, judges, calibration against human labels, and methodology that make a score meaningful. A calibrated judge can also provide a reward signal for training. This is applied ML methodology work, not a pretraining or testing role
  • We’re hiring across three areas. You’ll anchor on one and touch the others; which one is a conversation we have with you, not a slot we drop you into
  • The runtime that executes multi-turn agent scenarios end-to-end: stand up the environment and user simulator, drive the user↔agent↔world loop, collect transcripts, traces, and final state, run validators and scoring, and tear down
  • Handle scheduling, retries, high-concurrency execution, and run isolation at production dataset sizes
  • Create versioned specs, datasets, and reports, with run-to-run comparison as a first-class operation
  • Consolidate evals that run today as one-off workflows onto a single orchestration service—one source of truth, one place to schedule and retry
  • Establish a reliability floor and an SLO for the harness itself
  • Make evals self-serve so any team can run one without bespoke integration
  • Lead the move to OpenTelemetry-native observability for the agent platform, replacing the parallel per-service logging, correlation, and redaction mechanisms in use today
  • Build the span data model for agent trajectories—prompts, tool calls, plan updates, and outcomes—so a trajectory is queryable, not reconstructed by hand from log files
  • Implement trace context propagation across async boundaries and sessions that stay alive for minutes or hours
  • Make full prompts and completions survive the pipeline intact, and keep eval traffic from contaminating its own data
  • Build fault attribution and cross-run diffing to identify which component broke and what changed since the last green run
  • Build the debug surface support and harness engineers use, and the tracing contract with the team that builds the agent
  • Build the simulation environment: stateful fakes of the enterprise systems agents call—ITSM, HR, knowledge bases, and inventory—backed by a real datastore that persists changes during a run, so a created ticket is visible to a later read
  • Implement per-run data injection and programmatic setup and teardown so every run is hermetic and repeatable
  • Build LLM-driven user simulators for open-ended personas and scripted state-machine simulators for deterministic flows
  • Build contract-testing mocks against real API schemas in CI, so simulation fidelity can’t quietly drift as vendor APIs change
  • Work toward isolated sandbox environments reproducing the config, identity, search content, and permissions an agent actually reads—provisioned from an identical baseline and torn down every run
  • Across all three areas, lay the foundation for using eval signal to optimize the agent, not just measure it

Benefits

  • Generous family leave
  • Matched donations
  • Annual learning stipends
  • Flexible PTO
  • Competitive retirement plan
  • Paid volunteer time
  • Concurrent and async programming: Python asyncio, Go concurrency, and structured cancellation
  • Orchestration and workflow runtimes: DAG execution, scheduling, retries, backfills, and high-concurrency job systems (Temporal, Airflow, Argo, or something you built yourself)
  • gRPC/protobuf service and interface design
  • Observability internals as a builder, not just a user: OpenTelemetry SDKs and collectors, semantic conventions, span context propagation, and high-cardinality trace data
  • Strong in Python or Go (ideally both)
  • 0–2 years building production backend or infrastructure systems
  • Comfort making a non-deterministic system measurable. You don’t need an ML background, but you should find it interesting to turn fuzzy agent behavior into a signal engineers are willing to gate releases on
  • Distributed systems: idempotency, delivery guarantees, isolation, and—unusually central here—determinism and reproducibility
  • Data-intensive pipelines: high-volume ingest, schema evolution, and sampling and retention trade-offs
  • Experience in at least 3 of these areas:
  • Comfort with ambiguity; these are novel problems without textbook solutions
  • Experience designing and operating systems that handle real traffic at scale

Skills

Apply

Apply to Software Engineer (Agent Evaluation Platform) at ServiceNow

Create a free account first, or go straight to the employer.

Free RoleWatch account

  • Get an email when new jobs match your search
  • Save jobs and track every application
  • See how well your CV matches each job

Already have one?

No account needed. Opens the employer's own site in a new tab.

See also

Software Engineering jobs by country — openings, pay and top skills →