Skip to content

Exa

Software Engineer (Knowledge Engine, Data Infrastructure)

Posted Updated

Description

  • In this role, you’ll build the knowledge construction pipelines: turning everything from a 100k docs/s web crawl to vendor datasets into a graph of what is true about the world, and keeping it correct as data changes
  • This is greenfield work with a massive scope. There is no off-the-shelf system for building a model of the world from the open web, so you’ll make foundational decisions, from how claims are represented and resolved to how trust is assigned and how the graph evolves over time, that everything built on top of it depends on
  • Continuously synthesize the live web crawl, vendor deliveries and APIs, and other off-the-web sources into one typed graph: extraction that turns pages and records into ontology-shaped claims, each carrying its provenance, timestamp and confidence
  • Bring every new source in cleanly: versioned, append-only history with schema evolution, late-arriving and corrected records, and replayable cursors, so every claim traces back to the exact delivery that produced it
  • Decide what to trust: entity resolution across billions of records (blocking, pairwise scoring, clustering) into canonical people, companies, places and publications, then truth discovery that weighs source reliability, recency and corroboration to reconcile conflicting claims
  • Keep that picture current and convergent: streams arrive continuously, out of order and at very different rates, so the graph is eventually consistent by design. You’ll build incremental computation over change data feeds, where each update is a diff of what changed and every job is idempotent and resumable, so the graph converges to the same state however runs interleave, fail or replay, and readers only ever see committed, consistent versions
  • Measure precision and completeness: labeled evals for extraction and resolution, per-source coverage and correctness metrics, and lineage that pinpoints which source or stage broke and why
  • You understand incremental processing deeply: change data capture, idempotent writes, backfills and reprocessing
  • You’ve built and operated large-scale distributed data pipelines in production over petabytes of structured and unstructured data, with Spark over a lakehouse format (Delta Lake, Iceberg, Hudi)
  • You obsess over correctness, reliability and data quality as much as scale
  • Strong in Python and at least one systems language; you can read and tune Spark query plans
  • Entity resolution, record linkage, knowledge graphs or ontologies
  • Information extraction or truth discovery at scale
  • Lance or other columnar/vector-native formats

Skills

Apply

Apply to Software Engineer (Knowledge Engine, Data Infrastructure) at Exa

Create a free account first, or go straight to the employer.

Free RoleWatch account

  • Get an email when new jobs match your search
  • Save jobs and track every application
  • See how well your CV matches each job

Already have one?

No account needed. Opens the employer's own site in a new tab.

See also

Software Engineering jobs by country — openings, pay and top skills →