Staff Data Engineer

Role summary

The Staff Data Engineer designs, builds, and operates the data foundation and the evaluation infrastructure behind ServiceNow CRM Agentic AI. The role has two halves that reinforce each other. The first is conventional but demanding data engineering: architecture, pipelines, and transformation across structured and unstructured sources, held to production standards of reliability and quality. The second is newer and rarer: building the measurement layer that tells product and AI teams whether an AI agent actually did its job.

That second half changes the nature of the work. Agent behavior is probabilistic, so quality cannot be asserted, only measured—against metrics that have to be defined before they can be tracked, and against ground truth that someone has to establish and defend. This engineer defines those metrics with product and AI teams, sets the labeling and validation standards, builds the datasets that reflect how agents behave in the real world, and automates the pipelines and dashboards that turn agent execution logs into a signal the organization can act on.

the Staff Data Engineer owns these decisions across a domain rather than within a single project. They set design standards and quality gates others adopt, mentor junior engineers, and generalize evaluation patterns so each Agentic AI team is not rebuilding measurement from scratch.

What you do Design and architect data infrastructure

Design and oversee deployment of the data architecture and pipelines that capture, manage, and store structured and unstructured data from internal and external sources. Establish the processes and data flows across cloud services, local databases, and other applicable storage forms, and own the contracts between data producers and consumers.

Build and automate data transformation

Develop technical tools using machine learning and data-engineering techniques to cleanse, organize, and transform data. Implement automated processes that maintain the integrity of data structures and hold quality standards on an ongoing basis rather than at a point in time.

Define agentic evaluation metrics and ground truth

Partner with product and AI teams to define evaluation metrics for agentic workflows, including task and mission completeness, instruction adherence, tool use, and end-to-end workflow success. Establish ground truth labeling standards, annotation guidelines, and validation criteria, and design evaluation datasets that reflect real-world agent execution rather than idealized paths.

Build agentic evaluation pipelines

Design and implement automated evaluation infrastructure that measures AI agent performance using LLMs and agent execution logs. Create the dashboards, reporting, versioning, and reproducibility that make evaluation datasets and results trustworthy over time and comparable across releases.

Establish standards and continuous improvement

Create design standards and quality assurance processes for data systems. Define quality gates and validation frameworks, analyze workflow performance, and recommend optimizations that keep the platform aligned with evolving CRM AI requirements.

Lead cross-functional collaboration

Collaborate with product, engineering, and data science teams. Mentor junior engineers on data engineering and evaluation design, and generalize evaluation patterns and metrics so they can be reused across Agentic AI products rather than rebuilt per team.

Required qualifications

• Data engineering depth. 8+ years in data engineering, with a record of owning production data platforms end-to-end. Depth and demonstrated judgment matter more than tenure.
• Domain ownership. Demonstrated ownership of a data domain or platform, including architecture decisions, migrations, and the operational consequences of both.
• Evaluation infrastructure experience. Hands-on experience building measurement or evaluation infrastructure for machine learning or AI systems: evaluation pipelines, benchmark harnesses, quality dashboards, or golden datasets. This is the gating requirement for the role.
• Ground truth and labeling. Experience defining ground truth and running or directing labeling work, including guideline authoring and annotator agreement.
• Core technical stack. Production Python, strong SQL, distributed processing, workflow orchestration, a cloud data platform, and infrastructure-as-code, with continuous integration and on-call experience.
• Standard setting. Evidence of setting standards that others adopted, such as design standards, quality gates, or validation frameworks, rather than only meeting standards already in place.
• CRM and Order domain familiarity. Sales CRM, lead-to-cash, or order data familiarity, at a depth sufficient to judge whether an evaluation dataset reflects how sellers and agents actually work.
• Internal programs. Demonstrated experience in internal evaluation programs and patterns such as automated evaluation suites, data kits, or AI data factory approaches.
• Mentorship and generalization. Mentorship of junior engineers, and a record of generalizing a solution so that it was reused across teams or products.
• Education. Bachelor's degree in computer science, engineering, or a related technical field, or equivalent practical experience.