Software Engineer, Cybersecurity Benchmarks

This position is listed on behalf of a partner company, which manages all applications and next steps. Our partner is looking for a Software Engineer, Cybersecurity Benchmarks based in the United States.

As a Software Engineer, Cybersecurity Benchmarks, you will play a key role in evaluating the cybersecurity capabilities of advanced AI models through rigorous, reliable, and publicly accessible benchmarks. You will design, develop, and maintain evaluation frameworks while ensuring that testing environments remain secure against increasingly capable AI systems. Working at the intersection of software engineering, cybersecurity research, and artificial intelligence, you will contribute to innovative experiments and threat-modeling strategies. You will collaborate closely with researchers, analysts, and engineers to ensure evaluation results are accurate, meaningful, and actionable. Your work will help establish an independent understanding of AI cybersecurity capabilities and their potential risks. This is an opportunity to influence how emerging AI technologies are assessed while contributing to research that informs developers, policymakers, and the wider technology community.

• Design, develop, and maintain cybersecurity benchmarks for evaluating AI capabilities within the organization's evaluation infrastructure, primarily using the Inspect library.
• Develop and prototype new benchmark concepts, experiments, and evaluation methodologies to measure emerging AI cybersecurity capabilities.
• Maintain and improve existing benchmark suites, ensuring evaluations can be executed reliably and efficiently whenever new AI models are released.
• Design, implement, and strengthen secure sandboxing environments to prevent highly capable AI models from escaping their designated evaluation environments.
• Monitor emerging developments in AI agents, coding harnesses, and their interactions with software environments to anticipate potential security vulnerabilities.
• Apply threat modeling techniques to identify potential risks, guide benchmark design, and inform broader cybersecurity evaluation strategies.
• Contribute strategic recommendations on how AI cybersecurity capabilities should be measured, tested, and evaluated.
• Integrate with AI providers and configure benchmarks to run effectively within the organization's infrastructure.
• Collaborate with researchers, analysts, and fellow engineers to ensure evaluation data and outputs are accurate, insightful, and suitable for integration into research projects and publications.
• Help maintain robust evaluation infrastructure and continuously improve the reliability, security, and scalability of benchmarking processes.

Requirements:

• Several years of strong software engineering experience, including building, maintaining, and improving complex systems while consistently producing robust, reliable, and maintainable code.
• Practical experience designing and implementing software sandboxing solutions capable of defending against highly capable adversaries.
• Previous red teaming experience, including ethical hacking and identifying security weaknesses in software systems.
• Familiarity with agentic AI systems, AI coding agents, and the coding harnesses used to execute and interact with tools and environments.
• Strong problem-solving and creative thinking skills, with the ability to independently develop ideas for new cybersecurity benchmarks, experiments, and evaluation approaches.
• Genuine motivation to contribute to independent AI research and the development of rigorous, publicly accessible evaluations of AI capabilities.
• Professional-level English proficiency, with the ability to communicate technical concepts clearly and collaborate effectively in an international research environment.
• Ability to work independently in a fully remote setting while collaborating with multidisciplinary teams across different time zones.

Nice-to-Have Qualifications:

• Previous experience benchmarking AI systems and conducting model evaluations.
• Experience designing or managing Capture the Flag (CTF) infrastructure.
• Familiarity with using modern AI tools and systems for red teaming and ethical hacking.
• Strong understanding of AI coding harnesses, tool calls, and the mechanisms through which AI agents interact with external environments.
• Experience using the Inspect AI framework or familiarity with the Inspect library.
• Good understanding of current artificial intelligence developments, emerging capabilities, and cybersecurity trends.

Candidates who do not meet every listed qualification but have relevant experience and a strong interest in the role are encouraged to apply.

Benefits:

• Annual Salary: Competitive compensation ranging from $150,000 to $350,000, depending on experience and qualifications.
• Work Arrangement: Fully remote position, with opportunities to work from many countries.
• Flexible Schedule: Full-time employment with flexible working hours, ideally with some overlap with Pacific Time (UTC–8) and Greenwich Mean Time (UTC).
• Team Retreats: Opportunity to participate in three team retreats per year. Travel is not mandatory but is preferred.
• Research Impact: Opportunity to contribute to influential research and public evaluations shaping the understanding of AI capabilities and cybersecurity risks.
• Professional Growth: Work alongside researchers, analysts, and engineers at the forefront of AI benchmarking and cybersecurity research.
• Innovation: Freedom to propose and prototype original benchmark ideas and contribute directly to the organization's evaluation strategy.
• Meaningful Mission: Help develop independent, trustworthy research that informs policymakers, industry leaders, developers, and society about advances in artificial intelligence.
• Application Flexibility: Applications are reviewed on a rolling basis, with the possibility of starting as soon as the selected candidate is available.

We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.

We appreciate your interest and wish you the best!

Why Apply Through Jobgether?

Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.

#LI-CL1