AI Risk Management Researcher - Offense/Defense Balances

Job Summary

Lead research on the offense/defense balances of advanced AI systems and their capabilities. Use methods from safety engineering operational risk management cybersecurity and other disciplines to develop critique and make the case for new analytical methodologies with a focus on modeling societal impacts. These analyses and methodologies would be intended to be adapted and adopted by AI evaluators auditors standards bodies and policymakers.

About the Organization

The Center for AI Risk Management & Alignment (CARMA) envisages humanity being able to pull itself together enough to steer the reins of its perilous journey to transformative AI successfully. CARMA's mission is to help accomplish that to lower the risks to humanity and the biosphere from transformative AI. Via better grounding AI risk management via policy research squarely addressing AGI via showing a path to better tradeoffs in technical safety and by fostering global perspectives on durable safety CARMA aims to provide critical support to society for managing the outsized risks from advanced AI.

CARMA is a fiscally-sponsored project of Social & Environmental Entrepreneurs Inc. a California 501(c)(3) nonprofit public benefit corporation. We collaborate closely with a variety of NGOs in the space and are well connected to achieve real impact with the gaps we will.

Responsibilities

* Apply a security mindset to analyzing prospective AI proliferation and usage dynamics

* Combine methods from safety engineering operational risk management cybersecurity and a host of other disciplines to develop analyses and methodologies to model and weigh the offense/defense balances of differentially introduced capabilities

* Research and develop impact assessments related to global security national security public health wellbeing and other topics related to AI risk management

* Develop methods of aggregating modeled effects for AI models AI systems datasets services and other aggregate artifacts

* Write papers explaining groundings for the risk estimation methods you will develop

* Draft guidance for AI developers and auditors appropriate as contributions to AI standards working groups

Requirements

* A M.Sc . or higher or a B.Sc . plus 4 years of experience in either Computer Science Security Studies Risk Management AI Policy Cybersecurity or a closely related field

* A demonstrated focus on any one or more of: machine learning AI safety engineering AI policy complex systems operations research or operational risk management or other relevant domains

* Experience in any of the following: Security mindset Security studies research Cybersecurity Safety engineering AI governance Operational risk management Catastrophe risk management Operations research Industrial engineering Futures studies Foresight methods Leading labs Ontologies and knowledgebases Incentive studies Criminal psychology or Technical standards development

* Skilled in developing informal semi-formal and formal models

* Relevant publications

* Skilled in technical and expository writing

Pluses

* Fluidity in using the modern cognitive AI stack of LLMs prompt engineering and scaffolds

* Experience with scalable graph models

* Experience in probabilistic programming environments

CARMA/SEE is proud to be an Equal Opportunity Employer. We will not discriminate on the basis of race ethnicity sex age religion gender reassignment partnership status maternity or sexual orientation. We are by policy and action an inclusive organization and actively promote equal opportunities for all humans with the right mix of talent knowledge skills attitude and potential so hiring is only based on individual merit for the job. Note that we are unable to sponsor visas at this time but non-U.S. contractors would also be considered.

$150000 - $180000 a year

Salary plus good benefits