The Defense Advanced Research Projects Agency’s (DARPA) Defense Sciences Office (DSO) is seeking researchers who will work to ensure that artificial intelligence-powered robots don’t stray from their intended use and direction.
Foundation model (FM) robots signal a “dramatic break from existing autonomous systems” and DSO said it wants to ensure that the transformational capabilities it brings to robots can be relied upon. FM AI are large-scale pre-trained models that act as a general-purpose base for a variety of tasks.
Receiving up to $300,000 to complete the year-long study, selected researchers would provide assurances about robots using FMs, with a focus on robots that receive commands in natural language, operate in open-world environments, and incorporate FM in closed-loop decision making.
“Natural-language direction for open-world autonomy presents a critical challenge from a safety and assurance perspective, since current methods to assure learning-enabled systems are inadequate to address FMs operating in this paradigm,” reads the SAM.gov posting. “Current formal neural network verifiers have been effective in limited, narrow scenarios, but do not scale to large, state-of-the-art FMs.”
DSO’s request follows concerns about behaviors seen in FMs such as hallucinations, false confidence in reasoning, and “jailbreaking” manipulation.
“Assurances are crucial to deploy FM-enabled robots so they do not manifest these behaviors,” said DSO. “For example, in an unconstrained environment, a robot controlled by a hallucinating FM could fail to execute a critical task.”
DSO said it is particularly interested in assurances for complex robot instructions and behaviors, noting that research methods should include minimal intervention from human operators.
Research grantees would be expected to work full-time on the project, with DARPA noting that it may allow multiple researchers, but prefers individual researchers.
Proposals are due Jan. 13, 2025, and should explain why the researcher’s approach to assurance is innovative; include assumptions about FMs, robots, and environments; provide a research schedule; and include a literature review. If awarded, researchers would be expected to provide monthly technical and financial reports, and milestone reports.
