The Defense Information Systems Agency (DISA) has artificial intelligence (AI) on the brain, and coming in first on the list of the agency’s lineup of new AI-driven projects is “Concierge AI” – a new tool that integrates data with AI.
In an interview with MeriTalk, Steve Wallace, director of emerging technology and chief technology officer at DISA, said the agency remains committed to delivering on the Concierge AI tool by this summer.
Concierge AI aims to integrate data with AI and minimize friction for users to find and analyze data. According to Wallace, the tool’s functions will help improve decision-making processes. The digital concierge will take data from controlled unclassified information settings and drop them into a database where – with the help of large language models (LLMs) – that database will present users with answers.
DISA’s goal is to feed internal instructions and related data into the LLM database, and let users ask questions and receive accurate answers.
Wallace acknowledged that the start of the summer delivery period is fast approaching – approximately two and a half months away – but he explained that his team continues to move forward to meet that date.
“[We’re] moving along,” Wallace said. “My goal was this summer, and that’s what we originally set out to do, and I’m sticking with that.”
The program’s initial pilot – launched earlier this year – provided security personnel and users with “a level of comfort” in using LLMs in a government cloud environment at Impact Level 5.
However, there were some challenges concerning how the data was ingested. For example, the database didn’t respond well to tables and tables of contents.
“We’re going to use a different model. And a different commercially hosted [service] to accomplish that,” Wallace said. “We’re still trying … we’re still learning … and working our way through some of these processes.”
DISA also faces a staffing challenge as it attempts to work through tracking down capabilities needed to deliver an effective Concierge AI product.
“A lot of time when I speak to industry, I try to caution around everyone coming at us these days with an AI-related solution or product. That’s great, and we really appreciate it … [but] we only have a finite set of resources and if we’re constantly engaged with industry, hearing about the latest and greatest thing, we’re not doing the work,” Wallace said.
The AI digital concierge product has many use cases, including maintaining after-action reports and as a frontline help desk aid. However, Wallace cautioned against trying to apply LLM capabilities to every case.
“[At this moment] it’s more about matching the right use case versus trying to solve every use case necessarily with a large language model,” he said. “In our field, people tend to get very excited. ‘Oh, I can apply it to all these different things.’ But the reality is there’s a finite set of use cases,” Wallace said.
Another Upcoming AI Effort at DISA
In its fiscal year 2024 tech watchlist – an annual broad inventory of over two dozen new technologies the agency has identified as areas of interest – DISA listed several items covering specific capabilities and issues related to AI, including leveraging LLMs and employing guardrails to AI development and deployment.
Recently, the agency launched a request for information (RFI) – posted to SAM.gov on March 25th – seeking information from industry to assist in developing and planning potential new AI models, tools, and services to enhance DISA’s Defensive Cyber Operations.
“We’re soliciting input from industry around the use of AI and large language models in defensive cyber operations,” Wallace said. “We’re looking for feedback and to launch a prototype later this year to bring some of that stuff together.”
DISA wants an AI platform to include a data acquisition and processing solution, a model development pipeline, and an operations and sustainment strategy. This AI/ML platform can be on-premises, in the cloud, or a hybrid approach, the RFI states.
Responses are due by April 30.
Fostering an AI Culture at DISA
The Pentagon is firmly on board the AI train, but like in any organization, that’s accompanied by a certain level of reluctance and healthy skepticism. Wallace believes that proving the effectiveness of AI to the workforce is critical to fostering an AI culture at DISA.
“We want to get out of the Homer Simpson approach of coming into work and punching the same button day after day. And this is where we see AI, automation of tasks, robotic process automation, and these sorts of capabilities help us get out of that monotonous button pushing daily,” Wallace said.
“It’s okay to have some skepticism, but the world is moving, and we need to help and move in that direction,” he added.
Additionally, Wallace explained that he sees many AI opportunities across the department, not just in the services and the combatant commands. But he also explained the department is dedicated to onboarding AI securely and responsibly.
“Being open to AI also means being aware of any risks around the tools and models,” Wallace said.
“How do we prevent bias, whether unintentional or malicious? How do we prevent bias in the answers delivered? How do we validate that? Is that the actual data source, or has it been tampered with? Ensure the model isn’t hallucinating or hasn’t posted a hallucination because of [a] malicious act? So [we’re] working on [several] of those guardrails,” Wallace said.
He explained that the industry also is rushing to jump on the AI train, but “while we certainly want to move quickly, we don’t want to move in in a risky way,” Wallace said.
