Challenges in using AI for tooling in mission critical systems

While building SLIKWORKS, we used AI to automate much of the safety analysis work. Along the way, we encountered a number of challenges in applying LLMs and other ML models to this domain. These challenges are described in more detail in our SAE publication - AI-Powered FMEA in the Safety-Critical Industry. Below, we outline some of the key challenges we faced.

Fig: AI tool challenges in mission-critical systems

1. Hallucinations in LLMs

   
   

   Hallucinations in LLMs refer to instances where the model generates text that looks correct and confident, but is factually incorrect. One example we encountered was during FMEA (Failure Mode and Effects Analysis), where the model would produce incorrect results even for simple failure modes with obvious answers.

   
   

   LLMs are designed to generate an answer regardless of whether they're actually confident in it. In our case, this became a problem because we were asking too many questions and passing too much information: sometimes unrelated information. For instance, the model would reference the failure mode, effect, and detection mechanism of one component while trying to determine the failure mode and effects of an entirely different component, pulling in irrelevant data in the process. This made the question effectively ambiguous to the LLM, leading to irrelevant or incorrect answers.

   
   

   To mitigate this, we limited the context and amount of information passed to the model at any one time. We also introduced confidence scores to help filter out implausible answers.

   
   

2. Lack of context: LLMs are generic

   
   

   Frontier LLMs like those from OpenAI and Anthropic are trained on publicly available data. As a result, they may not have access to the specifications of proprietary components used in a customer's specific project. For example, an LLM will know general facts about GPUs or microcontrollers used in embedded controls, and what can typically go wrong with them. However, it won't know the specific design details, failure modes, or effects associated with a particular component developed by a specific organization.

   
   

   To mitigate this, Retrieval-Augmented Generation (RAG) can be used. A component's datasheet can be uploaded as a vector embedding, and this information can be passed to the LLM to generate failure modes, effects, and behaviors specific to that component.

   
   

3. Non-determinism of AI

   
   

   This is one of the biggest challenges when working with LLMs. Non-determinism refers to the phenomenon where the same input (same prompt, same model) can produce different outputs across different runs, rather than always returning the exact same answer every time. This can occur due to several factors: user-configured temperature settings on the LLM, infrastructure-level factors (such as batching behavior on the backend and GPU parallelism), and model updates made by the provider over time.

   
   

   Mitigating non-determinism is one of the most difficult problems in safety engineering. Some techniques include bounding answers : where LLM outputs are compared against known reference answers to keep them within an acceptable range, cross-verification using other models, and cross-verification against deterministic sources. In general, the entire chain of thought and reasoning traceability should be made transparent to the user.

   
   

4. Tool Classification and Qualification

   
   

   The tooling can be divided into a deterministic part and a non-deterministic part. The deterministic part functions similarly to any other tool used in safety engineering: a user enters data, it is stored in the backend, it can be retrieved, and it can be version-managed. This part of the tool can be classified and qualified following established safety standards such as ISO 26262 or IEC 61508. The non-deterministic part of the tool, which relies on LLMs/ML models, needs to be classified and qualified according to other, more specialized standards, such as ISO 22440 (still under development, as of Jul 2026)

   
   

5. Data privacy and security while using frontier LLMs

   
   

   Read this article - (Your Data, Your Rules: How SLIKWORKS Handles storage, privacy & GDPR) for details on how SLIKWORKS handles EU GDPR compliance, data privacy and security

   
   

6. Over trust in tool usage: Trust, but verify

   
   

   Given the rapid performance improvements of AI in recent months, it's easy to become complacent and trust its output without verification. However, LLMs in safety engineering are not meant to replace the human. They are meant to make human work easier, reduce the "busy work," and let safety engineers focus on higher-value thinking. It is important for engineers to stay grounded in the principle that there is always a human-in-the-loop intervention and check whenever AI is used in a mission-critical system.

Conclusions

Building SLIKWORKS taught us that LLMs bring real power to safety analysis, but also real challenges - hallucinations, non-determinism, domain gaps, and data privacy: all demand careful engineering, not blind trust. By combining deterministic safeguards, rigorous qualification standards, and a human-in-the-loop at every step, AI can genuinely accelerate safety engineering without compromising the rigor it depends on.