Building Trustworthy AI Systems for Critical Infrastructure

The problem

Buildings became critical infrastructure without anyone reclassifying them. In healthcare settings, ventilation is an infection-control system before it is a comfort system: reviews of hospital indoor environments link inadequate ventilation to the accumulation and spread of airborne pathogens, particulates, volatile organic compounds and anaesthetic gases, with secondary infection identified as a consequence.¹ Design defects in hospital HVAC translate directly into maintenance problems, service disruption and cost overruns, with air-handling units and ductwork singled out as critical to maintaining healthy conditions.²

The relationship is not simply more-is-better, which is itself instructive. A controlled reduction in operating-theatre air changes from a median of 32.8 to 20.7 per hour saved roughly 500,158 kWh annually with no significant change in surgical-site infection rates.³ The building was being run at a setting nobody had tested, in the safe direction, at measurable cost. Getting critical-facility ventilation right is a live engineering question in both directions, which is exactly the sort of question an optimisation system is now being pointed at.

Worth stating plainly at the outset: direct empirical measurement of what happens when HVAC fails in a data centre does not appear in the published record, and evidence for the consequences of ventilation failure in hospitals is largely indirect or modelled rather than observed.¹⁴ The premise of this article, that these are critical systems, is well supported in mechanism and thinly supported in incident data. That gap is part of the problem, because a risk nobody has measured is a risk nobody is pricing.

AI in these buildings has been climbing a ladder, and the assurance has not climbed with it.

It began by describing, then diagnosing, then recommending. It now increasingly acts. A model-free reinforcement learning system ran autonomously for 138 winter days across 13 real buildings, cutting heating energy by 29.7% against a multi-year historical baseline while maintaining comfort.⁵ Supervisory deep reinforcement learning in a large university building reported 36% lower energy cost than a comparable historical day, over a one-week experiment.⁶ Supervisory control against a calibrated model reported 6.5 to 7.6% annual HVAC savings, rising to 9.7% with supply-air ramp-up.⁷ These are real results and they are why the direction of travel is not in dispute.

They are also thin on the ground. One review found reinforcement learning present in only 23% of HVAC control studies,⁸ and a survey of real-world machine learning in building energy management systems identified 73 deployments in total, not all of them HVAC and not all of them supervisory.⁹ A field of a few dozen deployments producing double-digit savings figures is a field at the beginning of its evidence base, not the end.

What has not kept pace is the assurance offered alongside. The evidence presented for an AI system's fitness is almost always accuracy on a held-out set, which answers a question nobody in operations asked.

The gap is real, large and variable. It is not a constant you can subtract.

The most quantified version of that gap comes from clinical AI, where medical language models scored 84 to 90% on knowledge-based examinations and 45 to 69% on practice-based benchmarks, a shortfall of roughly 15 to 45 percentage points, with safety-focused performance lower still at 40 to 50%.¹⁰ The honest counterweight matters as much: a deployed pathology model showed no prospective decline at all, reaching an AUC of 0.890 prospectively against 0.847 internally and 0.870 externally.¹¹ Across predictive clinical decision-support studies, pooled real-world AUC came in at 0.652.¹² The gap is not a law. It is a range wide enough that no benchmark figure tells you where in it a given system will land.

Three properties of building operations make this worse than it sounds.

The world moves underneath the model. Data and concept drift are identified as a principal deployment challenge for machine learning in buildings.¹³ In model predictive control, forecasting models reused without refitting, and models trained on only three months of data, showed roughly 10% higher error than the baseline, while more than two years of training data delivered no significant further gain.¹⁴ The most useful finding comes from outside buildings: across four clinical AI systems tracked longitudinally after deployment, validation-era performance did not hold, calibration drift emerged consistently, and workflow-related changes in data availability and timing were more strongly associated with degradation than population-level shift.¹⁵ Translated to a building, the retrofit is not the main threat. The BMS integrator changing an export schedule is.

The counter-evidence deserves equal billing. Controllers deployed for six weeks with continuous online learning showed no meaningful deterioration.¹⁶ Nobody has published a decay curve for building AI across buildings and seasons, which means the correct claim is that drift is a documented mechanism with an unmeasured rate, not that performance reliably falls off a cliff.

The human safeguard degrades under the conditions it exists for. Every AI building deployment names a human in the loop as its answer to risk. The evidence says that arrangement is fragile in two directions at once, which is the part usually missed. Operators over-rely on capable automation, and the framing of the interface measurably moves that reliance.¹⁷¹⁸ They also override correct recommendations, to the point that human-automation teams can underperform the automation working alone, a problem that gets worse rather than better as the automation becomes more reliable.¹⁹ Adding transparency improved accurate use of automation and simultaneously increased agreement bias.¹⁸ There is no interface setting that solves both.

Alarm volume erodes response, measurably. Across 396,011 alarms in three Swedish emergency departments, each increase of 128 alarms per hour was associated with a one-second longer response to high-severity alarms and a 4.8-second longer response to low-severity ones, against medians of 11 and 132 seconds.²¹ In 118 hours of ICU observation, nurses did not respond to 53.3% of 460 alarms.²² Building operations already runs close to alarm saturation before AI adds to it. An oversight arrangement premised on an alert, sceptical operator at 03:00 on the four-hundredth uneventful night has not been tested against its own assumption.

Underneath all three sits an accountability question the industry has not answered. When an AI-influenced control decision contributes to an equipment failure or a compliance breach, the chain from the decision back to the evidence behind it is usually not reconstructable, because nobody built it to be.

Why current approaches fall short

Accuracy benchmarks measure the model on data resembling what it was fitted to. They say nothing about the edges, which is where critical infrastructure fails, and nothing about behaviour on input outside anything the model has seen. The benchmark-to-practice gap is large and highly variable,¹⁰¹² and it is not reliably present,¹¹ which is worse than a consistent penalty: it means the benchmark cannot be corrected, only distrusted.

Explainable AI is offered as the answer to opacity and is a partial one. Post-hoc methods differ in the features they select, the rankings they produce and even the signs of their attributions,²³ partly because they operationalise "importance" in incompatible ways.²⁴ Fidelity is not uniform: SHAP outperformed LIME on faithfulness, consistency, monotonicity and sensitivity in one security application,²⁵ and TreeSHAP reconstructed random-forest scores exactly while LIME proved simpler, less faithful and more variable across instances in healthcare tabular models.²⁶ Under adversarial perturbation the picture splits again, with SHAP retaining stronger feature alignment while LIME held better top-k rank stability but produced more spurious attributions.²⁷ No universal fidelity ranking is available. The practical conclusion for building operations is not that explanation is worthless but that an explanation of a model's internals is a weaker artefact than a traceable path through data an engineer can verify against the building itself.

Human in the loop, as a checkbox. Naming an approver satisfies a governance requirement and, unmodified, does not deliver oversight. A four-condition framework developed for medical AI sets the bar clearly: the person must understand the system and its errors, have sufficient time and cognitive capacity, hold real authority to override, and be able to intervene effectively.²⁸ Related work adds reliable error detection, calibrated trust, uncertainty communication, complementary task allocation, preserved human skill and clear accountability.²⁹³⁰ A review of 84 studies identified enabling design strategies for genuine collaboration rather than approval,³¹ and interview work emphasises the contextual and ethical judgment the human actually contributes.³² Over-reliance is a named risk in healthcare AI, though reported without a measured effect size.²⁰ Two findings should worry anyone designing an approval screen: incorrect AI advice reduces human accuracy, and does so most when the advice is presented before the person forms an independent judgment,³⁴ and fixed approval gates impose cognitive and latency costs of their own.³⁵ Where systems are opaque, complex and autonomous, they can simply exceed the comprehension of the person nominally supervising them.³⁶

Pilot projects. One building, one season, one attentive engineering team present throughout. Pilots demonstrate capability on a good day and generate almost no evidence about a bad day, which is the entire question for critical infrastructure. The sample-size problem is visible in the field's own results: the strongest deployment evidence available runs to 13 buildings over one heating season,⁵ and one of the most-cited savings figures rests on a one-week comparison against a single historical day.⁶

Responsible AI principles. Published principles are necessary and do not constrain a running system. ISO/IEC 42001 and the NIST AI Risk Management Framework are explicitly management-system and process instruments,⁵⁹⁶⁰ which is valuable, and is not the same as a technical limit on what a system may do at 03:00 on a Sunday.

Functional safety, transplanted directly. The instinct to apply IEC 61508 and its descendants is sound, and the transplant does not take cleanly. Classical functional safety assumes specifiable behaviour and verification against that specification. Assurance for machine-learned components has had to move from proving fully specifiable behaviour to constructing a lifecycle-based, evidence-rich argument over bounded properties, data quality, robustness, operational context, runtime safeguards and continual monitoring.³⁷³⁸ Formal approaches including reachability analysis, model checking, runtime verification, shielding and barrier functions can supply real guarantees, with acknowledged limits in scalability and real-world validation.³⁹ Where this has been tested against actual failures, crash investigations of automated systems found gaps in design-domain controls, redundancy, independent oversight, and in the reporting of machine-learning-specific evidence.⁴⁰

Cybersecurity as a separate workstream. An AI system with write access to building controls is an attack surface as well as a control system. Operational technology faces documented denial-of-service, data manipulation, device hijacking and physical tampering risks,⁵³ and attacks on energy infrastructure are a matter of record rather than speculation, as with the 2015 and 2016 attacks on the Ukrainian grid.⁵⁴ Machine-learned components add their own exposure: gradient-based evasion attacks have been demonstrated against power-system neural models, alongside poisoning, model extraction, reconstruction and membership inference in connected settings.⁵⁵⁵⁶⁵⁷ Building-specific incident data is largely absent from the literature; work on machine learning in building management systems addresses deployment constraints rather than documented breaches.⁵⁸ Assurance that stops at model quality has secured half the system.

What these approaches share is that they evaluate the model. Trustworthiness is a property of the system around it.

The core concept: earned, bounded, revocable authority

Trust here is not a sentiment and should not be managed as one. It is a decision about authority: what a system is permitted to do, under what conditions, on what evidence, and how the permission is withdrawn.

Three questions define it, and they are asked per decision rather than per system.

Answering those three produces a ladder rather than a switch. A system does not become trusted; it occupies a level, and it moves up by producing the evidence that level requires.

Rungs 4 and 5 are drawn dashed because no building deployment in the cited literature has earned them. The rung is a property of the decision, not of the product: one system occupies different rungs for different actions in the same building.

Two structural points carry most of the weight.

Earning authority, then, does not mean the model got better. It means the consequences of it being wrong were bounded, made detectable and made reversible before it was allowed to act.

How it works

Seven steps, in order, because each supplies what the next assumes.

Why it matters

Refusal by experienced engineers is rational, and it is the actual adoption bottleneck. The obstacle is not capability. A competent operator will not act on an output they cannot check, and they are right not to. Systems that surface their basis get used. Systems that assert get overridden and then ignored, and the override behaviour is itself a measured performance cost.¹⁹

Errors stay local instead of synchronising across a portfolio. A single engineer's bad judgment affects one building. A model deployed across forty buildings makes the same bad judgment in forty buildings on the same morning. Portfolio scale is sold as the upside of AI and is symmetrically the downside, which appears almost nowhere in vendor material. Per-building envelopes and bounded authority are what keep a wrong model a local problem.

Detection latency, not error rate, is what you are actually buying. Input-level monitoring turns a six-week discovery into a same-day one.¹⁵ That single change moves a whole class of decisions from unacceptable to acceptable without touching the model.

Liability follows the audit trail. A decision chain with provenance, versioning and approval records is defensible. A decision whose basis cannot be reconstructed is a loss whether or not the decision was right.

Regulation is converging on exactly these properties. The EU AI Act imposes risk management, data governance, automatic logging, transparency, human oversight and accuracy-robustness-cybersecurity duties on high-risk systems.⁶¹ NIS2 brings a wide set of essential and important entities into scope for security and incident-reporting obligations that reach the operational technology in their buildings.⁶² The recast EPBD pushes on the data side through the digital building logbook and building systems data access.⁶³ Whatever one concludes about which building applications fall inside the high-risk classification, a system built this way satisfies the requirements as a side effect rather than as a retrofit.

Trust becomes gradable rather than binary. The alternative to trusting an AI system with a building is not refusing it. It is granting the authority the evidence supports, and moving that level when the evidence moves.

Industry perspective

Three currents are pushing the same way.

Assurance practice is migrating from adjacent industries. Automotive, aerospace and clinical AI hit this problem earlier and converged on similar answers: an explicit operational design domain, a structured safety case rather than a performance claim, assurance extended across the whole machine learning pipeline rather than the model alone, runtime enforcement independent of the learned component, and continual post-deployment monitoring.^3738394243 None of this is novel engineering. It is unfamiliar in buildings, which is a different problem and a more tractable one.

Regulation is turning properties into obligations. Logging, traceability, oversight and robustness move from admirable to required for systems in scope of the AI Act,⁶¹ with security and incident duties arriving in parallel through NIS2 for critical entities.⁶² Management-system standards are maturing alongside them,⁵⁹⁶⁰ and they complement rather than replace a technical envelope.

Procurement is changing what it asks. The questions moving into tenders are shifting from what the model achieves to what happens when it is wrong: what the fallback is, what is logged, who can override, how drift is detected, and whether the semantic model is portable if the vendor is replaced. Standardisation gives that last question a real answer, as Brick, Project Haystack and the proposed ASHRAE Standard 223P converge rather than compete.⁶⁵⁶⁶ This is the healthiest of the three currents, because it prices assurance rather than legislating it.

How Struxiva applies these principles

Struxiva's position on the ladder is deliberate, and it is not the top rung. The platform describes, diagnoses and recommends. It does not write setpoints into plant. The reason is the argument above: the evidence base that would justify autonomous actuation in a critical building does not yet exist, in this industry or in our own results, and claiming otherwise would be a marketing decision rather than an engineering one.

What the platform does instead is make every rung below that one checkable.

The gate is the widest box in the diagram for a reason. Everything upstream of it is a proposal; nothing becomes an action, a report or an alert without passing a check written in code somebody can read.

Every figure carries its provenance. Source document, page, extraction method, validation status and date travel with the number, which is what makes rungs 0 and 1 defensible rather than asserted. An answer arrives with the path the system walked to reach it.

Confidence is computed from evidence, not requested from a model. It derives from extraction quality, source authority and cross-source agreement, and it routes. What reaches a person is what the system could not settle on its own, rather than a sample of everything, which is the alarm-budget principle applied at the source.²¹

Deterministic code owns everything that must be right. Ontology constraints, dimensional validation, unit conversion, reconciliation rules, graph traversal and every metric that reaches a report execute as inspectable code.⁶⁴ Generative components sit at the interpretation boundary, propose candidates with a source and a confidence, and write nothing directly.

Recommendations carry their basis, and they do not arrive first. A flagged fault comes with the evidence that produced it and the affected chain from the graph, so an engineer can disagree on grounds rather than on instinct. The interface is built so the recommendation supports a judgment rather than pre-empting one, which is the design implication of the anchoring evidence.³⁴

Monitoring watches the inputs. Points that stop reporting, exports that arrive late, distributions that leave the validated range and recommendation agreement rates are tracked continuously, because those signals move before any outcome does.¹⁵

The semantic model belongs to the customer. Built on open ontologies rather than a proprietary schema,⁶⁵⁶⁶ so the building's account of itself stays legible to any tool that speaks the same standard. Assurance includes being able to leave.

Key takeaways

• Accuracy is a property of a model. Trustworthiness is a property of the system around it, and only the second answers what an operator is asking.
• The gap between benchmark and practice is large, variable and not reliably present, which makes a benchmark figure something to distrust rather than something to discount by a fixed amount.
• Three questions set how much authority a system should hold, and they are asked per decision rather than per system: blast radius, reversibility and detection latency.
• Authority is a ladder from describing to acting autonomously, and each rung carries an evidence requirement rather than a promise.
• The operating envelope must be enforced by verified code outside the learned component. Shielding and constrained control provide this, with real guarantees when built on a formal safety model and weaker ones when built on a learned approximation.
• Every AI system with an operational role needs a defined safe state and a tested path to it. Degrading to conventional control is not a failure; failing without noticing is.
• Monitor drift on inputs rather than outcomes. Missing points and late exports give earlier warning than any result, and workflow changes degrade deployed models more than population shift does.
• Human oversight fails in two directions at once: operators over-rely on capable automation and also override correct recommendations, and teams can underperform the automation alone. Oversight has to be designed, budgeted and sequenced so the recommendation does not arrive before the judgment.
• Portfolio scale amplifies errors as efficiently as it amplifies value, and correlated failure across an estate is the risk vendor material consistently omits.
• Revocation criteria written before deployment are what make authority conditional. Without them, a trial is a permanent installation with an optimistic name.
• Runtime enforcement is the best-evidenced element of this design and has been demonstrated on HVAC. Most of the rest is evidenced in adjacent safety-critical domains and unmeasured in buildings, and the case is stronger stated that way.

Evidence & references

Bounded authority tells a building what its AI is allowed to do. It does not tell the building what will happen next. Every mechanism in this article is defensive: an envelope, a safe state, a monitor, a revocation trigger, each one a way of limiting the damage from a decision that turns out to be wrong. What none of them supply is a reason to believe the decision is right before it is made. That requires knowing how the building behaves, not just how it is connected, and behaviour is where the semantic model stops and physics, simulation and learned dynamics begin. The same discipline carries over, because a simulation that cannot say which of its inputs were assumed is no more trustworthy than a model that cannot say where its facts came from.

In future articles, we'll explore how continuous digital twins, fault detection, and predictive maintenance build on resolved identity, validated data, a semantic knowledge graph, a disciplined hybrid architecture, and bounded operational authority to create truly intelligent buildings.