Artificial intelligence is transforming how people interact with digital technology (Chignell, Hancock, and Loewenthal 1989; Zhang, Wang, and Yi 2025). For decades, most software followed predictable patterns: people acted and systems responded according to clearly defined rules. If you understood the structure of the interface, you could usually anticipate what would happen next.
AI changes that relationship. Instead of executing fixed procedures, many modern systems generate responses dynamically, drawing on patterns learned from large datasets and producing outputs that adapt to each situation. This flexibility lets systems do things that once seemed impossible, but it also adds a new layer of uncertainty. People often cannot predict how an AI system will behave, and the reasoning behind its responses stays hidden. Interaction begins to feel like a conversation with an unfamiliar mind, and in that environment cognitive distance can expand rapidly.
9.1 From Deterministic to Probabilistic Systems
Traditional software follows explicit rules
For most of computing history, software was deterministic. Developers defined conditions and actions, and a given input triggered a specific instruction. Even complex systems followed logic that could be traced through code, so people who understood the rules could form fairly accurate mental models. A spreadsheet applies formulas exactly as written; a search filter shows results that match the chosen criteria. These interactions take learning, but they remain conceptually stable.
Machine learning systems operate through statistical patterns
Systems built on machine learning work differently. Rather than relying solely on explicit rules, they learn patterns from large sets of examples, and their outputs reflect statistical relationships rather than fixed instructions. Language models, image recognition and recommendation engines all work this way. The neural network methods behind them, developed over decades by researchers including Geoffrey Hinton, Yoshua Bengio and Yann LeCun, are remarkably powerful (LeCun, Bengio, and Hinton 2015). They are also hard to interpret. An output may depend on millions or billions of learned parameters, and from an everyday user’s perspective the system becomes a black box.
Predictability becomes harder to maintain
Because AI systems generate responses dynamically, their outcomes are not always consistent. Similar inputs can produce different results. A recommendation system’s suggestions shift as it learns from new data. A language model may interpret the same question differently depending on context. These variations are natural consequences of probabilistic systems, but they disrupt the mental models people try to build. People expect digital systems to behave consistently, and when behavior shifts unexpectedly they lose confidence in their understanding. Table 9.1 summarizes why probabilistic behavior feels less conceptually stable.
| Deterministic systems | Probabilistic systems | |
|---|---|---|
| Response generation | Executes rules defined in advance | Predicts outputs from patterns learned from data |
| Repeatability | The same input usually gives the same output | Similar inputs can give different outputs |
| Visibility of reasons | Logic can be traced through procedural steps | Reasoning is distributed across many parameters |
| User strategy | Learn the rules and apply them | Keep testing assumptions and verifying results |
| Basis of trust | Consistency and predictability | Transparency, explanation and safeguards |
9.2 Anthropomorphism and the Illusion of Understanding
Humans attribute intention to intelligent behavior
When systems produce language or make recommendations that seem thoughtful, people tend to read that behavior as evidence of understanding. People readily treat computers as social actors, applying to them the expectations they hold of other people (Reeves and Nass 1996). The tendency is old: in the 1960s, users of Joseph Weizenbaum’s simple ELIZA program confided in it as if it understood them, to its creator’s dismay (Weizenbaum 1966). When a conversational system answers fluently today, people may assume it grasps the meaning behind its words, when it is producing statistically likely text. The appearance of understanding emerges from modeling, not from reasoning in the human sense.
Conversational interfaces strengthen the illusion
Modern AI systems often communicate through conversation. People ask questions, the system answers, and the exchange unfolds in a natural linguistic format. Conversational assistants now support everything from writing to programming, and the format makes them accessible and engaging. It also strengthens the impression that the system understands the conversation as a person would, and people may take confident language as evidence of reliability. When the system then produces incorrect or inconsistent information, the mismatch between expectation and reality becomes visible. Cognitive distance appears in the gap between perceived intelligence and actual mechanism, illustrated in Figure 9.1.
Misinterpretation creates fragile trust
Trust in AI systems often forms quickly. Early interactions produce useful responses, and people begin relying on the system for increasingly complex tasks. That trust may rest on an incomplete model. People may not know the system’s limitations, or may assume it verifies information or applies logic when generating responses. Research on automation has long warned that trust which outruns a system’s actual reliability leads to misuse, and trust that falls below it leads to disuse (Lee and See 2004). When the system produces an inaccurate result, miscalibrated trust can collapse suddenly. The person realizes their understanding was incomplete, and that moment of disillusionment reveals the cognitive distance that was present all along.
9.3 Opacity and the Challenge of Explanation
Many AI systems cannot easily explain their decisions
A defining challenge of modern AI is the difficulty of explaining how particular outputs are produced. Deep neural networks work through layered mathematical transformations that resist direct interpretation, and even the engineers who build them cannot always identify the factors behind a specific prediction. That opacity is a design problem. People often want to know why a system produced a particular result, and if it cannot offer a meaningful explanation they must accept the outcome without understanding it.
Explainable AI as an emerging field
Researchers have begun developing methods to make AI behavior more interpretable, a field usually called explainable AI (Doshi-Velez and Kim 2017). The goal is not to expose every internal parameter, but to provide explanations that help people interpret what the system did. Image classifiers can highlight the regions of a picture that influenced a decision (Selvaraju et al. 2017); other methods identify which input features most affected a particular prediction (Ribeiro, Singh, and Guestrin 2016); recommendation systems can indicate which of a user’s behaviors led to a suggestion. These explanations provide conceptual cues that help people build models of how the system works, and work on interpretability increasingly measures whether they actually improve human understanding rather than assuming they do (Poursabzi-Sangdeh et al. 2021). Figure 9.2 outlines a practical progression from raw output to an explanation people can act on.
- 1. Outcome statement. State clearly what the system produced or decided.
- 2. Primary reason. Name the strongest contributing factor in user-facing terms.
- 3. Confidence qualifier. Indicate certainty, limits and known conditions of uncertainty.
- 4. Verification guidance. Suggest what the person should check before acting.
- 5. Next step. Offer a concrete way to refine the input or request an alternative.
Explanations must match human reasoning
Providing explanations does not guarantee understanding. A technical description may accurately reflect a model’s workings and still be impossible for most people to interpret. Effective explanations connect system behavior to concepts people recognize. A recommendation system that says a suggestion appears because you watched a particular film, or bought a similar product, reflects a pattern anyone can follow. It translates complex statistical relationships into familiar reasoning, and that translation is what reduces cognitive distance. Studies of end users have found that explanations which are sound and complete improve the accuracy of people’s mental models, while explanations that oversimplify can leave them with confident but wrong ideas (Kulesza et al. 2013).
9.4 Designing AI Systems That Humans Can Understand
Communicate system capabilities clearly
One of the simplest ways to reduce cognitive distance in AI systems is to state openly what they can and cannot do. Guidelines for human–AI interaction begin exactly here: make clear what the system can do, and how well it can do it (Amershi et al. 2019). An AI writing assistant might explain that it generates text from patterns in its training data and can produce inaccurate information. That transparency encourages people to treat the system as a helpful tool rather than an infallible authority, and clear expectations support more stable mental models.
Provide visible reasoning cues
Interfaces can also reveal aspects of system reasoning through small design choices. Search engines show suggested queries related to the user’s input, hinting at how the request was interpreted. Recommendation platforms label rows “because you watched” or “similar to items you viewed”. These cues expose the logic guiding the system’s behavior, and people gradually develop a sense of how it organizes information and generates results, even when the full technical details stay hidden.
Encourage critical engagement
AI systems should not encourage passive acceptance of their outputs. Designers can promote critical engagement by letting people question results, refine their inputs and explore alternatives. Conversational systems can invite clarification or additional context; recommendation systems can let people adjust the signals they are based on. These opportunities make the interaction collaborative rather than opaque.
9.5 The Future of Human–AI Interaction
Artificial intelligence brings remarkable capabilities into digital systems, and it challenges long-standing assumptions about predictability and understanding. When behavior emerges from statistical patterns rather than explicit rules, people have to construct new kinds of mental models. Designers therefore carry an important responsibility: to keep AI-powered systems interpretable enough for people to reason about. Without that effort, cognitive distance will keep expanding, and people will rely on systems whose logic they cannot grasp. The conceptual transparency dimension of the Cognitive Distance Index (Section 6.5) offers one way to measure whether explanation is actually working. Reducing that distance may become one of the defining challenges of Human-Computer Interaction. The next chapter widens the lens to the institutions whose decisions increasingly reach people through such systems.