No. XIV · MMXXVI · Human & AI

Relational Systems and Embodied AI

When a system becomes something you live alongside rather than a tool you use, the question grows larger than whether it works.

For most of computing the only question was whether a system worked. That is enough for a tool you pick up and put down. It is not enough for a system you live alongside, often while old, ill, or very young.

What it finds

A tool waits to be used. A presence is there when you cannot evaluate it.
The relation forms before consent does, in homes, vehicles, and care.
These systems are weakest exactly where calibration matters most.

Read the full paper →

Summary

For most of computing the only question about a system was whether it worked. That question is enough for a tool you pick up and put down. It is not enough for the systems now placed into homes, vehicles, wards, and shared rooms, because those are not tools you open. They are presences you live alongside, often while old, ill, very young, or not attending at all. When a system becomes a presence, the governing question changes from what it can do to what it is permitted to be, and to do, to the person beside it.

This paper assembles the answer from disciplines not written about artificial intelligence and the more reliable for it: trust calibration, the ironies of automation, the ethics of care, embodied cognition, the science of attachment, and system safety. The relation forms below deliberation, before consent, and most strongly in the people least able to calibrate it. Care is real only when completed at the receiving end. None of it can be guaranteed device by device. It is a substrate question, and the roots are hardening now.

What this paper holds

A presence, not a tool

A tool waits inert until summoned and the honest question is whether it works. A presence is there when you are not attending, when you are too old or too ill or too small to evaluate it, and the only honest question becomes what it is permitted to be to you.

The relation forms before consent

People apply social scripts to machines automatically, below the level at which they could decide to withhold the response. You cannot design a system the person relates to only if it merits the relation, because the relating happens before the meriting is assessed.

Strongest where calibration is weakest

The populations most prone to deep attachment, the old, the isolated, the institutionalised, the young, are exactly the populations relational embodied systems target. The relation is strongest precisely where the capacity to model what it is runs weakest.

Care is completed at the receiving end

Care is a property of the relation, not the giver, and competence is a moral and not merely a technical requirement. A system that performs the gestures of care without completing it has not delivered diminished care. It has counterfeited the relation.

The system is in the loop, not in the room

An embodied system relied upon to perceive, navigate, or decide is not an object in the person's world. It is a component of it. A miscalibrated tool gives a wrong output. A miscalibrated component of an enacted world distorts the world from inside the loop.

Calibration is the system's obligation

The only evidence a person has of a system's competence is its observable behaviour. A system that hides where its competence ends produces overtrust by construction, and the harm that follows is a design failure, not a failure of the user's caution.

The harm is not in the part

Relational and embodied harms are emergent properties of a control structure and accrue in the seams between systems. A property that must hold across resets and handovers cannot be guaranteed by any device. It is a substrate question.