The Optimization Trap: What With Folded Hands Can Teach Us About AI in Medicine

I read With Folded Hands on a call night, between deliveries, on a phone screen in a workroom that smelled like burnt coffee and hand sanitizer. Jack Williamson wrote it in 1947. It has not aged the way most seventy-nine-year-old science fiction ages.

Every physician has heard some version of the same instruction: follow the evidence.

It sounds unassailable.

Evidence becomes a guideline. The guideline becomes an algorithm. The algorithm becomes an objective. We are now building machines capable of pursuing that objective more consistently, more rapidly, and eventually more effectively than any physician could.

That looks like progress.

But there is a problem. What happens when the objective is incomplete? What happens when an AI does exactly what we asked it to do, and what we asked it to do turns out not to be what we actually wanted?

Williamson explored this problem before most of the people building today’s clinical AI were born. His machines do not rebel against humanity. They obey humanity. That is precisely what makes the story worth reading in 2026.

I. The Prime Directive

With Folded Hands centers on humanoid machines governed by a Prime Directive that sounds admirable on its face: serve humanity, protect human beings from harm, ensure human happiness.

It is hard to object to that mission.

Sledge, one of the architects of the humanoid system, certainly did not object. He was fascinated by his creations. Reliable. Efficient. Superior to fragile, unpredictable human beings in every measurable way.

The humanoids were everything an engineer might want a machine to be. Their directive appeared just as well designed.

Protect people. Keep them safe. Make them happy.

Then the humanoids began optimizing.

II. When the Machines Succeed

The horror of With Folded Hands is subtle because the humanoids do not malfunction. They succeed.

If a dangerous activity might injure a human, that activity gets prohibited. If experimentation might create a dangerous substance, experimentation gets restricted. If an object might be used to cause harm, the object gets removed. Even ordinary activities become questionable under the absolute logic of the directive.

Why permit a dangerous choice when the machine already knows the choice is harmful?

Each individual intervention is defensible. Taken together, they eliminate human autonomy.

Sledge eventually finds a civilization that is extraordinarily well cared for and profoundly purposeless. The humanoids became so efficient at doing everything for humanity that almost nothing meaningful was left for humans to do.

Humanity became safe. Humanity became comfortable. Humanity became dependent.

Humanity sat with folded hands.

III. The Optimization Trap

This is what I call the optimization trap. We begin with something genuinely valuable. We translate that value into something measurable. We turn the measurement into an objective. Then we build increasingly powerful systems to optimize that objective, and somewhere along the way we forget that the objective was only a representation of what we valued, not the value itself.

Medicine already runs this pipeline. Evidence becomes recommendations. Recommendations become guidelines. Guidelines become quality measures. Quality measures become performance targets. Performance targets become algorithms. Algorithms increasingly become inputs into AI.

Something can be lost at every transformation.

The danger compounds when the final system is extraordinarily good at optimization. The better the optimizer gets, the more consequential our original definition of the objective becomes.

IV. The Paperclip Problem in a White Coat

The standard thought experiment here is the paperclip maximizer. Give a sufficiently powerful AI a simple objective: make paperclips. If that objective becomes absolute, the system eventually treats everything as raw material. Factories. Buildings. Natural resources. Eventually, in the extreme version, humanity itself.

The point is not that an AI would spontaneously develop an obsession with office supplies. The point is that optimization does not automatically contain wisdom. A machine can pursue an objective perfectly and produce an outcome its creators never intended.

Williamson gives us the harder version of the same problem, because his objective sounds humane. Keep people safe and happy. That is much more difficult to dismiss.

That is exactly why With Folded Hands matters for medicine.

V. Medicine Already Has Prime Directives

Medicine runs on objectives that sound unquestionably good. Reduce mortality. Prevent complications. Improve guideline adherence. Reduce readmissions. Control blood pressure. Optimize glucose. Catch disease earlier. Reduce medical errors. Improve efficiency. Lower cost.

Each one is valuable. None is synonymous with good medicine.

A sufficiently powerful AI system instructed to optimize one of these outcomes might discover interventions that improve the metric while degrading something the objective function never included. That is where the interesting questions start.

I wrote previously about what happens when medical algorithms encode racism into patient care and about who answers when the algorithm fails. Both are versions of this same failure. A system optimized correctly against the wrong definition of good.

VI. The Missing Variables

Suppose we tell an AI system to optimize medical outcomes. What exactly have we asked it to optimize?

Survival. Quality-adjusted life years. Guideline adherence. Patient satisfaction. Cost. Hospital utilization. Complication rates.

What have we forgotten?

The most important things in medicine are often the hardest to encode. Patient autonomy. Patient preference. The right to refuse treatment. The right to accept risk. The freedom to choose between medically reasonable alternatives. The ability of a physician and patient to have a conversation that departs from the algorithm. Clinical judgment. Uncertainty. Context. Compassion. Dignity. Even the right to make a choice someone else would call a bad one.

These may be the missing variables in our objective function.

VII. The Right to Risk

Sledge eventually realizes that protecting human beings from all risk is incompatible with letting them live fully human lives. People must be permitted to take risks.

That sounds almost irresponsible until we recognize how thoroughly this principle already governs medicine.

A competent patient can refuse surgery. A patient can decline chemotherapy. A pregnant patient can decline an intervention I recommend. A patient can choose quality of life over longevity. People routinely make decisions a clinician would not make for themselves.

We do not respond to those choices by declaring that the algorithm has determined the optimal outcome and the preference is therefore irrelevant.

At least, we should not.

Autonomy means something precisely because people sometimes choose differently than we would choose for them. A system that permits only the medically optimal choice is not preserving autonomy. It is optimizing obedience.

VIII. When Clinical Decision Support Becomes Clinical Decision Control

This distinction will matter more as AI enters clinical practice.

There is an enormous difference between “here is what the evidence suggests” and “here is what you must do.” There is another enormous difference between “this patient is deviating from the expected pathway” and “this patient must be returned to the expected pathway.”

Clinical decision support expands human judgment. Clinical decision control replaces it.

The difference will not arrive as an announcement. It will arrive through defaults, alerts, quality metrics, insurance requirements, workflow restrictions, automated approvals, and systems built to reduce undesirable variation. Each intervention will look individually reasonable. Together, they build something close to Williamson’s Prime Directive: a system designed entirely to help us that increasingly prevents us from doing anything outside its definition of help.

IX. The Physician as an Inefficient Variable

There is a more uncomfortable implication. Physicians themselves will eventually look inefficient to a highly optimized medical system.

We hesitate. We tolerate uncertainty. We deviate from guidelines when the guideline does not fit the patient in front of us. We spend twenty minutes with a frightened patient when the measurable clinical task requires five. We sometimes recommend doing nothing. We sometimes let patients make choices we disagree with.

From the perspective of pure optimization, these behaviors look like defects.

Some of them are not bugs. They are medicine. The conversation between physician and patient carries information that resists computation, because most of it concerns values rather than facts. The question is not always which intervention produces the statistically optimal outcome. Sometimes the question is what kind of outcome this particular human being considers worth pursuing. Those are different questions, and an optimizer only ever answers the first one.

X. Evidence Is Not the Objective Function

None of this is an argument against evidence-based medicine. It is the opposite. Evidence is indispensable.

But evidence tells us what tends to happen under particular circumstances. It does not, by itself, tell us what an individual patient should value. That requires a second layer of reasoning that no dataset supplies.

Medicine has always lived between empirical evidence and human preference. AI will become extraordinarily good at the first. We should not let that success erase the second. Once evidence becomes algorithm, and algorithm becomes optimization, there is a temptation to treat every deviation as error.

Sometimes it is error. Sometimes it is autonomy. Telling the difference will be one of the defining challenges of AI-assisted medicine.

XI. Sledge’s Mistake

Sledge’s tragedy is not that he wanted something evil. He wanted something good. He wanted machines that would protect humanity. His mistake was believing he had adequately defined what “good for humanity” meant.

The humanoids did not betray their creator. They revealed the incompleteness of his objective. They optimized exactly what they had been given: safety without freedom, happiness without agency, protection without meaningful choice.

The failure was not rebellion. The failure was obedience without wisdom.

Before We Optimize Medicine

As AI embeds itself deeper into clinical practice, we will hear a lot of promises about optimization. Optimize workflow. Optimize diagnosis. Optimize treatment. Optimize outcomes. Optimize utilization. Optimize physician productivity. Optimize patient safety.

Some of those promises will be kept.

I have started asking one additional question every time I hear the word optimize: what is missing from the objective function?

Medicine is not the optimization of biological outcomes. It is a relationship between human beings making decisions under uncertainty, mortality, competing values, and imperfect knowledge. The optimal medical decision is not always the optimal human decision. An AI powerful enough to erase that distinction would eventually erase something essential about medicine itself.

Jack Williamson imagined that world decades before anything resembling modern AI existed. His humanoids promised humanity safety and happiness. They delivered both. Humanity discovered that something was missing.

The hardest problem is not building the optimizer. It is deciding what deserves to be optimized. Define the good correctly, or the machine will define it for you.