Viable Systems, Judgment, and AI Safety

AI safety is becoming an organisational problem

• Alignment framing: values, hallucinations, constraints — yesterday’s questions.
• Today: AI systems execute real workflows inside real organisations.
• The right question: who has the authority to decide when the answer matters?

What organisations already know

• Viable Systems Model (Stafford Beer, 1972): authority down, signals up.
• Beer called this attenuation — not less control, better control.

• “Accounting is the numbers. Accountability is the human authority and the judgment.”

• AI excels at accounting. Accountability is different — someone must still own the decision.

The judgment layer

• Good Regulator Theorem (Conant & Ashby, 1970): every good regulator must contain a model of what it regulates.
• Humans model not just the software but the organisation: priorities, risk, history, people.
• Automating without preserving judgment makes it invisible, not absent — agentic debt.
• The intervention: make the judgment layer explicit; deliver it to the AI-augmented engineer.
• The more autonomous AI becomes, the more valuable this judgment becomes — not less.

We need the judgment layer separated, the authority of the AI-augmented engineer preserved, and the system built to deliver both.