AI and Software

Neil D. Lawrence

TBC, Central London

Opening: Software Advantage When Code Gets Cheap

Today’s Arc

  • What AI is — and what it is not
  • How AI is reshaping SaaS
  • The judgment layer — and what we are building at Trent

Part 1: What AI Is (and Isn’t) for Software Businesses

Information and Embodiment

Claude Shannon

Embodiment Factors

bits/min billions 2,000
billion
calculations/s 		~100 a billion
embodiment 20 minutes 5 billion years

Communication Bandwidth

  • Human communication: walking pace (2000 bits/minute)
  • Machine communication: light speed (billions of bits/second)
  • Our sharing walks, machine sharing …

For sale: baby shoes, never worn

Homo Atomicus

  • Where we have homo economicus the machine can step in.
  • Quantitative vs Qualitative gap
  • Homo atomicus is …
    • Not in A-level results
    • Not in hospital waiting lists
    • In the quality of human interaction

Implication for SaaS

  • AI commoditises production of many software artefacts
  • It does not create the judgment layer
  • Durable advantage sits where humans still own consequence

Part 2: How AI Is Reshaping SaaS

The Great AI Fallacy

The Attention Economy

  • Human intelligence is locked-in
  • This makes it a bottleneck

Herbert Simon on Information

What information consumes is rather obvious: it consumes the attention of its recipients. Hence a wealth of information creates a poverty of attention …

Simon (1971)

How Information Flows in Organisations

Hub-and-Spoke

Peer-to-Peer

Bezos memo to Amazon in 2002

The API Mandate

  • All teams will henceforth expose their data and functionality through service interfaces.
  • Teams must communicate with each other through these interfaces.

  • There will be no other form of inter-process communication allowed: no direct linking, no direct reads of another team’s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.

  • It doesn’t matter what technology they use.
  • All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.

Duality of Corporation and Information

  • What is less written about is corporate structure.
  • This information infrastructure is reflected in the corporation.
  • Two pizza teams with devolved autonomy.
  • Bound together through corporate culture.

Conway’s Law

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

Conway (n.d.)

Intellectual Debt in Software Organisations

Intellectual Debt

Technical Debt

  • Compare with technical debt.
  • Highlighted by Sculley et al. (2015).

Separation of Concerns

  • Decompose your complex problem/task into parts.
  • Each part manageable (e.g. by a small team)
  • Recompose to solve total problem

Addresses Complex Challenge

  • Highly successful approach to complex tasks.
  • Tuned to the human bandwidth limitation.
  • But the whole system still hard to understand.

Intellectual Debt

  • Technical debt is the inability to maintain your complex software system.
  • Intellectual debt is the inability to explain your software system.

Where Superficial Automation Fails

Superficial Automation

  • AI enables automation of surface-level tasks
  • Examples: Email writing, document summarization
  • Risk of losing deeper value in the process

Hidden Value

  • Email writing builds relationships
  • Documentation creates institutional memory
  • Human pauses enable reflection

The Automation Challenge

Good Process Drives Purpose

Atrophy and Cognitive Flattening

A Warning on Trust

The Horizon scandal

Part 3: How the Industry May Evolve

From Features to Interfaces

  • Generative AI as a human-analogue machine (HAM)
  • Information amplifier — not a substitute for judgment
  • Old org charts and pricing models become obsolete

Human Analogue Machine

HAM

HAM

Productivity Flywheel

Attention Reinvestment Cycle

AI cannot replace atomic human

Agentic Debt

  • Agentic AI could pay down technical and intellectual debt.
  • But it can create agentic debt: delegation without authority/authorship

Agentic Debt

  • Delegation of workflows without crisp boundaries.
  • Agentic debt is about unsafe or illegible delegation.

The Judgment Layer

Judgment Layer: Organisations as emulsions

  • Organisations are a stable mixture: automatable routines and irreducible human context.
  • In a working emulsion the two phases are inseparable — like oil and water held in suspension.
  • The judgment layer is the interface: what gets challenged, what gets waived, what triggers a halt.

What the judgment layer looks like

  • Approval exceptions: “we usually accept this class of risk without escalation.”
  • Escalation triggers: “this pattern always goes to a human, even if the rule says approve.”
  • Context-dependent waivers: policy says X, but the situation means Y is safer.

The Good Regulator Theorem

The Good Regulator theorem

  • Every good regulator of a system must be a model of that system.
  • The judgement layer is the regulator: it must still model what it oversees.
  • If the system outruns the model, regulation fails — even if the tools still run.

Why we minimise models

  • We minimise so that we can maintain a model.
  • Smallest decision-relevant representation that still supports oversight.
  • Prefer decomposable systems over opaque monoliths.

Against maximisation by default

  • “Use the largest model” is a default, not a design.
  • Capability without a modellable envelope is how debt compounds.
  • Ask: what is the least system that still earns regulated trust?

Strategic Priorities for Software Leaders

  • People first, not model first
  • Make the judgment layer explicit in product and ops
  • Keep human authority where consequence lives
  • Watch for intellectual and agentic debt

What will evolve in SaaS

  • Production of software gets cheaper; judgment gets scarcer
  • Seat-based pricing under pressure as agents multiply work
  • Moats shift toward data, distribution, trust, and domain judgment
  • Winners redesign the firm around a legible judgment layer

AI arrives workflow by workflow

A mistake markets make

  • Markets often treat AI as one centralised deployment
  • Value arrives workflow by workflow, with people who know the work
  • “Centralised deployment” is a supplier story; useful deployment is a demand story

Implication for software groups

  • Do not value a portfolio as if AI were a single centre-led programme
  • Underwrite the local judgment layer in each workflow
  • Support demand-side adoption — do not only sell a supplier story

Close: Trent and the Judgment Layer in Practice

Why did I start Trent?

The Trent Approach

  • Work with People who are close to Systems.
  • Address real pain points now.
  • Hold a view on the long term change (information systems).
  • Use the best agentic AI technology (proprietary).

What Trent Delivers

  • Make the judgment layer explicit — not buried in tacit ops
  • Preserve authority of the AI-augmented engineer
  • Delegate with evidence, time bounds, and recovery paths
  • Pay down agentic debt instead of accumulating it

Takeaways

  • AI commoditises many software artefacts — not the judgment layer
  • Advantage sits in attention, trust, and legible human authority
  • Intellectual and agentic debt are balance-sheet risks
  • Build so judgment remains structurally authoritative — as we are at Trent

Questions

  • Where is your judgment layer today — and is it legible?
  • Where is agentic debt already accruing in the product?
  • How does your pricing survive when agents do the work?

Thanks!

  • company: Trent AI

  • book: The Atomic Human

  • twitter: @lawrennd

  • The Atomic Human pages atomic human, the 13 , baby shoes 368, topography, information 34-9, 43-8, 57, 62, 104, 115-16, 127, 140, 192, 196, 199, 291, 334, 354-5, anthropomorphization (‘anthrox’) 30-31, 90-91, 93-4, 100, 132, 148, 153, 163, 216-17, 239, 276, 326, 342, intellectual debt 84, 85, 349, 365, separation of concerns 84-85, 103, 109, 199, 284, 371, intellectual debt 84-85, 349, 365, 376, Horizon scandal 371, ignorance: HAMs 347, test pilot 163-8, 189, 190, 192-3, 196, 197, 200, 211, 245, human-analogue machine (HAMs) 343-347, 359-359, 365-368.

  • A mistake markets make when they value software groups

  • blog: http://inverseprobability.com

References

Conway, M.E., n.d. How do committees invent? Datamation 14, 28–31.
Lawrence, N.D., 2024. The atomic human: Understanding ourselves in the age of AI. Allen Lane.
Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., Dennison, D., 2015. Hidden technical debt in machine learning systems, in: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (Eds.), Advances in Neural Information Processing Systems 28. Curran Associates, Inc., pp. 2503–2511.
Simon, H.A., 1971. Designing organizations for an information-rich world. Johns Hopkins University Press, Baltimore, MD.