
Deploying Machine Learning: Intellectual Debt and AutoAI

Virtual Advances in Data Science Seminar, Manchester

Artificial vs Natural Systems

• Consider natural intelligence, or natural systems
• Contrast between an artificial system and an natural system.
• The key difference between the two is that artificial systems are designed whereas natural systems are evolved.

Natural Systems are Evolved

Survival of the fittest

?

Natural Systems are Evolved

Survival of the fittest

Herbet Spencer, 1864

Natural Systems are Evolved

Non-survival of the non-fit

Mistake we Make

• Equate fitness for objective function.
• Assume static environment and known objective.

Technical Debt

• Compare with technical debt.
• Highlighted by Sculley et al. (2015).

Intellectual Debt

• Technical debt is the inability to maintain your complex software system.

• Intellectual debt is the inability to explain your software system.

SafeBoda

With road accidents set to match HIV/AIDS as the highest cause of death in low/middle income countries by 2030, SafeBoda’s aim is to modernise informal transportation and ensure safe access to mobility.

FIT Models to FIT Systems

• Focus in machine learning has been on FAcT learning.
• Fairness, accountability and Transparency in individual models.
• But individual models aren’t the problem.
• Fariness, interpetability and transparency required for whole system.

Milan

1. A general-purpose stream algebra that encodes relationships between data streams (the Milan Intermediate Language or Milan IL)

2. A Scala library for building programs in that algebra.

3. A compiler that takes programs expressed in Milan IL and produces a Flink application that executes the program.

AutoAI: FIT Models to FIT Systems

• Streaming algebra provides a tube map for data.
• Can now deploy classical statistical and software verification techniques.
• e.g. outlier detection, verification that prohibited characteristics are not used
• Also explore ML techniques for fairness, interpretability, transparency for system instead of model.

Conclusion

• AI is fundamentally ML System Design
• We are not ready to deploy automation in uncontrolled environments.
• Until we can monitoring and update will be key.

References

Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J.-F., Dennison, D., 2015. Hidden technical debt in machine learning systems, in: Cortes, C., Lawrence, N.D., Lee, D.D., Sugiyama, M., Garnett, R. (Eds.), Advances in Neural Information Processing Systems 28. Curran Associates, Inc., pp. 2503–2511.