Machine Learning and the Physical World
[edit]
Abstract
Machine learning is a data driven endeavour, but real world systems are physical and mechanistic. In this talk we will review approaches to integrating machine learning with real world systems. Our focus will be on emulation (otherwise known as surrogate modeling).
The Centrifugal Governor
The centrifugal governor was made famous by Boulton and Watt when it was deployed in the steam engine. Studying stability in the governor is the main subject of James Clerk Maxwell's paper on the theoretical analysis of governors (Maxwell 1867), a founding paper of control theory. In this spirit, Wiener used the name cybernetics to describe the field of control and communication in animals and the machine (Wiener 1948). Cybernetics is the Greek word for governor, which comes from the latin for helmsman.
A governor is one of the simplest artificial intelligence systems. It senses the speed of an engine, and acts to change the position of the valve on the engine to slow it down.
Although it's a mechanical system a governor can be seen as automating a role that a human would have traditionally played. It is an early example of artificial intelligence.
The centrifugal governor has several parameters, the weight of the balls used, the length of the linkages and the limits on the balls movement.
Two principle differences exist between the centrifugal governor and artificial intelligence systems of today.
 The centrifugal governor is a physical system and it is an integral part of a wider physical system that it regulates (the engine).
 The parameters of the governor were set by hand, our modern artificial intelligence systems have their parameters set by data.
Machine learning allows us to extract knowledge from data to form a prediction.
$$ \text{data} + \text{model} \xrightarrow{\text{compute}} \text{prediction}$$
A machine learning prediction is made by combining a model with data to form the prediction. The manner in which this is done gives us the machine learning algorithm.
Machine learning models are mathematical models which make weak assumptions about data, e.g. smoothness assumptions. By combining these assumptions with the data we observe we can interpolate between data points or, occasionally, extrapolate into the future.
Machine learning is a technology which strongly overlaps with the methodology of statistics. From a historical/philosophical view point, machine learning differs from statistics in that the focus in the machine learning community has been primarily on accuracy of prediction, whereas the focus in statistics is typically on the interpretability of a model and/or validating a hypothesis through data collection.
The rapid increase in the availability of compute and data has led to the increased prominence of machine learning. This prominence is surfacing in two different, but overlapping domains: data science and artificial intelligence.
Artificial Intelligence and Data Science
Artificial intelligence has the objective of endowing computers with humanlike intelligent capabilities. For example, understanding an image (computer vision) or the contents of some speech (speech recognition), the meaning of a sentence (natural language processing) or the translation of a sentence (machine translation).
Supervised Learning for AI
The machine learning approach to artificial intelligence is to collect and annotate a large data set from humans. The problem is characterized by input data (e.g. a particular image) and a label (e.g. is there a car in the image yes/no). The machine learning algorithm fits a mathematical function (I call this the prediction function) to map from the input image to the label. The parameters of the prediction function are set by minimizing an error between the function’s predictions and the true data. This mathematical function that encapsulates this error is known as the objective function.
This approach to machine learning is known as supervised learning. Various approaches to supervised learning use different prediction functions, objective functions or different optimization algorithms to fit them.
For example, deep learning makes use of neural networks to form the predictions. A neural network is a particular type of mathematical function that allows the algorithm designer to introduce invariances into the function.
An invariance is an important way of including prior understanding in a machine learning model. For example, in an image, a car is still a car regardless of whether it’s in the upper left or lower right corner of the image. This is known as translation invariance. A neural network encodes translation invariance in convolutional layers. Convolutional neural networks are widely used in image recognition tasks.
An alternative structure is known as a recurrent neural network (RNN). RNNs neural networks encode temporal structure. They use auto regressive connections in their hidden layers, they can be seen as time series models which have nonlinear autoregressive basis functions. They are widely used in speech recognition and machine translation.
Machine learning has been deployed in Speech Recognition (e.g. Alexa, deep neural networks, convolutional neural networks for speech recognition), in computer vision (e.g. Amazon Go, convolutional neural networks for person recognition and pose detection).
The field of data science is related to AI, but philosophically different. It arises because we are increasingly creating large amounts of data through happenstance rather than active collection. In the modern era data is laid down by almost all our activities. The objective of data science is to extract insights from this data.
Classically, in the field of statistics, data analysis proceeds by assuming that the question (or scientific hypothesis) comes before the data is created. E.g., if I want to determine the effectiveness of a particular drug I perform a design for my data collection. I use foundational approaches such as randomization to account for confounders. This made a lot of sense in an era where data had to be actively collected. The reduction in cost of data collection and storage now means that many data sets are available which weren’t collected with a particular question in mind. This is a challenge because bias in the way data was acquired can corrupt the insights we derive. We can perform randomized control trials (or A/B tests) to verify our conclusions, but the opportunity is to use data science techniques to better guide our question selection or even answer a question without the expense of a full randomized control trial (referred to as A/B testing in modern internet parlance).
 There is a gap between the world of data science and AI.
 The mapping of the virtual onto the physical world.
 E.g. Causal understanding.
Machine Learning in Supply Chain
Supply chain is a large scale automated decision making network. Our aim is to make decisions not only based on our models of customer behavior (as observed through data), but also by accounting for the structure of our fulfilment center, and delivery network.
Many of the most important questions in supply chain take the form of counterfactuals. E.g. “What would happen if we opened a manufacturing facility in Cambridge?” A counter factual is a question that implies a mechanistic understanding of a system. It goes beyond simple smoothness assumptions or translation invariants. It requires a physical, or mechanistic understanding of the supply chain network. For this reason the type of models we deploy in supply chain often involve simulations or more mechanistic understanding of the network.
In supply chain Machine Learning alone is not enough, we need to bridge between models that contain real mechanisms and models that are entirely data driven.
This is challenging, because as we introduce more mechanism to the models we use, it becomes harder to develop efficient algorithms to match those models to data.
Uncertainty Quantification
Uncertainty quantification (UQ) is the science of quantitative characterization and reduction of uncertainties in both computational and real world applications. It tries to determine how likely certain outcomes are if some aspects of the system are not exactly known.
We will to illustrate different concepts of Uncertainty Quantification (UQ) and the role that Gaussian processes play in this field. Based on a simple simulator of a car moving between a valley and a mountain, we are going to illustrate the following concepts:
Systems emulation. Many real world decisions are based on simulations that can be computationally very demanding. We will show how simulators can be replaced by emulators: Gaussian process models fitted on a few simulations that can be used to replace the simulator. Emulators are cheap to compute, fast to run, and always provide ways to quantify the uncertainty of how precise they are compared the original simulator.
Emulators in optimization problems. We will show how emulators can be used to optimize blackbox functions that are expensive to evaluate. This field is also called Bayesian Optimization and has gained an increasing relevance in machine learning as emulators can be used to optimize computer simulations (and machine learning algorithms) quite efficiently.
Multifidelity emulation methods. In many scenarios we have simulators of different quality about the same measure of interest. In these cases the goal is to merge all sources of information under the same model so the final emulator is cheaper and more accurate than an emulator fitted only using data from the most accurate and expensive simulator.
Mountain Car Simulator
To illustrate the above mentioned concepts we we use the mountain car simulator. This simulator is widely used in machine learning to test reinforcement learning algorithms. The goal is to define a control policy on a car whose objective is to climb a mountain. Graphically, the problem looks as follows:
The goal is to define a sequence of actions (push the car right or left with certain intensity) to make the car reach the flag after a number T of time steps.
At each time step t, the car is characterized by a vector $\inputVector_{t} = (p_t,v_t)$ of states which are respectively the the position and velocity of the car at time t. For a sequence of states (an episode), the dynamics of the car is given by
$$\inputVector_{t+1} = \mappingFunction(\inputVector_{t},\textbf{u}_{t})$$
where u_{t} is the value of an action force, which in this example corresponds to push car to the left (negative value) or to the right (positive value). The actions across a full episode are represented in a policy $\textbf{u}_{t} = \pi(\inputVector_{t},\theta)$ that acts according to the current state of the car and some parameters θ. In the following examples we will assume that the policy is linear which allows us to write $\pi(\inputVector_{t},\theta)$ as
$$\pi(\inputVector,\theta)= \theta_0 + \theta_p p + \theta_vv.$$
For t = 1, …, T now given some initial state $\inputVector_{0}$ and some some values of each u_{t}, we can simulate the full dynamics of the car for a full episode using Gym. The values of u_{t} are fully determined by the parameters of the linear controller.
After each episode of length T is complete, a reward function R_{T}(θ) is computed. In the mountain car example the reward is computed as 100 for reaching the target of the hill on the right hand side, minus the squared sum of actions (a real negative to push to the left and a real positive to push to the right) from start to goal. Note that our reward depend on θ as we make it dependent on the parameters of the linear controller.
Emulate the Mountain Car
import gym
env = gym.make('MountainCarContinuousv0')
Our goal in this section is to find the parameters θ of the linear controller such that
θ^{*} = argmax_{θ}R_{T}(θ).
In this section, we directly use Bayesian optimization to solve this problem. We will use GPyOpt so we first define the objective function:
import mountain_car as mc
import GPyOpt
obj_func = lambda x: mc.run_simulation(env, x)[0]
objective = GPyOpt.core.task.SingleObjective(obj_func)
For each set of parameter values of the linear controller we can run an episode of the simulator (that we fix to have a horizon of T = 500) to generate the reward. Using as input the parameters of the controller and as outputs the rewards we can build a Gaussian process emulator of the reward.
We start defining the input space, which is threedimensional:
##  We define the input space of the emulator
space= [{'name':'postion_parameter', 'type':'continuous', 'domain':(1.2, +1)},
{'name':'velocity_parameter', 'type':'continuous', 'domain':(1/0.07, +1/0.07)},
{'name':'constant', 'type':'continuous', 'domain':(1, +1)}]
design_space = GPyOpt.Design_space(space=space)
Now we initizialize a Gaussian process emulator.
model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)
In Bayesian optimization an acquisition function is used to balance exploration and exploitation to evaluate new locations close to the optimum of the objective. In this notebook we select the expected improvement (EI). For further details have a look to the review paper of Shahriari et al (2015).
aquisition_optimizer = GPyOpt.optimization.AcquisitionOptimizer(design_space)
acquisition = GPyOpt.acquisitions.AcquisitionEI(model, design_space, optimizer=aquisition_optimizer)
evaluator = GPyOpt.core.evaluators.Sequential(acquisition) # Collect points sequentially, no parallelization.
To initalize the model we start sampling some initial points (25) for the linear controler randomly.
from GPyOpt.experiment_design.random_design import RandomDesign
n_initial_points = 25
random_design = RandomDesign(design_space)
initial_design = random_design.get_samples(n_initial_points)
Before we start any optimization, lets have a look to the behavior of the car with the first of these initial points that we have selected randomly.
import numpy as np
random_controller = initial_design[0,:]
_, _, _, frames = mc.run_simulation(env, np.atleast_2d(random_controller), render=True)
anim=mc.animate_frames(frames, 'Random linear controller')
from IPython.core.display import HTML
As we can see the random linear controller does not manage to push the car to the top of the mountain. Now, let's optimize the regret using Bayesian optimization and the emulator for the reward. We try 50 new parameters chosen by the EI.
max_iter = 50
bo = GPyOpt.methods.ModularBayesianOptimization(model, design_space, objective, acquisition, evaluator, initial_design)
bo.run_optimization(max_iter = max_iter )
Now we visualize the result for the best controller that we have found with Bayesian optimization.
_, _, _, frames = mc.run_simulation(env, np.atleast_2d(bo.x_opt), render=True)
anim=mc.animate_frames(frames, 'Best controller after 50 iterations of Bayesian optimization')
he car can now make it to the top of the mountain! Emulating the reward function and using the EI helped as to find a linear controller that solves the problem.
Data Efficient Emulation
In the previous section we solved the mountain car problem by directly emulating the reward but no considerations about the dynamics $\inputVector_{t+1} = \mappingFunction(\inputVector_{t},\textbf{u}_{t})$ of the system were made. Note that we had to run 75 episodes of 500 steps each to solve the problem, which required to call the simulator 500 × 75 = 37500 times. In this section we will show how it is possible to reduce this number by building an emulator for f that can later be used to directly optimize the control.
The inputs of the model for the dynamics are the velocity, the position and the value of the control so create this space accordingly.
import gym
env = gym.make('MountainCarContinuousv0')
import GPyOpt
space_dynamics = [{'name':'position', 'type':'continuous', 'domain':[1.2, +0.6]},
{'name':'velocity', 'type':'continuous', 'domain':[0.07, +0.07]},
{'name':'action', 'type':'continuous', 'domain':[1, +1]}]
design_space_dynamics = GPyOpt.Design_space(space=space_dynamics)
The outputs are the velocity and the position. Indeed our model will capture the change in position and velocity on time. That is, we will model
Δv_{t + 1} = v_{t + 1} − v_{t}
Δx_{t + 1} = p_{t + 1} − p_{t}
with Gaussian processes with prior mean v_{t} and p_{t} respectively. As a covariance function, we use a Matern52. We need therefore two models to capture the full dynamics of the system.
position_model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)
velocity_model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)
Next, we sample some input parameters and use the simulator to compute the outputs. Note that in this case we are not running the full episodes, we are just using the simulator to compute $\inputVector_{t+1}$ given $\inputVector_{t}$ and u_{t}.
import numpy as np
from GPyOpt.experiment_design.random_design import RandomDesign
import mountain_car as mc
###  Random locations of the inputs
n_initial_points = 500
random_design_dynamics = RandomDesign(design_space_dynamics)
initial_design_dynamics = random_design_dynamics.get_samples(n_initial_points)
###  Simulation of the (normalized) outputs
y = np.zeros((initial_design_dynamics.shape[0], 2))
for i in range(initial_design_dynamics.shape[0]):
y[i, :] = mc.simulation(initial_design_dynamics[i, :])
# Normalize the data from the simulation
y_normalisation = np.std(y, axis=0)
y_normalised = y/y_normalisation
In general we might use much smarter strategies to design our emulation of the simulator. For example, we could use the variance of the predictive distributions of the models to collect points using uncertainty sampling, which will give us a better coverage of the space. For simplicity, we move ahead with the 500 randomly selected points.
Now that we have a data set, we can update the emulators for the location and the velocity.
position_model.updateModel(initial_design_dynamics, y[:, [0]], None, None)
velocity_model.updateModel(initial_design_dynamics, y[:, [1]], None, None)
We can now have a look to how the emulator and the simulator match. First, we show a contour plot of the car aceleration for each pair of can position and velocity. You can use the bar bellow to play with the values of the controler to compare the emulator and the simulator.
from IPython.html.widgets import interact
We can see how the emulator is doing a fairly good job approximating the simulator. On the edges, however, it struggles to captures the dynamics of the simulator.
Given some input parameters of the linear controlling, how do the dynamics of the emulator and simulator match? In the following figure we show the position and velocity of the car for the 500 time steps of an episode in which the parameters of the linear controller have been fixed beforehand. The value of the input control is also shown.
controller_gains = np.atleast_2d([0, .6, 1]) # change the valus of the linear controller to observe the trayectories.
We now make explicit use of the emulator, using it to replace the simulator and optimize the linear controller. Note that in this optimization, we don't need to query the simulator anymore as we can reproduce the full dynamics of an episode using the emulator. For illustrative purposes, in this example we fix the initial location of the car.
We define the objective reward function in terms of the simulator.
###  Optimize control parameters with emulator
car_initial_location = np.asarray([0.58912799, 0])
###  Reward objective function using the emulator
obj_func_emulator = lambda x: mc.run_emulation([position_model, velocity_model], x, car_initial_location)[0]
objective_emulator = GPyOpt.core.task.SingleObjective(obj_func_emulator)
And as before, we use Bayesian optimization to find the best possible linear controller.
###  Elements of the optimization that will use the multifidelity emulator
model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)
The design space is the three continuous variables that make up the linear controller.
space= [{'name':'linear_1', 'type':'continuous', 'domain':(1/1.2, +1)},
{'name':'linear_2', 'type':'continuous', 'domain':(1/0.07, +1/0.07)},
{'name':'constant', 'type':'continuous', 'domain':(1, +1)}]
design_space = GPyOpt.Design_space(space=space)
aquisition_optimizer = GPyOpt.optimization.AcquisitionOptimizer(design_space)
random_design = RandomDesign(design_space)
initial_design = random_design.get_samples(25)
We set the acquisition function to be expected improvement using GPyOpt
.
acquisition = GPyOpt.acquisitions.AcquisitionEI(model, design_space, optimizer=aquisition_optimizer)
evaluator = GPyOpt.core.evaluators.Sequential(acquisition)
bo_emulator = GPyOpt.methods.ModularBayesianOptimization(model, design_space, objective_emulator, acquisition, evaluator, initial_design)
bo_emulator.run_optimization(max_iter=50)
_, _, _, frames = mc.run_simulation(env, np.atleast_2d(bo_emulator.x_opt), render=True)
anim=mc.animate_frames(frames, 'Best controller using the emulator of the dynamics')
from IPython.core.display import HTML
And the problem is again solved, but in this case we have replaced the simulator of the car dynamics by a Gaussian process emulator that we learned by calling the simulator only 500 times. Compared to the 37500 calls that we needed when applying Bayesian optimization directly on the simulator this is a great gain.
In some scenarios we have simulators of the same environment that have different fidelities, that is that reflect with different level of accuracy the dynamics of the real world. Running simulations of the different fidelities also have a different cost: hight fidelity simulations are more expensive the cheaper ones. If we have access to these simulators we can combine high and low fidelity simulations under the same model.
So let's assume that we have two simulators of the mountain car dynamics, one of high fidelity (the one we have used) and another one of low fidelity. The traditional approach to this form of multifidelity emulation is to assume that
$$\mappingFunction_i\left(\inputVector\right) = \rho\mappingFunction_{i1}\left(\inputVector\right) + \delta_i\left(\inputVector \right)$$
where $\mappingFunction_{i1}\left(\inputVector\right)$ is a low fidelity simulation of the problem of interest and $\mappingFunction_i\left(\inputVector\right)$ is a higher fidelity simulation. The function $\delta_i\left(\inputVector \right)$ represents the difference between the lower and higher fidelity simulation, which is considered additive. The additive form of this covariance means that if $\mappingFunction_{0}\left(\inputVector\right)$ and $\left\{\delta_i\left(\inputVector \right)\right\}_{i=1}^m$ are all Gaussian processes, then the process over all fidelities of simuation will be a joint Gaussian process.
But with Deep Gaussian processes we can consider the form
$$\mappingFunction_i\left(\inputVector\right) = \mappingFunctionTwo_{i}\left(\mappingFunction_{i1}\left(\inputVector\right)\right) + \delta_i\left(\inputVector \right),$$
where the low fidelity representation is non linearly transformed by $\mappingFunctionTwo(\cdot)$ before use in the process. This is the approach taken in Perdikaris et al. (2017). But once we accept that these models can be composed, a highly flexible framework can emerge. A key point is that the data enters the model at different levels, and represents different aspects. For example these correspond to the two fidelities of the mountain car simulator.
We start by sampling both of them at 250 random input locations.
import gym
env = gym.make('MountainCarContinuousv0')
import GPyOpt
###  Collect points from low and high fidelity simulator  ###
space = GPyOpt.Design_space([
{'name':'position', 'type':'continuous', 'domain':(1.2, +1)},
{'name':'velocity', 'type':'continuous', 'domain':(0.07, +0.07)},
{'name':'action', 'type':'continuous', 'domain':(1, +1)}])
n_points = 250
random_design = GPyOpt.experiment_design.RandomDesign(space)
x_random = random_design.get_samples(n_points)
Next, we evaluate the high and low fidelity simualtors at those locations.
import numpy as np
import mountain_car as mc
d_position_hf = np.zeros((n_points, 1))
d_velocity_hf = np.zeros((n_points, 1))
d_position_lf = np.zeros((n_points, 1))
d_velocity_lf = np.zeros((n_points, 1))
#  Collect high fidelity points
for i in range(0, n_points):
d_position_hf[i], d_velocity_hf[i] = mc.simulation(x_random[i, :])
#  Collect low fidelity points
for i in range(0, n_points):
d_position_lf[i], d_velocity_lf[i] = mc.low_cost_simulation(x_random[i, :])
It is time to build the multifidelity model for both the position and the velocity.
As we did in the previous section we use the emulator to optimize the simulator. In this case we use the high fidelity output of the emulator.
###  Optimize controller parameters
obj_func = lambda x: mc.run_simulation(env, x)[0]
obj_func_emulator = lambda x: mc.run_emulation([position_model, velocity_model], x, car_initial_location)[0]
objective_multifidelity = GPyOpt.core.task.SingleObjective(obj_func)
And we optimize using Bayesian optimzation.
from GPyOpt.experiment_design.random_design import RandomDesign
model = GPyOpt.models.GPModel(optimize_restarts=5, verbose=False, exact_feval=True, ARD=True)
space= [{'name':'linear_1', 'type':'continuous', 'domain':(1/1.2, +1)},
{'name':'linear_2', 'type':'continuous', 'domain':(1/0.07, +1/0.07)},
{'name':'constant', 'type':'continuous', 'domain':(1, +1)}]
design_space = GPyOpt.Design_space(space=space)
aquisition_optimizer = GPyOpt.optimization.AcquisitionOptimizer(design_space)
n_initial_points = 25
random_design = RandomDesign(design_space)
initial_design = random_design.get_samples(n_initial_points)
acquisition = GPyOpt.acquisitions.AcquisitionEI(model, design_space, optimizer=aquisition_optimizer)
evaluator = GPyOpt.core.evaluators.Sequential(acquisition)
bo_multifidelity = GPyOpt.methods.ModularBayesianOptimization(model, design_space, objective_multifidelity, acquisition, evaluator, initial_design)
bo_multifidelity.run_optimization(max_iter=50)
_, _, _, frames = mc.run_simulation(env, np.atleast_2d(bo_multifidelity.x_opt), render=True)
anim=mc.animate_frames(frames, 'Best controller with multifidelity emulator')
from IPython.core.display import HTML
And problem solved! We see how the problem is also solved with 250 observations of the high fidelity simulator and 250 of the low fidelity simulator.
 Multifidelity emulation: build surrogate models when data is obtained from multiple information sources that have different fidelity and/or cost;
 Bayesian optimisation: optimise physical experiments and tune parameters of machine learning algorithms;
 Experimental design/Active learning: design the most informative experiments and perform active learning with machine learning models;
 Sensitivity analysis: analyse the influence of inputs on the outputs of a given system;
 Bayesian quadrature: efficiently compute the integrals of functions that are expensive to evaluate.
MxFusion
</tr> </table>MxFusion
Why another framework?
Key Requirements
Specialized inference methods + models, without requiring users to reimplement nor understand them every time. Leverage expert knowledge. Efficient inference, flexible framework. Existing frameworks either did one or the other: flexible, or efficient.
What does it look like?
Modelling
Inference
m = Model()
m.mu = Variable()
m.s = Variable(transformation=PositiveTransformation())
m.Y = Normal.define_variable(mean=m.mu, variance=m.s)
 Variable
 Distribution
Function
log_pdf
draw_samples
 Variational Inference
MCMC Sampling (soon) Built on MXNet Gluon (imperative code, not static graph)
infr = GradBasedInference(inference_algorithm=MAP(model=m, observed=[m.Y]))
infr.run(Y=data)
 Model + Inference together form building blocks.
 Just doing modular modeling with universal inference doesn't really scale, need specialized inference methods for specialized modelling objects like nonparametrics.
PILCO: A Modelbased Policy Search
Common reinforcement learning methods suffer from data inefficiency, which can be a issue in real world applications where gathering sufficiently large amounts of data pose economic issues and may be impossible. propose a modelbased policy search method known as PILCO in part to address this issue. PILCO uses a Gaussian process (GP) for learning the dynamics of the environment and optimizes a parametric policy function using the learned dynamics model.
We construct an implementation of PILCO using MXFusion. This implementation follows the main idea of PILCO and has a few enhancements in addition to the published implementation. The enhancements are as follows: * Use Monte Carlo integration instead of moment estimation. We approximate the expected reward using Monte Carlo integration instead of the proposed moment estimation approach. This removes the bias in the expected reward computation and enables a wide range of choices of kernels and policy functions. In the original work, only RBF and linear kernel and only linear and RBF network policy can be used. * Use automatic differentiation. Thanks to automatic differentiation, no gradient derivation is needed. * A unified interface of Gaussian process. MXFusion provides an unified inferface of GP modules. We allows us to easily switch among plan GP, variational sparse GP and stocastic variational GP implementations.
This notebook depends on MXNet, MXFusion and Open AI Gym. These packages can be installed into your Python environment by running the following commands.
pip install mxnet mxfusion gym
Set the global configuration.
import os
os.environ['MXNET_ENGINE_TYPE'] = 'NaiveEngine'
from mxfusion.common import config
config.DEFAULT_DTYPE = 'float64'
%matplotlib inline
We use the inverted pendulum swingup problem as an example. We use the Pendulumv0 environment in Open AI Gym. The task is to swing the pendulum up and balance it at the inverted position. This is a classical control problem and is known to be unsolvable with a linear controller.
To solve this problem with PILCO, we need three components:
 Execute a policy in an real environment (an Open AI Gym simulator in this example) and collect data.
 Fit a GP model as the model for the dynamics of the environment.
 Optimize the policy given the dynamics model learned from all the data that have been collected so far.
The overall PILCO algorithm is to iterate the above three steps until a policy that can solve the problem is found.
Execute the Environment
import gym
env = gym.make('Pendulumv0')
The state of the pendulum environment is a 3D vector. The first two dimensions are the 2D location of the end point of the pendulum. The third dimension encodes the angular speed of the pendulum. The action space is a 1D vector in [2, 2].
We write a helper function for executing the environment with a given policy.
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import animation
def run_one_episode(env, policy, initial_state=None, max_steps=200, verbose=False, render=False):
"""
Drives an episode of the OpenAI gym environment using the policy to decide next actions.
"""
observation = env.reset()
if initial_state is not None:
env.env.state = initial_state
observation = env.env._get_obs()
env._max_episode_steps = max_steps
step_idx = 0
done = False
total_reward = 0
frames = []
all_actions = []
all_observations = [observation]
while not done:
if render:
frames.append(env.render(mode = 'rgb_array'))
if verbose:
print(observation)
action = policy(observation)
observation, reward, done, info = env.step(action)
all_observations.append(observation)
all_actions.append(action)
total_reward += reward
step_idx += 1
if done or step_idx>=max_steps1:
print("Episode finished after {} timesteps because {}".format(step_idx+1, "'done' reached" if done else "Max timesteps reached"))
break
if render:
fig = plt.figure()
ax = fig.gca()
fig.tight_layout()
patch = ax.imshow(frames[0])
ax.axis('off')
def animate(i):
patch.set_data(frames[i])
anim = animation.FuncAnimation(plt.gcf(), animate, frames = len(frames), interval=20)
return total_reward, np.array(all_observations, dtype=np.float64,), np.array(all_actions, dtype=np.float64), anim
else:
return total_reward, np.array(all_observations, dtype=np.float64,), np.array(all_actions, dtype=np.float64)
We first apply a random policy and see how the environment reacts. The random policy uniformly samples in the space of action.
def random_policy(state):
return env.action_space.sample()
The animation is generated with the following commands:
anim = run_one_episode(env, random_policy, max_steps=500, render=True, verbose=False)[1]
with open('animation_random_policy.html', 'w') as f:
f.write(anim.to_jshtml())
Pendulum
The dynamics model of pendulum can be written as
$$
p(\dataScalar_{t+1}\dataScalar_t, a_t)
$$
where $\dataScalar_t$ is the state vector at the time t and a_{t} is the action taken at the time t.
PILCO uses a Gaussian process to model the above dynamics.
def prepare_data(state_list, action_list, win_in):
"""
Prepares a list of states and a list of actions as inputs to the Gaussian Process for training.
"""
X_list = []
Y_list = []
for state_array, action_array in zip(state_list, action_list):
# the state and action array shape should be aligned.
assert state_array.shape[0]1 == action_array.shape[0]
for i in range(state_array.shape[0]win_in):
Y_list.append(state_array[i+win_in:i+win_in+1])
X_list.append(np.hstack([state_array[i:i+win_in].flatten(), action_array[i:i+win_in].flatten()]))
X = np.vstack(X_list)
Y = np.vstack(Y_list)
return X, Y
In this example, we do a maximum likelihood estimate for the model hyper parameters. In MXFusion
, Gaussian process regression model is available as a module, which includes a dediated inference algorithm.
import mxnet as mx
from mxfusion import Model, Variable
from mxfusion.components.variables import PositiveTransformation
from mxfusion.components.distributions.gp.kernels import RBF
from mxfusion.modules.gp_modules import GPRegression
from mxfusion.inference import GradBasedInference, MAP
Define and fit the model
def fit_model(state_list, action_list, win_in, verbose=True):
"""
Fits a Gaussian Process model to the state / action pairs passed in.
This creates a model of the environment which is used during
policy optimization instead of querying the environment directly.
See mxfusion.gp_modules for additional types of GP models to fit,
including Sparse GP and Stochastic Varitional Inference Sparse GP.
"""
X, Y = prepare_data(state_list, action_list, win_in)
m = Model()
m.N = Variable()
m.X = Variable(shape=(m.N, X.shape[1]))
m.noise_var = Variable(shape=(1,), transformation=PositiveTransformation(),
initial_value=0.01)
m.kernel = RBF(input_dim=X.shape[1], variance=1, lengthscale=1, ARD=True)
m.Y = GPRegression.define_variable(
X=m.X, kernel=m.kernel, noise_var=m.noise_var,
shape=(m.N, Y.shape[1]))
m.Y.factor.gp_log_pdf.jitter = 1e6
infr = GradBasedInference(
inference_algorithm=MAP(model=m, observed=[m.X, m.Y]))
infr.run(X=mx.nd.array(X, dtype='float64'),
Y=mx.nd.array(Y, dtype='float64'),
max_iter=1000, learning_rate=0.1, verbose=verbose)
return m, infr, X, Y
Policy
PILCO computes the expected reward of a policy given the dynamics model. First, we need to define the parametric form of the policy. In this example, we use a neural network with one hidden layer. As the action space is [2, 2], we apply a tanh
transformation and multiply the come with two. This enforces the returned actions stay within the range.
We define a neural network with one hidden layer and and output constrained between [2,2] for the policy.
from mxnet.gluon import HybridBlock
from mxnet.gluon.nn import Dense
class NNController(HybridBlock):
"""Define a neural network policy network."""
def __init__(self, prefix=None, params=None):
super(NNController, self).__init__(prefix=prefix, params=params)
self.dense1 = Dense(100, in_units=len(env.observation_space.high), dtype='float64', activation='relu')
self.dense2 = Dense(1, in_units=100, dtype='float64', activation='tanh')
def hybrid_forward(self, F, x):
out = self.dense2(self.dense1(x))*2 # Scale up the output
return out
policy = NNController()
policy.collect_params().initialize(mx.initializer.Xavier(magnitude=1))
To compute the expected reward, we also need to define a reward function. This reward function is defined by us according to the task. The main component is the height of the pendulum. We also penalize the force and the angular momentum.
class CostFunction(mx.gluon.HybridBlock):
"""
The goal is to get the pendulum upright and stable as quickly as possible.
Taken from the code for Pendulum.
"""
def hybrid_forward(self, F, state, action):
"""
:param state: [np.cos(theta), np.sin(theta), ~ momentum(theta)]
a > 0 when pendulum is upright, largest when pendulum is hanging down completely.
b > penalty for taking action
c > penalty for pendulum momentum
"""
a_scale = 2.
b_scale = .001
c_scale = .1
a = F.sum(a_scale * (state[:,:,0:1] 1) ** 2, axis=1)
b = F.sum(b_scale * action ** 2, axis=1)
c = F.sum(c_scale * state[:,:,2:3] ** 2, axis=1)
return (a + c + b)
cost = CostFunction()
The expected reward function can be written as
$$
R = \mathbb{E}_{p(\dataScalar_T, \ldots,
\dataScalar_0)}\left(\sum_{t=0}^\top r(\dataScalar_t)\right)
$$
where r(⋅) is the reward function, $p(\dataScalar_T, \ldots, \dataScalar_0)$ is the joint distribution when applying the policy to the dynamics model:
$$
p(\dataScalar_T, \ldots, \dataScalar_0) = p(\dataScalar_0) \prod_{t=1}^\top p(\dataScalar_t\dataScalar_{t1}, a_{t1}),
$$
where $a_{t1} = \pi(\dataScalar_{t1})$ is the action taken at the time t − 1, which is the outcome of the policy π(⋅).
The expected reward function is implemented as follows.
Obtaining the policy gradients
from mxfusion.inference.inference_alg import SamplingAlgorithm
class PolicyUpdateGPParametricApprox(SamplingAlgorithm):
"""Class for the policy update for PILCO."""
def compute(self, F, variables):
s_0 = self.initial_state_generator(self.num_samples)
a_0 = self.policy(s_0)
a_t_plus_1 = a_0
x_t = F.expand_dims(F.concat(s_0, a_0, dim=1), axis=1)
gp = self.model.Y.factor
sample_func = gp.draw_parametric_samples(F, variables, self.num_samples, self.approx_samples)
cost = 0
for t in range(self.n_time_steps):
s_t_plus_1 = sample_func(F, x_t)
cost = cost + self.cost_function(s_t_plus_1, a_t_plus_1)
a_t_plus_1 = mx.nd.expand_dims(self.policy(s_t_plus_1), axis=2)
x_t = mx.nd.concat(s_t_plus_1, a_t_plus_1, dim=2)
total_cost = F.mean(cost)
return total_cost, total_cost
We optimize the policy with respect to the expected reward by using a gradient optimizer.
from mxfusion.inference import GradTransferInference
def optimize_policy(policy, cost_func, model, infr, model_data_X, model_data_Y,
initial_state_generator, num_grad_steps,
learning_rate=1e2, num_time_steps=100,
num_samples=10, verbose=True):
"""
Takes as primary inputs a policy, cost function, and trained model.
Optimizes the policy for num_grad_steps number of iterations.
"""
mb_alg = PolicyUpdateGPParametricApprox(
model=model, observed=[model.X, model.Y], cost_function=cost_func,
policy=policy, n_time_steps=num_time_steps,
initial_state_generator=initial_state_generator,
num_samples=num_samples)
infr_pred = GradTransferInference(
mb_alg, infr_params=infr.params, train_params=policy.collect_params())
infr_pred.run(
max_iter=num_grad_steps,
X=mx.nd.array(model_data_X, dtype='float64'),
Y=mx.nd.array(model_data_Y, dtype='float64'),
verbose=verbose, learning_rate=learning_rate)
return policy
The Loop
We need to define a function that provides random initial states.
def initial_state_generator(num_initial_states):
"""
Starts from valid states by drawing theta and momentum
then computing np.cos(theta) and np.sin(theta) for state[0:2].s
"""
return mx.nd.array(
[env.observation_space.sample() for i in range(num_initial_states)],
dtype='float64')
num_episode = 20 # how many model fit + policy optimization episodes to run
num_samples = 100 # how many sample trajectories the policy optimization loop uses
num_grad_steps = 1000 # how many gradient steps the optimizer takes per episode
num_time_steps = 100 # how far to roll out each sample trajectory
learning_rate = 1e3 # learning rate for the policy optimization
all_states = []
all_actions = []
for i_ep in range(num_episode):
# Run an episode and collect data.
if i_ep == 0:
policy_func = lambda x: env.action_space.sample()
else:
policy_func = lambda x: policy(mx.nd.expand_dims(mx.nd.array(x, dtype='float64'), axis=0)).asnumpy()[0]
total_reward, states, actions = run_one_episode(
env, policy_func, max_steps=num_time_steps)
all_states.append(states)
all_actions.append(actions)
# Fit a model.
model, infr, model_data_X, model_data_Y = fit_model(
all_states, all_actions, win_in=1, verbose=True)
# Optimize the policy.
policy = optimize_policy(
policy, cost, model, infr, model_data_X, model_data_Y,
initial_state_generator, num_grad_steps=num_grad_steps,
num_samples=num_samples, learning_rate=learning_rate,
num_time_steps=num_time_steps)
Policy after the first episode (random exploration):
Policy after the 5th episode:
https://github.com/amzn/mxfusion
Long term Aim
 Simulate/Emulate the components of the system.
 Validate with real world using multifidelity.
 Interpret system using e.g. sensitivity analysis.
 Perform end to end learning to optimize.
 Maintain interpretability.
References
Maxwell, James Clerk. 1867. “On Governors.” Proceedings of the Royal Society of London 16. The Royal Society: 270–83. http://www.jstor.org/stable/112510.
Perdikaris, P., M. Raissi, A. Damianou, N. D. Lawrence, and G. E. Karniadakis. 2017. “Nonlinear Information Fusion Algorithms for DataEfficient MultiFidelity Modelling.” Proceedings of the Royal Society of London A: Mathematical, Physical and Engineering Sciences 473 (2198). The Royal Society. doi:10.1098/rspa.2016.0751.
Wiener, Norbert. 1948. Cybernetics: Control and Communication in the Animal and the Machine. Cambridge, MA: MIT Press.
