MLAI Core Module

Machine Learning and Adaptive Intelligence (MLAI) Core Module

This module provides the core machine learning functionality for the MLAI package, designed for teaching and lecturing on machine learning fundamentals. The module includes implementations of key algorithms with a focus on clarity and educational value.

Key Components:

Linear Models (LM): Basic linear regression with various basis functions
Bayesian Linear Models (BLM): Linear models with Bayesian inference
Gaussian Processes (GP): Non-parametric Bayesian models
Logistic Regression (LR): Binary classification
Neural Networks: Simple feedforward networks with dropout
Kernel Functions: Various covariance functions for GPs
Perceptron: Basic binary classifier for teaching

Mathematical Focus:

The implementations emphasize mathematical transparency, with clear connections between code and mathematical notation. Each class and function includes mathematical explanations where relevant.

Educational Design:

Simple, readable implementations suitable for teaching
Clear separation of concepts
Extensive use of mathematical notation in variable names
Comprehensive examples and documentation

For detailed usage examples, see the tutorials in gp_tutorial.py, deepgp_tutorial.py, and mountain_car.py.

Author: Neil D. Lawrence License: MIT

mlai.mlai.filename_join(filename, directory=None)[source]

Join a filename to a directory and create directory if it doesn’t exist.

This utility function ensures that the target directory exists before attempting to create files, which is useful for saving figures, animations, and other outputs during tutorials.

Parameters:

filename (str) – The name of the file to create
directory (str, optional) – The directory path. If None, returns just the filename. If the directory doesn’t exist, it will be created.

Returns:

The full path to the file

Return type:

str

Examples

>>> filename_join("plot.png", "figures")
'figures/plot.png'
>>> filename_join("data.csv")
'data.csv'

mlai.mlai.write_animation(anim, filename, directory=None, **kwargs)[source]

Write a matplotlib animation to a file.

This function saves animations (e.g., from matplotlib.animation) to files in the specified directory, creating the directory if necessary.

Parameters:

anim (matplotlib.animation.Animation) – The animation object to save
filename (str) – The name of the output file (e.g., ‘animation.gif’, ‘animation.mp4’)
directory (str, optional) – The directory to save the animation in. If None, saves in current directory.
**kwargs –
Additional arguments passed to anim.save()

Examples

>>> import matplotlib.animation as animation
>>> # Create animation...
>>> write_animation(anim, "learning_process.gif", "animations")

mlai.mlai.write_animation_html(anim, filename, directory=None)[source]

Save a matplotlib animation as an HTML file with embedded JavaScript.

This function creates an HTML file containing the animation that can be viewed in a web browser. The animation is embedded as JavaScript code.

Parameters:

anim (matplotlib.animation.Animation) – The animation object to save
filename (str) – The name of the output HTML file
directory (str, optional) – The directory to save the HTML file in. If None, saves in current directory.

Examples

>>> import matplotlib.animation as animation
>>> # Create animation...
>>> write_animation_html(anim, "learning_process.html", "animations")

mlai.mlai.write_figure(filename, figure=None, directory=None, frameon=None, **kwargs)[source]

Save a matplotlib figure to a file with proper formatting.

This function saves figures with transparent background by default, which is useful for presentations and publications. The function automatically creates the target directory if it doesn’t exist.

Parameters:

filename (str) – The name of the output file (e.g., ‘plot.png’, ‘figure.pdf’)
figure (matplotlib.figure.Figure, optional) – The figure to save. If None, saves the current figure.
directory (str, optional) – The directory to save the figure in. If None, saves in current directory.
frameon (bool, optional) – Whether to draw a frame around the figure. If None, uses matplotlib default.
**kwargs –
Additional arguments passed to plt.savefig() or figure.savefig()

Examples

>>> plt.plot([1, 2, 3], [1, 4, 2])
>>> write_figure("linear_plot.png", directory="figures")
>>> write_figure("presentation_plot.png", transparent=False, dpi=300)

mlai.mlai.init_perceptron(x_plus, x_minus, seed=1000001)[source]

Initialize the perceptron algorithm with random weights and bias.

The perceptron is a simple binary classifier that learns a linear decision boundary. This function initializes the weight vector w and bias b by randomly selecting a point from either the positive or negative class and setting the normal vector accordingly.

Mathematical formulation: The perceptron decision function is f(x) = w^T x + b, where: - w is the weight vector (normal to the decision boundary) - b is the bias term - x is the input vector

Parameters:

x_plus (numpy.ndarray) – Positive class data points, shape (n_plus, n_features)
x_minus (numpy.ndarray) – Negative class data points, shape (n_minus, n_features)
seed (int, optional) – Random seed for reproducible initialization

Returns:

Initial weight vector, shape (n_features,)

Return type:

numpy.ndarray

Returns:

Initial bias term

Return type:

float

Returns:

The randomly selected point used for initialization

Return type:

numpy.ndarray

Examples

>>> x_plus = np.array([[1, 2], [2, 3], [3, 4]])
>>> x_minus = np.array([[0, 0], [1, 0], [0, 1]])
>>> w, b, x_select = init_perceptron(x_plus, x_minus)
>>> print(f"Weight vector: {w}, Bias: {b}")

mlai.mlai.update_perceptron(w, b, x_plus, x_minus, learn_rate)[source]

Update the perceptron weights and bias using stochastic gradient descent.

This function implements one step of the perceptron learning algorithm. It randomly selects a training point and updates the weights if the point is misclassified.

Mathematical formulation: The perceptron update rule is: - If y_i = +1 and w^T x_i + b ≤ 0: w ← w + η x_i, b ← b + η - If y_i = -1 and w^T x_i + b > 0: w ← w - η x_i, b ← b - η

where η is the learning rate.

Parameters:

w (numpy.ndarray) – Current weight vector, shape (n_features,)
b (float) – Current bias term
x_plus (numpy.ndarray) – Positive class data points, shape (n_plus, n_features)
x_minus (numpy.ndarray) – Negative class data points, shape (n_minus, n_features)
learn_rate (float) – Learning rate (step size) for weight updates

Returns:

Updated weight vector

Return type:

numpy.ndarray

Returns:

Updated bias term

Return type:

float

Returns:

The randomly selected point that was used for the update

Return type:

numpy.ndarray

Returns:

True if weights were updated, False otherwise

Return type:

bool

Examples

>>> w = np.array([0.5, -0.3])
>>> b = 0.1
>>> x_plus = np.array([[1, 2], [2, 3]])
>>> x_minus = np.array([[0, 0], [1, 0]])
>>> w_new, b_new, x_select, updated = update_perceptron(w, b, x_plus, x_minus, 0.1)

class mlai.mlai.Model[source]

Bases: object

Abstract base class for all machine learning models.

This class defines the interface that all models in MLAI must implement. It provides a common structure for different types of models while allowing for specific implementations of objective functions and fitting procedures.

objective() : float: Compute the objective function value (to be minimized)

fit() : None: Fit the model to the data (to be implemented by subclasses)

Examples:

>>> class MyModel(Model):
...     def objective(self):
...         return 0.0  # Implement objective
...     def fit(self):
...         pass  # Implement fitting

__init__()[source]

objective()[source]

Compute the objective function value.

This method should return a scalar value that represents the model’s objective function (typically to be minimized).

Returns:: The objective function value
Return type:: float
Raises:: NotImplementedError – If not implemented by subclass

fit()[source]

Fit the model to the data.

This method should implement the model fitting procedure, updating the model parameters to minimize the objective function.

Raises:: NotImplementedError – If not implemented by subclass

class mlai.mlai.ProbModel[source]

Bases: Model

Abstract base class for probabilistic models.

This class extends the base Model class to provide a framework for probabilistic models that can compute log-likelihoods. The objective function is defined as the negative log-likelihood, which is commonly minimized in maximum likelihood estimation.

Mathematical formulation: The objective function is: J(θ) = -log p(y|X, θ) where θ are the model parameters, y are the targets, and X are the inputs.

objective() : float: Compute the negative log-likelihood

log_likelihood() : float: Compute the log-likelihood (to be implemented by subclasses)

Examples:

>>> class GaussianModel(ProbModel):
...     def log_likelihood(self):
...         return -0.5 * np.sum((y - μ)**2 / σ²)  # Gaussian log-likelihood

__init__()[source]

objective()[source]

Compute the negative log-likelihood.

This is the standard objective function for probabilistic models, which is minimized during maximum likelihood estimation.

Returns:: The negative log-likelihood: -log p(y|X, θ)
Return type:: float

log_likelihood()[source]

Compute the log-likelihood of the data given the model parameters.

This method should be implemented by subclasses to provide the specific log-likelihood computation for their model.

Returns:: The log-likelihood: log p(y|X, θ)
Return type:: float
Raises:: NotImplementedError – If not implemented by subclass

fit()

Fit the model to the data.

This method should implement the model fitting procedure, updating the model parameters to minimize the objective function.

Raises:: NotImplementedError – If not implemented by subclass

class mlai.mlai.MapModel(X, y)[source]

Bases: Model

Abstract base class for models that provide a mapping from inputs X to outputs y.

This class extends the base Model class to provide functionality for supervised learning models that map input features to target values. It includes methods for computing prediction errors and root mean square error.

Mathematical formulation: The model provides a mapping: f: X → y where f is the learned function, X are the inputs, and y are the targets.

Parameters:

X (numpy.ndarray) – Input features, shape (n_samples, n_features)
y (numpy.ndarray) – Target values, shape (n_samples,)

predict(X) : numpy.ndarray: Make predictions for new inputs

rmse() : float: Compute the root mean square error

update_sum_squares() : None: Update the sum of squared errors

Examples:

>>> class LinearModel(MapModel):
...     def predict(self, X):
...         return X @ self.w + self.b
...     def update_sum_squares(self):
...         self.sum_squares = np.sum((self.y - self.predict(self.X))**2)

__init__(X, y)[source]

update_sum_squares()[source]

Update the sum of squared errors.

This method should be implemented by subclasses to compute the sum of squared differences between predictions and targets.

Raises:: NotImplementedError – If not implemented by subclass

rmse()[source]

Compute the root mean square error.

The RMSE is defined as: √(Σ(y_i - ŷ_i)² / n) where y_i are the true values, ŷ_i are the predictions, and n is the number of samples.

Returns:: The root mean square error
Return type:: float

predict(X)[source]

Make predictions for new input data.

This method should be implemented by subclasses to provide the specific prediction function for their model.

Parameters:: X (numpy.ndarray) – Input features, shape (n_samples, n_features)
Returns:: Predictions, shape (n_samples,)
Return type:: numpy.ndarray
Raises:: NotImplementedError – If not implemented by subclass

fit()

Fit the model to the data.

This method should implement the model fitting procedure, updating the model parameters to minimize the objective function.

Raises:: NotImplementedError – If not implemented by subclass

objective()

Compute the objective function value.

This method should return a scalar value that represents the model’s objective function (typically to be minimized).

Returns:: The objective function value
Return type:: float
Raises:: NotImplementedError – If not implemented by subclass

class mlai.mlai.ProbMapModel(X, y)[source]

Bases: ProbModel, MapModel

Probabilistic model that provides a mapping from X to y.

__init__(X, y)[source]

fit()

Fit the model to the data.

This method should implement the model fitting procedure, updating the model parameters to minimize the objective function.

Raises:: NotImplementedError – If not implemented by subclass

log_likelihood()

Compute the log-likelihood of the data given the model parameters.

This method should be implemented by subclasses to provide the specific log-likelihood computation for their model.

Returns:: The log-likelihood: log p(y|X, θ)
Return type:: float
Raises:: NotImplementedError – If not implemented by subclass

objective()

Compute the negative log-likelihood.

This is the standard objective function for probabilistic models, which is minimized during maximum likelihood estimation.

Returns:: The negative log-likelihood: -log p(y|X, θ)
Return type:: float

predict(X)

Make predictions for new input data.

This method should be implemented by subclasses to provide the specific prediction function for their model.

Parameters:: X (numpy.ndarray) – Input features, shape (n_samples, n_features)
Returns:: Predictions, shape (n_samples,)
Return type:: numpy.ndarray
Raises:: NotImplementedError – If not implemented by subclass

rmse()

Compute the root mean square error.

The RMSE is defined as: √(Σ(y_i - ŷ_i)² / n) where y_i are the true values, ŷ_i are the predictions, and n is the number of samples.

Returns:: The root mean square error
Return type:: float

update_sum_squares()

Update the sum of squared errors.

This method should be implemented by subclasses to compute the sum of squared differences between predictions and targets.

Raises:: NotImplementedError – If not implemented by subclass

class mlai.mlai.LM(X, y, basis)[source]

Bases: ProbMapModel

Linear Model with basis function expansion.

This class implements a linear model that uses basis functions to transform the input features. The model assumes Gaussian noise and uses maximum likelihood estimation to fit the parameters.

Mathematical formulation: The model is: y = Φ(X)w + ε, where ε ~ N(0, σ²I) - Φ(X) is the basis function matrix - w are the weights to be learned - σ² is the noise variance

The maximum likelihood solution is: w* = (Φ^T Φ)^(-1) Φ^T y

Parameters:

X (numpy.ndarray) – Input features, shape (n_samples, n_features)
y (numpy.ndarray) – Target values, shape (n_samples, 1)
basis (Basis) – Basis function object that provides the feature transformation

w_star

Fitted weight vector

Type:: numpy.ndarray

sigma2

Estimated noise variance

Type:: float

Phi

Basis function matrix Φ(X)

Type:: numpy.ndarray

name

Model name based on the basis function

Type:: str

objective_name

Name of the objective function

Type:: str

Examples

>>> from mlai import polynomial
>>> basis = polynomial(num_basis=4)
>>> model = LM(X, y, basis)
>>> model.fit()
>>> y_pred, _ = model.predict(X_test)

__init__(X, y, basis)[source]

Initialize a linear model with basis function expansion.

Parameters:

X (numpy.ndarray) – Input features, shape (n_samples, n_features)
y (numpy.ndarray) – Target values, shape (n_samples, 1)
basis (Basis) – Basis function object that provides the feature transformation

set_param(name, val, update_fit=True)[source]

Set a model parameter to a given value.

This method allows updating model parameters (including basis function parameters) and optionally refits the model if the parameter change affects the basis function matrix.

Parameters:

name (str) – Name of the parameter to set
val (float) – New value for the parameter
update_fit (bool, optional) – Whether to refit the model after setting the parameter. Default is True.

Raises:

ValueError – If the parameter name is not recognized

Examples

>>> model.set_param('sigma2', 0.5)
>>> model.set_param('num_basis', 6)  # Basis function parameter

update_QR()[source]

Perform QR decomposition on the basis matrix.

The QR decomposition is used for numerically stable computation of the least squares solution. The basis matrix Φ is decomposed as: Φ = QR, where Q is orthogonal and R is upper triangular.

This decomposition allows efficient computation of the weight vector: w* = R^(-1) Q^T y

fit()[source]

Fit the linear model using maximum likelihood estimation.

This method computes the optimal weight vector w* using the least squares solution and estimates the noise variance σ².

Mathematical formulation: - w* = R^(-1) Q^T y (using QR decomposition) - σ² = Σ(y_i - ŷ_i)² / n (maximum likelihood estimate)

Examples

>>> model = LM(X, y, basis)
>>> model.fit()
>>> print(f"Weights: {model.w_star}")
>>> print(f"Noise variance: {model.sigma2}")

predict(X)[source]

Make predictions for new input data.

This method applies the learned weight vector to the basis function transformation of the input data.

Mathematical formulation: ŷ = Φ(X) w* where Φ(X) is the basis function matrix and w* are the fitted weights.

Parameters:: X (numpy.ndarray) – Input features, shape (n_samples, n_features)
Returns:: (predictions, None) where predictions are the predicted values and None indicates no uncertainty estimate (this is a deterministic model)
Return type:: tuple

Examples

>>> y_pred, _ = model.predict(X_test)
>>> print(f"Predictions shape: {y_pred.shape}")

update_f()[source]: Update values at the prediction points.

update_sum_squares()[source]: Compute the sum of squares error.

objective()[source]: Compute the objective function.

log_likelihood()[source]: Compute the log likelihood.

rmse()

Compute the root mean square error.

The RMSE is defined as: √(Σ(y_i - ŷ_i)² / n) where y_i are the true values, ŷ_i are the predictions, and n is the number of samples.

Returns:: The root mean square error
Return type:: float

class mlai.mlai.Basis(function, number, **kwargs)[source]

Bases: object

Basis function wrapper class.

This class provides a wrapper around basis functions to standardize their interface and store their parameters.

Parameters:

function (function) – The basis function to wrap
number (int) – Number of basis functions
**kwargs –
Additional arguments passed to the basis function

__init__(function, number, **kwargs)[source]

Initialize the basis function wrapper.

Parameters:

function (function) – The basis function to wrap
number (int) – Number of basis functions
**kwargs –
Additional arguments passed to the basis function

Phi(X)[source]

Compute the basis function matrix.

Parameters:: X (numpy.ndarray) – Input features, shape (n_samples, n_features)
Returns:: Basis function matrix Φ(X), shape (n_samples, num_basis)
Return type:: numpy.ndarray

mlai.mlai.linear(x, **kwargs)[source]

Define the linear basis function.

Creates a basis matrix with a constant term (bias) and linear terms.

Parameters:

x (numpy.ndarray) – Input features, shape (n_samples, n_features)
**kwargs –
Additional arguments (ignored for linear basis)

Returns:

Basis matrix with constant and linear terms, shape (n_samples, n_features + 1)

Return type:

numpy.ndarray

Examples

>>> X = np.array([[1, 2], [3, 4]])
>>> Phi = linear(X)
>>> print(Phi.shape)  # (2, 3) - includes bias term

mlai.mlai.polynomial(x, num_basis=4, data_limits=[-1.0, 1.0])[source]

Define the polynomial basis function.

Creates a basis matrix with polynomial terms up to degree (num_basis - 1). The input is normalized to the range [-1, 1] for numerical stability.

Parameters:

x (numpy.ndarray) – Input features, shape (n_samples, n_features)
num_basis (int, optional) – Number of basis functions (polynomial degree + 1)
data_limits (list, optional) – Range for normalizing the input data [min, max]

Returns:

Polynomial basis matrix, shape (n_samples, num_basis)

Return type:

numpy.ndarray

Examples

>>> X = np.array([[0.5], [1.0]])
>>> Phi = polynomial(X, num_basis=3)
>>> print(Phi.shape)  # (2, 3) - constant, linear, quadratic terms

mlai.mlai.radial(x, num_basis=4, data_limits=[-1.0, 1.0], width=None)[source]

Define the radial basis function (RBF).

Creates a basis matrix using Gaussian radial basis functions centered at evenly spaced points across the data range.

Parameters:

x (numpy.ndarray) – Input features, shape (n_samples, n_features)
num_basis (int, optional) – Number of radial basis functions
data_limits (list, optional) – Range for centering the basis functions [min, max]
width (float, optional) – Width parameter for the Gaussian functions. If None, auto-computed.

Returns:

Radial basis matrix, shape (n_samples, num_basis)

Return type:

numpy.ndarray

Examples

>>> X = np.array([[0.5], [1.0]])
>>> Phi = radial(X, num_basis=3)
>>> print(Phi.shape)  # (2, 3)

mlai.mlai.fourier(x, num_basis=4, data_limits=[-1.0, 1.0], frequency_range=None)[source]

Define the Fourier basis function.

Creates a basis matrix using sine and cosine functions with different frequencies. The first basis function is a constant (1), followed by alternating sine and cosine terms.

Parameters:

x (numpy.ndarray) – Input features, shape (n_samples, n_features)
num_basis (int, optional) – Number of basis functions (including constant term)
data_limits (list, optional) – Range for scaling the frequencies [min, max]
frequency_range (list, optional) – Specific frequencies to use. If None, auto-computed.

Returns:

Fourier basis matrix, shape (n_samples, num_basis)

Return type:

numpy.ndarray

Examples

>>> X = np.array([[0.5], [1.0]])
>>> Phi = fourier(X, num_basis=4)
>>> print(Phi.shape)  # (2, 4) - constant, sin, cos, sin

mlai.mlai.relu(x, num_basis=4, data_limits=[-1.0, 1.0], gain=None)[source]

Define the rectified linear units (ReLU) basis function.

Creates a basis matrix using ReLU activation functions with different thresholds. The first basis function is a constant (1), followed by ReLU functions with varying thresholds and gains.

Parameters:

x (numpy.ndarray) – Input features, shape (n_samples, n_features)
num_basis (int, optional) – Number of basis functions (including constant term)
data_limits (list, optional) – Range for positioning the ReLU thresholds [min, max]
gain (numpy.ndarray, optional) – Gain parameters for each ReLU function. If None, auto-computed.

Returns:

ReLU basis matrix, shape (n_samples, num_basis)

Return type:

numpy.ndarray

Examples

>>> X = np.array([[0.5], [1.0]])
>>> Phi = relu(X, num_basis=3)
>>> print(Phi.shape)  # (2, 3) - constant + 2 ReLU functions

mlai.mlai.hyperbolic_tangent(x, num_basis=4, data_limits=[-1.0, 1.0], gain=None)[source]

Define the hyperbolic tangent basis function.

Creates a basis matrix using tanh activation functions with different thresholds and gains. The first basis function is a constant (1), followed by tanh functions with varying parameters.

Parameters:

x (numpy.ndarray) – Input features, shape (n_samples, n_features)
num_basis (int, optional) – Number of basis functions (including constant term)
data_limits (list, optional) – Range for positioning the tanh thresholds [min, max]
gain (numpy.ndarray, optional) – Gain parameters for each tanh function. If None, auto-computed.

Returns:

Tanh basis matrix, shape (n_samples, num_basis)

Return type:

numpy.ndarray

Examples

>>> X = np.array([[0.5], [1.0]])
>>> Phi = hyperbolic_tangent(X, num_basis=3)
>>> print(Phi.shape)  # (2, 3) - constant + 2 tanh functions

class mlai.mlai.Noise[source]

Bases: ProbModel

Abstract base class for noise models.

This class extends ProbModel to provide a framework for modeling noise distributions in probabilistic models.

__init__()[source]

fit()

Fit the model to the data.

This method should implement the model fitting procedure, updating the model parameters to minimize the objective function.

Raises:: NotImplementedError – If not implemented by subclass

log_likelihood()

Compute the log-likelihood of the data given the model parameters.

This method should be implemented by subclasses to provide the specific log-likelihood computation for their model.

Returns:: The log-likelihood: log p(y|X, θ)
Return type:: float
Raises:: NotImplementedError – If not implemented by subclass

objective()

Compute the negative log-likelihood.

This is the standard objective function for probabilistic models, which is minimized during maximum likelihood estimation.

Returns:: The negative log-likelihood: -log p(y|X, θ)
Return type:: float

class mlai.mlai.Gaussian(offset=0.0, scale=1.0)[source]

Bases: Noise

Gaussian noise model.

This class implements a Gaussian noise model with configurable offset and scale parameters.

Parameters:

offset (float, optional) – Offset parameter for the noise model
scale (float, optional) – Scale parameter for the noise model

__init__(offset=0.0, scale=1.0)[source]

log_likelihood(mu, varsigma, y)[source]

Compute the log likelihood of the data under a Gaussian noise model.

Parameters:

mu (numpy.ndarray) – Input mean locations for the log likelihood
varsigma (numpy.ndarray) – Input variance locations for the log likelihood
y (numpy.ndarray) – Target locations for the log likelihood

Returns:

Log likelihood value

Return type:

float

grad_vals(mu, varsigma, y)[source]

Compute the gradient of noise log Z with respect to input mean and variance.

Parameters:

mu (numpy.ndarray) – Mean input locations with respect to which gradients are being computed
varsigma (numpy.ndarray) – Variance input locations with respect to which gradients are being computed
y (numpy.ndarray) – Noise model output observed values associated with the given points

Returns:

Tuple containing the gradient of log Z with respect to the input mean and the gradient of log Z with respect to the input variance

Return type:

tuple

fit()

Fit the model to the data.

This method should implement the model fitting procedure, updating the model parameters to minimize the objective function.

Raises:: NotImplementedError – If not implemented by subclass

objective()

Compute the negative log-likelihood.

This is the standard objective function for probabilistic models, which is minimized during maximum likelihood estimation.

Returns:: The negative log-likelihood: -log p(y|X, θ)
Return type:: float

class mlai.mlai.SimpleNeuralNetwork(nodes)[source]

Bases: Model

A simple one-layer neural network.

This class implements a basic neural network with one hidden layer using ReLU activation functions.

Parameters:: nodes (int) – Number of hidden nodes

__init__(nodes)[source]

predict(x)[source]

Compute output given current basis functions.

Parameters:: x (float) – Input value
Returns:: Network output
Return type:: float

fit()

Fit the model to the data.

This method should implement the model fitting procedure, updating the model parameters to minimize the objective function.

Raises:: NotImplementedError – If not implemented by subclass

objective()

Compute the objective function value.

This method should return a scalar value that represents the model’s objective function (typically to be minimized).

Returns:: The objective function value
Return type:: float
Raises:: NotImplementedError – If not implemented by subclass

class mlai.mlai.SimpleDropoutNeuralNetwork(nodes, drop_p=0.5)[source]

Bases: SimpleNeuralNetwork

Simple neural network with dropout.

This class extends SimpleNeuralNetwork to include dropout regularization during training.

Parameters:

nodes (int) – Number of hidden nodes
drop_p (float, optional) – Dropout probability

__init__(nodes, drop_p=0.5)[source]

do_samp()[source]

Sample the set of basis functions to use.

This method randomly selects which basis functions to use based on the dropout probability.

predict(x)[source]

Compute output given current basis functions used.

Parameters:: x (float) – Input value
Returns:: Network output using only the sampled basis functions
Return type:: float

fit()

Fit the model to the data.

This method should implement the model fitting procedure, updating the model parameters to minimize the objective function.

Raises:: NotImplementedError – If not implemented by subclass

objective()

Compute the objective function value.

This method should return a scalar value that represents the model’s objective function (typically to be minimized).

Returns:: The objective function value
Return type:: float
Raises:: NotImplementedError – If not implemented by subclass

class mlai.mlai.NonparametricDropoutNeuralNetwork(alpha=10, beta=1, n=1000)[source]

Bases: SimpleDropoutNeuralNetwork

A non-parametric dropout neural network.

This class implements a neural network with non-parametric dropout using the Indian Buffet Process (IBP) to control the dropout mechanism.

Parameters:

alpha (float, optional) – Alpha parameter of the IBP controlling dropout
beta (float, optional) – Beta parameter of the two-parameter IBP controlling dropout
n (int, optional) – Number of data points for computing expected features

__init__(alpha=10, beta=1, n=1000)[source]

do_samp()[source]

Sample the next set of basis functions to be used.

This method implements the Indian Buffet Process (IBP) sampling to determine which basis functions to use in the current iteration.

fit()

Fit the model to the data.

This method should implement the model fitting procedure, updating the model parameters to minimize the objective function.

Raises:: NotImplementedError – If not implemented by subclass

objective()

Compute the objective function value.

This method should return a scalar value that represents the model’s objective function (typically to be minimized).

Returns:: The objective function value
Return type:: float
Raises:: NotImplementedError – If not implemented by subclass

predict(x)

Compute output given current basis functions used.

Parameters:: x (float) – Input value
Returns:: Network output using only the sampled basis functions
Return type:: float

class mlai.mlai.BLM(X, y, basis, alpha=1.0, sigma2=1.0)[source]

Bases: LM

Bayesian Linear Model.

This class implements a Bayesian linear model with a Gaussian prior on the weights and Gaussian noise. The model provides uncertainty estimates for predictions.

Parameters:

X (numpy.ndarray) – Input values
y (numpy.ndarray) – Target values, shape (n_samples, 1)
basis (Basis) – Basis function
alpha (float, optional) – Scale of prior on parameters (default: 1.0)
sigma2 (float, optional) – Noise variance (default: 1.0)

__init__(X, y, basis, alpha=1.0, sigma2=1.0)[source]

Initialize the Bayesian Linear Model.

Parameters:

X (numpy.ndarray) – Input values
y (numpy.ndarray) – Target values, shape (n_samples, 1)
basis (Basis) – Basis function
alpha (float, optional) – Scale of prior on parameters (default: 1.0)
sigma2 (float, optional) – Noise variance (default: 1.0)

set_param(name, val, update_fit=True)[source]

Set a parameter to a given value.

Parameters:

name (str) – Name of the parameter to set
val (float) – New value for the parameter
update_fit (bool, optional) – Whether to refit the model after setting the parameter

update_QR()[source]

Perform the QR decomposition on the basis matrix.

This method performs QR decomposition on the augmented basis matrix that includes the prior regularization term.

fit()[source]

Minimize the objective function with respect to the parameters.

This method computes the posterior mean and covariance of the weights using the QR decomposition approach.

predict(X, full_cov=False)[source]

Return the result of the prediction function.

Parameters:

X (numpy.ndarray) – Input features for prediction
full_cov (bool, optional) – Whether to return full covariance matrix

Returns:

Tuple of (predictions, uncertainties)

Return type:

tuple

update_f()[source]

Update values at the prediction points.

This method computes the posterior mean and variance of the function values at the training points.

update_sum_squares()[source]

Compute the sum of squares error.

This method computes the sum of squared differences between the observed targets and the posterior mean predictions.

objective()[source]

Compute the objective function.

For the Bayesian linear model, this is the negative log-likelihood.

update_nll()[source]

Precompute terms needed for the log likelihood.

This method computes the log determinant and quadratic terms that are used in the negative log-likelihood calculation.

nll_split()[source]

Compute the determinant and quadratic term of the negative log likelihood.

Returns:: Tuple of (log_det, quadratic) terms
Return type:: tuple

log_likelihood()[source]

Compute the log likelihood.

Returns:: Log likelihood value
Return type:: float

rmse()

Compute the root mean square error.

The RMSE is defined as: √(Σ(y_i - ŷ_i)² / n) where y_i are the true values, ŷ_i are the predictions, and n is the number of samples.

Returns:: The root mean square error
Return type:: float

mlai.mlai.load_pgm(filename, directory=None, byteorder='>')[source]

Return image data from a raw PGM file as a numpy array.

Format specification: http://netpbm.sourceforge.net/doc/pgm.html

Parameters:

filename (str) – Name of the PGM file to load
directory (str, optional) – Directory containing the file (optional)
byteorder (str, optional) – Byte order for 16-bit images (‘>’ for big-endian, ‘<’ for little-endian)

Returns:

Image data as a numpy array

Return type:

numpy.ndarray

Raises:

ValueError – If the file is not a valid raw PGM file

class mlai.mlai.LR(X, y, basis)[source]

Bases: ProbMapModel

Logistic regression model.

Parameters:

X (numpy.ndarray) – Input values
y (numpy.ndarray) – Target values
basis (function) – Basis function

__init__(X, y, basis)[source]

predict(X)[source]

Generate the prediction function and the basis matrix.

Parameters:: X (numpy.ndarray) – Input features for prediction
Returns:: Tuple of (predicted probabilities, basis matrix)
Return type:: tuple

gradient()[source]

Generate the gradient of the parameter vector.

Returns:: Gradient vector
Return type:: numpy.ndarray

fit(learning_rate=0.1, max_iterations=1000, tolerance=1e-06)[source]

Fit the logistic regression model using gradient descent.

Parameters:

learning_rate (float, optional) – Learning rate for gradient descent
max_iterations (int, optional) – Maximum number of iterations
tolerance (float, optional) – Convergence tolerance

compute_g(f)[source]

Compute the transformation and its logarithms.

Parameters:: f (numpy.ndarray) – Linear combination of features and weights
Returns:: Tuple of (g, log_g, log_gminus)
Return type:: tuple

update_g()[source]: Compute the prediction function on training data.

objective()[source]

Compute the objective function (log-likelihood).

Returns:: Log-likelihood value
Return type:: float

log_likelihood()

Compute the log-likelihood of the data given the model parameters.

This method should be implemented by subclasses to provide the specific log-likelihood computation for their model.

Returns:: The log-likelihood: log p(y|X, θ)
Return type:: float
Raises:: NotImplementedError – If not implemented by subclass

rmse()

Compute the root mean square error.

The RMSE is defined as: √(Σ(y_i - ŷ_i)² / n) where y_i are the true values, ŷ_i are the predictions, and n is the number of samples.

Returns:: The root mean square error
Return type:: float

update_sum_squares()

Update the sum of squared errors.

This method should be implemented by subclasses to compute the sum of squared differences between predictions and targets.

Raises:: NotImplementedError – If not implemented by subclass

class mlai.mlai.GP(X, y, sigma2, kernel)[source]

Bases: ProbMapModel

Gaussian Process model.

Parameters:

X (numpy.ndarray) – Input values
y (numpy.ndarray) – Target values
sigma2 (float) – Noise variance
kernel (function) – Covariance function

__init__(X, y, sigma2, kernel)[source]

update_inverse()[source]: Pre-compute the inverse covariance and related quantities.

fit()[source]: Fit the Gaussian Process model (no-op placeholder).

update_nll()[source]: Precompute the log determinant and quadratic term from the negative log likelihod

nll_split()[source]: Return the two components of the negative log likelihood

log_likelihood()[source]

Compute the log likelihood.

Returns:: Log likelihood value
Return type:: float

objective()[source]

Compute the objective function.

Returns:: Objective function value
Return type:: float

predict(X_test, full_cov=False)[source]

Give a mean and a variance of the prediction.

Parameters:

X_test (numpy.ndarray) – Input features for prediction
full_cov (bool, optional) – Whether to return full covariance matrix

Returns:

Tuple of (mean, variance)

Return type:

tuple

rmse()

Compute the root mean square error.

The RMSE is defined as: √(Σ(y_i - ŷ_i)² / n) where y_i are the true values, ŷ_i are the predictions, and n is the number of samples.

Returns:: The root mean square error
Return type:: float

update_sum_squares()

Update the sum of squared errors.

This method should be implemented by subclasses to compute the sum of squared differences between predictions and targets.

Raises:: NotImplementedError – If not implemented by subclass

mlai.mlai.posterior_f(self, X_test)[source]: Compute the posterior distribution for f in the GP

mlai.mlai.update_inverse(self)[source]: Update the inverse covariance in a numerically more stable manner

class mlai.mlai.Kernel(function, name=None, shortname=None, formula=None, **kwargs)[source]

Bases: object

Covariance function

Parameters:

function (function) – covariance function
name (string) – name of covariance function
shortname (string) – abbreviated name of covariance function
formula (string) – latex formula of covariance function
function – covariance function
**kwargs – See below

Keyword Arguments:

__init__(function, name=None, shortname=None, formula=None, **kwargs)[source]

K(X, X2=None)[source]

Compute the full covariance function given a kernel function for two data points.

Parameters:

X (numpy.ndarray) – First set of data points
X2 (numpy.ndarray, optional) – Second set of data points (optional, defaults to X)

Returns:

Covariance matrix

Return type:

numpy.ndarray

diag(X)[source]

Compute the diagonal of the covariance function

Parameters:: X (numpy.ndarray) – Data points to compute the diagonal for
Returns:: Diagonal of the covariance matrix
Return type:: numpy.ndarray

mlai.mlai.exponentiated_quadratic(x, x_prime, variance=1.0, lengthscale=1.0)[source]

Exponentiated quadratic covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
lengthscale (float, optional) – Lengthscale parameter

Returns:

Covariance value

Return type:

float

mlai.mlai.eq_cov(x, x_prime, variance=1.0, lengthscale=1.0)[source]

Exponentiated quadratic covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
lengthscale (float, optional) – Lengthscale parameter

Returns:

Covariance value

Return type:

float

mlai.mlai.matern32_cov(x, x_prime, variance=1.0, lengthscale=1.0)[source]

Matern 3/2 covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
lengthscale (float, optional) – Lengthscale parameter

Returns:

Covariance value

Return type:

float

mlai.mlai.matern52_cov(x, x_prime, variance=1.0, lengthscale=1.0)[source]

Matern 5/2 covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
lengthscale (float, optional) – Lengthscale parameter

Returns:

Covariance value

Return type:

float

mlai.mlai.icm_cov(x, x_prime, B, subkernel, **kwargs)[source]

Intrinsic coregionalization model. First index is outputs considered for covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
B (numpy.ndarray) – Coregionalization matrix
subkernel (function) – Sub-kernel function
**kwargs –
Additional arguments for the sub-kernel

Returns:

Covariance value

Return type:

float

mlai.mlai.slfm_cov(x, x_prime, W, subkernel, **kwargs)[source]

Semi-parametric latent factor model covariance function. First index is the output of the covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
W (numpy.ndarray) – Latent factor matrix
subkernel (function) – Sub-kernel function
**kwargs –
Additional arguments for the sub-kernel

Returns:

Covariance value

Return type:

float

mlai.mlai.prod_kern(x, x_prime, kernargs)[source]

Product covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
kernargs (list of tuples) – List of tuples (kernel, kwargs)

Returns:

Product of covariance values

Return type:

float

mlai.mlai.relu_cov(x, x_prime, variance=1.0, scale=1.0, w=1.0, b=5.0, alpha=0.0)[source]

Covariance function for a ReLU based neural network.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall scale of the covariance
scale (float, optional) – Overall scale of the weights on the input.
w (float, optional) – Overall scale of the bias on the input
b – Smoothness of the relu activation

Returns:

Covariance value

Return type:

float

mlai.mlai.polynomial_cov(x, x_prime, variance=1.0, degree=2.0, w=1.0, b=1.0)[source]

Polynomial covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
degree (int, optional) – Degree of the polynomial
w (float, optional) – Overall scale of the weights on the input.
b (float, optional) – Overall scale of the bias on the input

Returns:

Covariance value

Return type:

float

mlai.mlai.linear_cov(x, x_prime, variance=1.0)[source]

Linear covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance

Returns:

Covariance value

Return type:

float

mlai.mlai.bias_cov(x, x_prime, variance=1.0)[source]

Bias covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance

Returns:

Covariance value

Return type:

float

mlai.mlai.mlp_cov(x, x_prime, variance=1.0, w=1.0, b=1.0)[source]

MLP covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
w (float, optional) – Overall scale of the weights on the input.
b (float, optional) – Overall scale of the bias on the input

Returns:

Covariance value

Return type:

float

mlai.mlai.sinc_cov(x, x_prime, variance=1.0, w=1.0)[source]

Sinc covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
w (float, optional) – Overall scale of the weights on the input.

Returns:

Covariance value

Return type:

float

mlai.mlai.ou_cov(x, x_prime, variance=1.0, lengthscale=1.0)[source]

Ornstein Uhlenbeck covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
lengthscale (float, optional) – Lengthscale parameter

Returns:

Covariance value

Return type:

float

mlai.mlai.brownian_cov(t, t_prime, variance=1.0)[source]

Brownian motion covariance function.

Parameters:

t (float) – First time point
t_prime (float) – Second time point
variance (float, optional) – Overall variance of the covariance

Returns:

Covariance value

Return type:

float

mlai.mlai.periodic_cov(x, x_prime, variance=1.0, lengthscale=1.0, w=1.0)[source]

Periodic covariance function

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
lengthscale (float, optional) – Lengthscale parameter
w (float, optional) – Overall scale of the weights on the input.

Returns:

Covariance value

Return type:

float

mlai.mlai.ratquad_cov(x, x_prime, variance=1.0, lengthscale=1.0, alpha=1.0)[source]

Rational quadratic covariance function

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
variance (float, optional) – Overall variance of the covariance
lengthscale (float, optional) – Lengthscale parameter
alpha (float, optional) – Smoothness parameter

Returns:

Covariance value

Return type:

float

mlai.mlai.prod_cov(x, x_prime, kerns, kern_args)[source]

Product covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
kerns (list of functions) – List of kernel functions
kern_args (list of dicts) – List of dictionaries of kernel arguments

Returns:

Product of covariance values

Return type:

float

mlai.mlai.add_cov(x, x_prime, kerns, kern_args)[source]

Additive covariance function.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
kerns (list of functions) – List of kernel functions
kern_args (list of dicts) – List of dictionaries of kernel arguments

Returns:

Summed covariance value

Return type:

float

mlai.mlai.basis_cov(x, x_prime, basis)[source]

Basis function covariance.

Parameters:

x (numpy.ndarray) – First data point
x_prime (numpy.ndarray) – Second data point
basis (Basis) – Basis function object

Returns:

Covariance value

Return type:

float

mlai.mlai.contour_data(model, data, length_scales, log_SNRs)[source]

Evaluate the GP objective function for a given data set for a range of signal to noise ratios and a range of lengthscales.

Parameters:

model (GP) – The GP model to evaluate
data (dict) – The data dictionary containing ‘Y’
length_scales (list of floats) – a list of length scales to explore for the contour plot.
log_SNRs (list of floats) – a list of base 10 logarithm signal to noise ratios to explore for the contour plot.

Returns:

Array of log-likelihood values for different length scales and SNRs

Return type:

numpy.ndarray

mlai.mlai.radial_multivariate(x, num_basis=4, width=None, random_state=0)[source]

Multivariate radial basis function (RBF) for multi-dimensional input.

Parameters:

x (numpy.ndarray) – Input features, shape (n_samples, n_features)
num_basis (int, optional) – Number of radial basis functions
width (float, optional) – Width parameter for the Gaussian functions. If None, auto-computed.
random_state (int, optional) – Seed for reproducible center placement

Returns:

Radial basis matrix, shape (n_samples, num_basis)

Return type:

numpy.ndarray