Skip to content

Regression

In machine learning it's common to have a regression task. This is where you have a set of inputs and outputs, and you want to find a function that maps the inputs to the outputs. In Radiate, we can use genetic programming to evolve a tree or graph to do just that. The regression problem is a special type of problem that simplifies this process. It provides functionality to normalize/standardize/OHE the inputs and outputs, as well as calculate the fitness of a genotype based on how well it maps the inputs to the outputs.

The regression problem (fitness function) takes a set of inputs and outputs, and optionally a loss function. Each loss reduces the per-sample error between the genotype's predictions and the targets to a single score (averaged over the dataset), which the engine then minimizes:

  • MSE (mean squared error) — the default; averages the squared error, so large misses are penalized heavily. A solid general-purpose choice.
  • MAE (mean absolute error) — averages the absolute error; less sensitive to outliers than MSE.
  • CrossEntropy (XEnt) — for classification. The outputs are treated as logits, so you don't need a sigmoid or softmax on the output nodes. With several outputs (one-hot labels) it's softmax cross-entropy across a sample's outputs; with a single output (0/1 labels) it's sigmoid binary cross-entropy.
  • Diff — averages the raw signed difference (target - pred). The simplest option, but because over- and under-predictions cancel out it's rarely a good standalone target — handy for custom setups.

Let's take a quick look at how we would put together a regression problem using a graph. A tree works exactly the same way — just swap the GraphCodec (or rd.Engine.graph(...)) for its tree equivalent.

GP regression and the GIL

The regression fitness function runs purely in rust, as such any executor can be used (including multi-threaded executors) without any issues with the GIL regardless of the python version being used.

import radiate as rd

rd.random.seed(518)


def compute(x: float) -> float:
    return 4.0 * x**3 - 3.0 * x**2 + x


inputs = []
answers = []

# Create a simple dataset of 20 points where the input is between -1 and 1
# and the output is the result of the compute function above
input = -1.0
for _ in range(-10, 10):
    input += 0.1
    inputs.append([input])
    answers.append([compute(input)])

# Create a simple GraphCodec which takes one input and produces one output
# The graph will have vertex nodes that can add, subtract, or multiply
# The edges will have a weight operation and the output will sum its inputs
codec = rd.GraphCodec.directed(
    shape=(1, 1),
    vertex=[rd.Op.sub(), rd.Op.mul(), rd.Op.sum()],
    edge=rd.Op.weight(),
    output=rd.Op.sum(),
)

# All we have to do to create a regression problem is provide the features & targets for our
# dataset. Optionally, we can provide a loss function as well - the default is mean squared error (MSE).
loss = rd.MSE  # Options are: rd.MSE, rd.MAE, rd.XEnt (CrossEntropy), rd.Diff.
dtype = rd.Float64
fitness_func = rd.Fitness.regression(dtype, inputs, answers, loss=loss)

engine = (
    rd.Engine(codec)
    # Alternative to the `.fitness(fitness_func).minimizing()` approach
    # is to use `.regression(...)` directly as seen below
    # .regression(inputs, answers, loss=loss)
    .fitness(fitness_func)
    .minimizing()  # We want to minimize the loss
    .limit(
        rd.Limit.score(0.001), rd.Limit.generations(100)
    )  # Stop when we reach a loss of 0.001 or after 100 generations
    .alter(
        rd.Cross.graph(0.5, 0.5),
        rd.Mutate.op(0.07, 0.05),
        rd.Mutate.graph(
            0.1, 0.1, allow_recurrent=False
        ),  # True if evolving recurrent graphs is allowed
    )
)

# Run the genetic engine with a score (error) limit of 0.001 or a maximum of 100 generations
result = engine.run(log=True)

Radiate is also fully compatible with DataFrame libraries like Pandas and Polars, so wiring DataFrames into regression problems is straightforward. For an example we'll use polars as it's quickly becoming the go-to DataFrame library in python. Using the .regression(..) method below is an attempt to simplify the configuration of regression problems. It also automatically switches the engine's optimization target to minimization as most regression losses are minimized.

The call to this method is flexible and allows you to specify the target columns, feature columns, and loss function. But it also handles the simple case we saw above where we just want to provide a list of inputs and outputs.

import polars as pl
import radiate as rd

rd.random.seed(518)

# Just a simple dataset with 2 features and a target that is a function of those features.
df = pl.DataFrame(
    {
        "feature_one": [i for i in range(20)],
        "feature_two": [i**2 for i in range(20)],
        "target": [4.0 * i**3 - 3.0 * i**2 + i for i in range(20)],
    }
)

# Build a regression that evolves Graphs with two input features and one output
# (notice that the shape of the Graphs is equal to the number of features and targets in the DataFrame).
# The loss function is mean absolute error (MAE) in this case, but that's only to show flexibility.
engine = (
    rd.Engine.graph(
        shape=(
            2,
            1,
        ),  # <- notice how we have two features now, so the input shape is (2, 1)
        vertex=[rd.Op.sub(), rd.Op.mul(), rd.Op.sum()],
        edge=rd.Op.weight(),
        output=rd.Op.sum(),
    )
    .regression(
        df,
        target_cols=["target"],
        feature_cols=["feature_one", "feature_two"],
        loss=rd.MAE,
    )
    .limit(rd.Limit.score(0.001), rd.Limit.generations(100))
    .alter(
        rd.Cross.graph(0.5, 0.5),
        rd.Mutate.op(0.07, 0.05),
        rd.Mutate.graph(0.1, 0.1),
    )
)

result = engine.run(log=True)
const MIN_SCORE: f32 = 0.001;

fn main() {
    // Set the random seed for reproducibility
    random_provider::seed(1000);

    // Define our node set for the graph - the operations that can be used by each
    // node type in the graph during evolution
    let set = node_set! {
        Input => Op::var(0),
        Edge => Op::weight(),
        Vertex => vec![Op::sub(), Op::mul(), Op::sum()],
        Output => Op::sum()
    };

    let engine = GeneticEngine::builder()
        .codec(GraphCodec::directed(1, 1, set).unwrap())
        .fitness_fn(Regression::new(dataset(), Loss::MSE))
        .minimizing()
        .alter(alters!(
            GraphCrossover::new(0.5, 0.5),
            OperationMutator::new(0.07, 0.05),
            GraphMutator::new(0.1, 0.1)
        ))
        .build();

    engine
        .iter()
        .logging()
        .until_score(MIN_SCORE)
        .last()
        .inspect(display)
        .expect("No result from engine run");
}

// A simple function to take the output of the engine and display the accuracy
// of the result on the dataset
fn display(result: &Generation<GraphChromosome<Op<f32>>, Graph<Op<f32>>>) {
    Accuracy::default()
        .named("Regression Graph")
        .on(&dataset().into())
        .loss(Loss::MSE)
        .eval(result.value())
        .inspect(|acc| {
            println!("{result:?}\n{acc:?}\n{}", result.metrics().dashboard());
        });
}

fn dataset() -> impl Into<DataSet<f32>> {
    let mut inputs = Vec::new();
    let mut answers = Vec::new();

    let mut input = -1.0;
    for _ in -10..10 {
        input += 0.1;
        inputs.push(vec![input]);
        answers.push(vec![compute(input)]);
    }

    (inputs, answers)
}

fn compute(x: f32) -> f32 {
    4.0 * x.powf(3.0) - 3.0 * x.powf(2.0) + x
}

Datasets

radiate ships a set of small, well-known datasets so you can try out a regression problem without bringing your own data. Datasets that are generated randomly follow the global random seed, so they are reproducible.

Dataset Kind Shape
iris classification 150 samples, 4 features, 3 one-hot labels (shuffled)
xor classification the 4-row XOR truth table
parity(bits) classification every bits-wide bit pattern, labelled 1 when an odd number of bits are set
moons(n, noise) classification two interleaving half circles, 2 features, a 0/1 label
circles(n, noise) classification two concentric circles, 2 features, a 0/1 label
spirals(n, noise) classification two intertwined spirals, 2 features, a 0/1 label
simple_memory(n) sequence n steps of a repeating 4-step pattern (input pulses every 4th step, the target the step before), for testing recurrent graphs
cubic regression 4x³ - 3x² + x at 20 evenly spaced points
koza1 regression x⁴ + x³ + x² + x at 20 random points in [-1, 1)
nguyen7 regression ln(x + 1) + ln(x² + 1) at 20 random points in [0, 2)
pagie1 regression 1/(1 + x⁻⁴) + 1/(1 + y⁻⁴) on a 26 × 26 grid over [-5, 5]

A dataset can be shuffled, split into two parts, and have its feature columns standardized (mean 0, standard deviation 1) or normalized (scaled to [0, 1]). The two-class datasets (moons, circles, spirals) come out grouped by class, so shuffle them before splitting.

rd.DataSet exposes its features and labels as numpy arrays (.features, .labels, or both with .pair()), which plug straight into .regression(...). Each builder takes an optional dtype.

import radiate as rd

rd.random.seed(518)

# Built-in datasets come ready to use - here Fisher's iris: 4 features, 3 one-hot labels
data = rd.DataSet.iris()
samples, n_features, n_labels = data.shape

# Scale each feature column to mean 0 and standard deviation 1,
# then take the features and labels as numpy arrays
features, labels = data.standardize().pair()

engine = (
    rd.Engine.graph(
        shape=(n_features, n_labels),
        vertex=[rd.Op.add(), rd.Op.mul(), rd.Op.sigmoid(), rd.Op.relu()],
        edge=rd.Op.weight(),
        output=rd.Op.sigmoid(),
    )
    .regression(features, labels, loss=rd.MSE)
    .alter(
        rd.Cross.graph(0.5, 0.5),
        rd.Mutate.op(0.07, 0.05),
        rd.Mutate.graph(0.1, 0.1),
    )
    .limit(rd.Limit.generations(10))
)

DataSet<F> is generic over f32/f64 and can be passed directly to Regression::new. friedman1(n, noise) (5 informative features plus 5 noise features) is also available.

// Built-in datasets come ready to use - here Fisher's iris: 4 features, 3 one-hot labels
// `split(0.75)` puts the first 75% of the rows in `train` and the rest in `test`
let (mut train, mut test) = DataSet::<f32>::iris().split(0.75);

// Scale each feature column to mean 0 and standard deviation 1
train.standardize();
test.standardize();

let (samples, n_features, n_labels) = train.shape();

let set = node_set! {
    Input => Op::vars(0..n_features),
    Edge => Op::weight(),
    Vertex => vec![Op::add(), Op::mul(), Op::sigmoid(), Op::relu()],
    Output => Op::sigmoid()
};

let engine = GeneticEngine::builder()
    .codec(GraphCodec::directed(n_features, n_labels, set).unwrap())
    .fitness_fn(Regression::new(train, Loss::MSE))
    .minimizing()
    .alter(alters!(
        GraphCrossover::new(0.5, 0.5),
        OperationMutator::new(0.07, 0.05),
        GraphMutator::new(0.1, 0.1),
    ))
    .build();

More robust examples can be found in the next section or in the tree and graph examples in the git repository.