boost_mirror/math
1
0
Fork 0
mirror of https://github.com/boostorg/math.git synced 2025-05-11 21:33:52 +00:00
Code Issues Packages Projects Releases Wiki Activity

TLC for optimization docs

Browse Source
This commit is contained in:
Nick Thompson 2024-02-12 13:22:02 -08:00 committed by Nick
parent 380a1cf225
commit b7a62c4a29
5 changed files with 136 additions and 97 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

65
doc/optimization/cma_es.qbk
View File

@ -26,14 +26,14 @@ LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
};
template <typename ArgumentContainer, class Func, class URBG>
ArgumentContainer cma_es(
const Func cost_function,
random_search_parameters<ArgumentContainer> const &params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> target_value = std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr);
ArgumentContainer cma_es(const Func cost_function,
cma_es_parameters<ArgumentContainer> const &params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach
= std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr);
} // namespaces
``
@ -44,39 +44,48 @@ Our implementation attempts to follow "The CMA evolution strategy: A tutorial" e
[heading Parameters]
`lower_bounds`: A container representing the lower bounds of the optimization space along each dimension. The `.size()` of the bounds should return the dimension of the problem.
`upper_bounds`: A container representing the upper bounds of the optimization space along each dimension. It should have the same size of `lower_bounds`, and each element should be >= the corresponding element of `lower_bounds`.
`initial_guess`: An optional guess for where we should start looking for solutions. This is provided for consistency with other optimization functions-it's not particularly useful for this function.
`max_generations`: The maximum number of generations before giving up.
`population_size`: The number of function calls per generation. Defaults to 4 + 4log(n), where n is the dimension.
`learning_rate`: Unlikely to be necessary-refer to the reference for when this should be changed.
* `lower_bounds`: A container representing the lower bounds of the optimization space along each dimension.
The `.size()` of the bounds should return the dimension of the problem.
* `upper_bounds`: A container representing the upper bounds of the optimization space along each dimension.
It should have the same size of `lower_bounds`, and each element should be >= the corresponding element of `lower_bounds`.
* `initial_guess`: An optional guess for where we should start looking for solutions.
* `max_generations`: The maximum number of generations before giving up.
* `population_size`: The number of function calls per generation.
Defaults to `4 + 3log(D)`, where `D` is the dimension.
* `learning_rate`: Unlikely to be necessary-refer to the reference for when this should be changed.
[heading The function]
``
template <typename ArgumentContainer, class Func, class URBG>
ArgumentContainer cma_es(const Func cost_function,
cma_es_parameters<ArgumentContainer> const &params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach = std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr)
cma_es_parameters<ArgumentContainer> const &params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach
= std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr)
``
Parameters:
`cost_function`: The cost function to be minimized.
`params`: The parameters to the algorithm as described above.
`gen`: A uniform random bit generator, like `std::mt19937_64`.
`value_to_reach`: An optional value that, if reached, stops the optimization. This is the most robust way to terminate the calculation, but in most cases the optimal value of the cost function is unknown. If it is, use it! Physical considerations can often be used to find optimal values for cost functions.
`cancellation`: An optional atomic boolean to allow the user to stop the computation and gracefully return the best result found up to that point. N.B.: Cancellation is not immediate; the in-progress generation finishes.
`current_minimum_cost`: An optional atomic variable to store the current minimum cost during optimization. This allows developers to (e.g.) plot the progress of the minimization over time and in conjunction with the `cancellation` argument allow halting the computation when the progress stagnates.
`queries`: An optional vector to store intermediate results during optimization. This is useful for debugging and perhaps volume rendering of the objective function after the calculation is complete.
* `cost_function`: The cost function to be minimized.
* `params`: The parameters to the algorithm as described above.
* `gen`: A uniform random bit generator, like `std::mt19937_64`.
* `value_to_reach`: An optional value that, if reached, stops the optimization.
This is the most robust way to terminate the calculation, but in most cases the optimal value of the cost function is unknown.
If it is, use it! Physical considerations can often be used to find optimal values for cost functions.
* `cancellation`: An optional atomic boolean to allow the user to stop the computation and gracefully return the best result found up to that point.
N.B.: Cancellation is not immediate; the in-progress generation finishes.
* `current_minimum_cost`: An optional atomic variable to store the current minimum cost during optimization.
This allows developers to (e.g.) plot the progress of the minimization over time and in conjunction with the `cancellation` argument allow halting the computation when the progress stagnates.
* `queries`: An optional vector to store intermediate results during optimization.
This is useful for debugging and perhaps volume rendering of the objective function after the calculation is complete.
Returns:
The argument vector corresponding to the minimum cost found by the optimization.
The `ArgumentContainer` corresponding to the minimum cost found by the optimization.
[h4:examples Examples]

59
doc/optimization/differential_evolution.qbk
View File

@ -24,7 +24,7 @@ LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
size_t NP = 200;
size_t max_generations = 1000;
size_t threads = std::thread::hardware_concurrency();
ArgumentContainer const * initial_guess = nullptr;
ArgumentContainer const * initial_guess = nullptr;
};
template <typename ArgumentContainer, class Func, class URBG>
@ -45,14 +45,19 @@ We justify this design choice by reference to the "No free lunch" theorem of Wol
[heading Parameters]
`lower_bounds`: A container representing the lower bounds of the optimization space along each dimension. The `.size()` of the bounds should return the dimension of the problem.
`upper_bounds`: A container representing the upper bounds of the optimization space along each dimension. It should have the same size of `lower_bounds`, and each element should be >= the corresponding element of `lower_bounds`.
`mutation_factor`: The F or scale factor controlling the rate at which the population evolves (default is 0.65).
`crossover_probability`: The crossover probability determining the trade-off between exploration and exploitation (default is 0.5).
`NP`: The population size (default is 200). Parallelization occurs over the population, so this should be "large".
`max_generations`: The maximum number of generations for the optimization (default is 1000).
`threads`: The number of threads to use for parallelization (default is the hardware concurrency). If the objective function is already multithreaded, then this should be set to 1 to prevent oversubscription.
`initial_guess`: An optional guess for where we should start looking for solutions.
* `lower_bounds`: A container representing the lower bounds of the optimization space along each dimension.
The `.size()` of the bounds should return the dimension of the problem.
* `upper_bounds`: A container representing the upper bounds of the optimization space along each dimension.
It should have the same size of `lower_bounds`, and each element should be >= the corresponding element of `lower_bounds`.
* `mutation_factor`: Also known as `F` or scale factor in the literature.
It controls the rate at which the population evolves (default is 0.65).
* `crossover_probability`: The crossover probability determining the trade-off between exploration and exploitation (default is 0.5).
* `NP`: The population size (default is 200).
Parallelization occurs over the population, so this should be "large".
* `max_generations`: The maximum number of generations for the optimization (default is 1000).
* `threads`: The number of threads to use for parallelization (default is the hardware concurrency).
If the objective function is already multithreaded, then this should be set to 1 to prevent oversubscription.
* `initial_guess`: An optional guess for where we should start looking for solutions.
The defaults were chosen by a reading of Price, Storn, and Lampinen, chapter 3, with the exception of the population size, which we have chosen a bit larger than they found as core counts have increased since publication of this reference and parallelization occurs within each generation.
Note that there is a tradeoff between finding global minima and convergence speed.
@ -63,27 +68,35 @@ The most robust way of decreasing the probability of getting stuck in a local mi
``
template <typename ArgumentContainer, class Func, class URBG>
ArgumentContainer differential_evolution(const Func cost_function,
differential_evolution_parameters<ArgumentContainer> const &de_params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach = std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr)
differential_evolution_parameters<ArgumentContainer> const &de_params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach
= std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr)
``
Parameters:
`cost_function`: The cost function to be minimized.
`de_params`: The parameters to the algorithm as described above.
`rng`: A uniform random bit generator, like `std::mt19937_64`.
`value_to_reach`: An optional value that, if reached, stops the optimization. This is the most robust way to terminate the calculation, but in most cases the optimal value of the cost function is unknown. If it is, use it! See the referenced book for clear examples of when target values can be deduced.
`cancellation`: An optional atomic boolean to allow the user to stop the computation and gracefully return the best result found up to that point. N.B.: Cancellation is not immediate; the in-progress generation finishes.
`queries`: An optional vector to store intermediate results during optimization. This is useful for debugging and perhaps volume rendering of the objective function after the calculation is complete.
`current_minimum_cost`: An optional atomic variable to store the current minimum cost during optimization. This allows developers to (e.g.) plot the progress of the minimization over time and in conjunction with the `cancellation` argument allow halting the computation when the progress stagnates. Refer to Price, Storn, and Lampinen, Section 3.2 for caveats with this approach.
* `cost_function`: The cost function to be minimized.
* `de_params`: The parameters to the algorithm as described above.
* `rng`: A uniform random bit generator, like `std::mt19937_64`.
* `value_to_reach`: An optional value that, if reached, stops the optimization.
This is the most robust way to terminate the calculation, but in most cases the optimal value of the cost function is unknown.
If it is, use it!
See the referenced book for clear examples of when target values can be deduced.
* `cancellation`: An optional atomic boolean to allow the user to stop the computation and gracefully return the best result found up to that point.
N.B.: Cancellation is not immediate; the in-progress generation finishes.
* `queries`: An optional vector to store intermediate results during optimization.
This is useful for debugging and perhaps volume rendering of the objective function after the calculation is complete.
* `current_minimum_cost`: An optional atomic variable to store the current minimum cost during optimization.
This allows developers to (e.g.) plot the progress of the minimization over time and in conjunction with the cancellation argument allow halting the computation when the progress stagnates.
Refer to Price, Storn, and Lampinen, Section 3.2 for caveats with this approach.
Returns:
The argument vector corresponding to the minimum cost found by the optimization.
The `ArgumentContainer` corresponding to the minimum cost found by the optimization.
N.B.: The termination criteria is an "OR", not an "AND".
So if the maximum generations is hit, the iteration stops, even if (say) a `value_to_reach` has not been attained.

44
doc/optimization/jso.qbk
View File

@ -46,11 +46,13 @@ Again this function has been designed more for progress observability, graceful
[heading Parameters]
`lower_bounds`: A container representing the lower bounds of the optimization space along each dimension. The `.size()` of the bounds should return the dimension of the problem.
`upper_bounds`: A container representing the upper bounds of the optimization space along each dimension. It should have the same size of `lower_bounds`, and each element should be >= the corresponding element of `lower_bounds`.
`initial_population_size`: How big the first generation should be. Defaults to ceil(25log(D+1)sqrt(D)).
`max_function_evaluations`: Defaults to 10000D, where D is the dimension of the space.
`initial_guess`: An optional guess for where we should start looking for solutions.
* `lower_bounds`: A container representing the lower bounds of the optimization space along each dimension.
The `.size()` of the bounds should return the dimension of the problem.
* `upper_bounds`: A container representing the upper bounds of the optimization space along each dimension.
It should have the same size of `lower_bounds`, and each element should be >= the corresponding element of `lower_bounds`.
* `initial_population_size`: How big the first generation should be. Defaults to `ceil(25log(D+1)sqrt(D))` where `D` is the dimension of the problem.
* `max_function_evaluations`: Defaults to 10000D, where `D` is the dimension of the space.
* `initial_guess`: An optional guess for where we should start looking for solutions.
The defaults were chosen from a reading of Brest 2017.
@ -59,23 +61,29 @@ The defaults were chosen from a reading of Brest 2017.
``
template <typename ArgumentContainer, class Func, class URBG>
ArgumentContainer jso(const Func cost_function,
jso_parameters<ArgumentContainer> const &jso_params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach = std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr)
jso_parameters<ArgumentContainer> const &jso_params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach
= std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr)
``
Parameters:
`cost_function`: The cost function to be minimized.
`jso_params`: The parameters to the algorithm as described above.
`gen`: A uniform random bit generator, like `std::mt19937_64`.
`value_to_reach`: An optional value that, if reached, stops the optimization. This is the most robust way to terminate the calculation, but in most cases the optimal value of the cost function is unknown. If it is, use it! Physical considerations can often be used to find optimal values for cost functions.
`cancellation`: An optional atomic boolean to allow the user to stop the computation and gracefully return the best result found up to that point. N.B.: Cancellation is not immediate; the in-progress generation finishes.
`current_minimum_cost`: An optional atomic variable to store the current minimum cost during optimization. This allows developers to (e.g.) plot the progress of the minimization over time and in conjunction with the `cancellation` argument allow halting the computation when the progress stagnates.
`queries`: An optional vector to store intermediate results during optimization. This is useful for debugging and perhaps volume rendering of the objective function after the calculation is complete.
* `cost_function`: The cost function to be minimized.
* `jso_params`: The parameters to the algorithm as described above.
* `gen`: A uniform random bit generator, like `std::mt19937_64`.
* `value_to_reach`: An optional value that, if reached, stops the optimization.
This is the most robust way to terminate the calculation, but in most cases the optimal value of the cost function is unknown.
If it is, use it! Physical considerations can often be used to find optimal values for cost functions.
* `cancellation`: An optional atomic boolean to allow the user to stop the computation and gracefully return the best result found up to that point.
N.B.: Cancellation is not immediate; the in-progress generation finishes.
* `current_minimum_cost`: An optional atomic variable to store the current minimum cost during optimization.
This allows developers to (e.g.) plot the progress of the minimization over time and in conjunction with the `cancellation` argument allow halting the computation when the progress stagnates.
* `queries`: An optional vector to store intermediate results during optimization.
This is useful for debugging and perhaps volume rendering of the objective function after the calculation is complete.
Returns:

63
doc/optimization/random_search.qbk
View File

@ -23,14 +23,14 @@ LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
};
template <typename ArgumentContainer, class Func, class URBG>
ArgumentContainer random_search(
const Func cost_function,
random_search_parameters<ArgumentContainer> const &params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> target_value = std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr);
ArgumentContainer random_search(const Func cost_function,
random_search_parameters<ArgumentContainer> const &params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach
= std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr);
} // namespaces
``
@ -40,7 +40,7 @@ There is no special sauce to this algorithm-it merely blasts function calls over
It's existence is justified by the "No free lunch" theorem in optimization,
which "establishes that for any algorithm, any elevated performance over one class of problems is offset by performance over another class."
In practice, it is not clear that the conditions of the NFL theorem holds,
and on test cases, `random_search` is slower and less accurate than (say) differential evolution and jSO.
and on test cases, `random_search` is slower and less accurate than (say) differential evolution, jSO, and CMA-ES.
However, it is often the case that rapid convergence is not the goal:
For example, we often want to spend some time exploring the cost function surface before moving to a faster converging algorithm.
In addition, random search is embarrassingly parallel, which allows us to avoid Amdahl's law-induced performance problems.
@ -48,37 +48,46 @@ In addition, random search is embarrassingly parallel, which allows us to avoid
[heading Parameters]
`lower_bounds`: A container representing the lower bounds of the optimization space along each dimension. The `.size()` of the bounds should return the dimension of the problem.
`upper_bounds`: A container representing the upper bounds of the optimization space along each dimension. It should have the same size of `lower_bounds`, and each element should be >= the corresponding element of `lower_bounds`.
`max_function_calls`: Defaults to 10000*threads.
`initial_guess`: An optional guess for where we should start looking for solutions. This is provided for consistency with other optimization functions-it's not particularly useful for this function.
* `lower_bounds`: A container representing the lower bounds of the optimization space along each dimension.
The `.size()` of the bounds should return the dimension of the problem.
* `upper_bounds`: A container representing the upper bounds of the optimization space along each dimension.
It should have the same size of `lower_bounds`, and each element should be >= the corresponding element of `lower_bounds`.
* `max_function_calls`: Defaults to 10000*threads.
* `initial_guess`: An optional guess for where we should start looking for solutions.
This is provided for consistency with other optimization functions-it's not particularly useful for this function.
[heading The function]
``
template <typename ArgumentContainer, class Func, class URBG>
ArgumentContainer random_search(const Func cost_function,
random_search_parameters<ArgumentContainer> const &params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach = std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr)
random_search_parameters<ArgumentContainer> const &params,
URBG &gen,
std::invoke_result_t<Func, ArgumentContainer> value_to_reach
= std::numeric_limits<std::invoke_result_t<Func, ArgumentContainer>>::quiet_NaN(),
std::atomic<bool> *cancellation = nullptr,
std::atomic<std::invoke_result_t<Func, ArgumentContainer>> *current_minimum_cost = nullptr,
std::vector<std::pair<ArgumentContainer, std::invoke_result_t<Func, ArgumentContainer>>> *queries = nullptr)
``
Parameters:
`cost_function`: The cost function to be minimized.
`params`: The parameters to the algorithm as described above.
`gen`: A uniform random bit generator, like `std::mt19937_64`.
`value_to_reach`: An optional value that, if reached, stops the optimization. This is the most robust way to terminate the calculation, but in most cases the optimal value of the cost function is unknown. If it is, use it! Physical considerations can often be used to find optimal values for cost functions.
`cancellation`: An optional atomic boolean to allow the user to stop the computation and gracefully return the best result found up to that point. N.B.: Cancellation is not immediate; the in-progress generation finishes.
`current_minimum_cost`: An optional atomic variable to store the current minimum cost during optimization. This allows developers to (e.g.) plot the progress of the minimization over time and in conjunction with the `cancellation` argument allow halting the computation when the progress stagnates.
`queries`: An optional vector to store intermediate results during optimization. This is useful for debugging and perhaps volume rendering of the objective function after the calculation is complete.
* `cost_function`: The cost function to be minimized.
* `params`: The parameters to the algorithm as described above.
* `gen`: A uniform random bit generator, like `std::mt19937_64`.
* `value_to_reach`: An optional value that, if reached, stops the optimization.
This is the most robust way to terminate the calculation, but in most cases the optimal value of the cost function is unknown.
If it is, use it! Physical considerations can often be used to find optimal values for cost functions.
* `cancellation`: An optional atomic boolean to allow the user to stop the computation and gracefully return the best result found up to that point.
N.B.: Cancellation is not immediate; the in-progress generation finishes.
* `current_minimum_cost`: An optional atomic variable to store the current minimum cost during optimization.
This allows developers to (e.g.) plot the progress of the minimization over time and in conjunction with the `cancellation` argument allow halting the computation when the progress stagnates.
* `queries`: An optional vector to store intermediate results during optimization.
This is useful for debugging and perhaps volume rendering of the objective function after the calculation is complete.
Returns:
The argument vector corresponding to the minimum cost found by the optimization.
The `ArgumentContainer` corresponding to the minimum cost found by the optimization.
[h4:examples Examples]

2
example/random_search_example.cpp
View File

@ -59,7 +59,7 @@ int main() {
auto future = std::async(std::launch::async, f);
std::future_status status = future.wait_for(3ms);
while (!cancel && (status != std::future_status::ready)) {
status = future.wait_for(3ms);
status = future.wait_for(1ms);
std::cout << "Current cost is " << current_minimum_cost << "\r";
}