Automatic decision which random distribution is "best"
Suppose you have a problem to solve and several solution strategies. Each strategy has been simulated in an appropriate model and the results have been rated using a real number for each strategy and simulation run.
Question: which strategy should be chosen?
Suppose, e.g., that the cost for one strategy is N(10, 1) distributed and another Exp(0.1). The expectation values are the same (for the exponential distribution it is 1/λ), but the median, deviation and skewness differ largely. If the strategy is applied often (how often?), the expectation value is a good quality indicator. But what if it is applied only once (or, for that matter, n times - for what n should I choose which strategy?)
One method I currently use is to supply a soft- and a hardlimit for the cost. This is appropriate in many cases, e.g. if the problem is pathfinding and the strategies are several means of transportation (if I want to go to a concert, I want to be there when it starts (hard limit), but to be 30min early would be nice (soft limit). However, if I am there even earlier, I gain nothing since I have to wait anyways). The algorithm then selects the strategy with the least cost-occurrences over the hard limit, and if several are zero, the soft limits are used in the same way (after that, the sample expectation values).
However, this is not satisfactory if the best solution for the n-time execution of a strategy is seeked or no limits exist. I have no idea how to apply hypothesis testing in this case of largely overlapping confidence intervals, as is to be expected from "similar" solutions and completely unknown (and sometimes quite "odd") distribution functions.
Any thoughts on how to solve that problem (or hints for books/papers which address a similar problem) are appreciated.
Question: which strategy should be chosen?
Suppose, e.g., that the cost for one strategy is N(10, 1) distributed and another Exp(0.1). The expectation values are the same (for the exponential distribution it is 1/λ), but the median, deviation and skewness differ largely. If the strategy is applied often (how often?), the expectation value is a good quality indicator. But what if it is applied only once (or, for that matter, n times - for what n should I choose which strategy?)
One method I currently use is to supply a soft- and a hardlimit for the cost. This is appropriate in many cases, e.g. if the problem is pathfinding and the strategies are several means of transportation (if I want to go to a concert, I want to be there when it starts (hard limit), but to be 30min early would be nice (soft limit). However, if I am there even earlier, I gain nothing since I have to wait anyways). The algorithm then selects the strategy with the least cost-occurrences over the hard limit, and if several are zero, the soft limits are used in the same way (after that, the sample expectation values).
However, this is not satisfactory if the best solution for the n-time execution of a strategy is seeked or no limits exist. I have no idea how to apply hypothesis testing in this case of largely overlapping confidence intervals, as is to be expected from "similar" solutions and completely unknown (and sometimes quite "odd") distribution functions.
Any thoughts on how to solve that problem (or hints for books/papers which address a similar problem) are appreciated.
Topic archived. No new replies allowed.