Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

CMA-ES: fix numerical overflow errors #126

Merged
merged 5 commits into from
Apr 22, 2020
Merged

CMA-ES: fix numerical overflow errors #126

merged 5 commits into from
Apr 22, 2020

Conversation

c-bata
Copy link
Owner

@c-bata c-bata commented Apr 22, 2020

  • Cache B and D for faster sampling
  • Avoid numerical overflow errors.

@github-actions
Copy link

github-actions bot commented Apr 22, 2020 •
edited
Loading

Benchmark result of rosenbrock function

plot curve image

  • Report ID: 293a87a8376102a724ac08d574bfba6b2b6d76c75f02cfc7c763479a46c52f7c
  • Kurobako Version: 0.1.10
  • Number of Solvers: 3
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (CMA-ES) 1 1
Optuna (CMA-ES) 1 1
Random 0 0

Individual Results

(1) Problem: Rosenbrock Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Optuna (CMA-ES) (study) 0.109364 +- 0.129570 107191.183 +- 225377.712 3.155 +- 0.043
1 Goptuna (CMA-ES) (study) 0.136456 +- 0.112128 35681.308 +- 44949.456 0.066 +- 0.003
3 Random (study) 2.699247 +- 1.015527 220762.949 +- 370768.308 0.000 +- 0.000

Solvers

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 0ad8d093bea14b3a09cf7d28c548620c8df56b6dfd330f74aa6cd234c7898c44

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Optuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.3.0, kurobako-py=0.1.5"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 01f15f09812e2d814a26d1219a981765c157b45100698158c37abe239cea997b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/rosenbrock_problem",
    "args": []
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 5215baebf9217aa9dd2b3904fde351f6eba751db45f36a9b9f84e616754878ab

ID: 767ec7bc7a60a8108a1e2885a50073ed21887babf6355d916e1ce7c1c4711caf

ID: ff11e028559c0dd7fba670fdbaa978c691eed425893e295908fde1ab9f6a3501

@github-actions
Copy link

github-actions bot commented Apr 22, 2020 •
edited
Loading

Benchmark result of himmelblau function

plot curve image

  • Report ID: 5bb8081d357e48d9642edebbc503b1fddf408ef4dcdf2a722446fd60208791c3
  • Kurobako Version: 0.1.10
  • Number of Solvers: 3
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (CMA-ES) 1 0
Optuna (CMA-ES) 2 1
Random 0 0

Individual Results

(1) Problem: Himmelblau Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Optuna (CMA-ES) (study) 0.000000 +- 0.000000 695.255 +- 214.574 3.675 +- 0.073
2 Goptuna (CMA-ES) (study) 0.000024 +- 0.000043 1382.963 +- 884.115 0.079 +- 0.005
3 Random (study) 0.384155 +- 0.272343 1142.812 +- 686.575 0.000 +- 0.000

Solvers

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 0ad8d093bea14b3a09cf7d28c548620c8df56b6dfd330f74aa6cd234c7898c44

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Optuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.3.0, kurobako-py=0.1.5"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 7fdd0558317568e22a2e73e288d22ae45e759e2927344a1d7d35855b93f7200f

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/himmelblau_problem",
    "args": []
  }
}

specification:

{
  "name": "Himmelblau Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -4.0,
        "high": 4.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Himmelblau",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: ae946b970cfe58137d8cdb686af8fdea34750af1108e1ce4e9902ed5c4bbb7da

ID: 684ad9080bf355bb5760fd9212020f1ea2119a8a0aa377f1552b265b6e2de5f5

ID: d3db75251403a5a572731f0b93433b3142f7ef594e7b0f317f0768fb30b34572

@c-bata c-bata changed the title Fix cmaes numerical overflow CMA-Fix numerical overflow errors Apr 22, 2020
@c-bata c-bata changed the title CMA-Fix numerical overflow errors CMA-ES: fix numerical overflow errors Apr 22, 2020
@c-bata
Copy link
Owner Author

c-bata commented Apr 22, 2020 

(venv) $ go run _examples/cmaes/blackhole/main.go 
...
2020/04/22 20:55:44 [INFO] Trial finished: trialID=9994 state=Complete evaluation=0.000000
2020/04/22 20:55:44 [INFO] Trial finished: trialID=9995 state=Complete evaluation=0.000000
2020/04/22 20:55:44 [INFO] Trial finished: trialID=9996 state=Complete evaluation=0.000000
2020/04/22 20:55:44 [INFO] Trial finished: trialID=9997 state=Complete evaluation=0.000000
2020/04/22 20:55:44 [INFO] Trial finished: trialID=9998 state=Complete evaluation=0.000000
2020/04/22 20:55:44 [INFO] Trial finished: trialID=9999 state=Complete evaluation=0.000000
2020/04/22 20:55:44 Best evaluation=0.000000 (x1=2.000000, x2=-5.000000)

@c-bata
Copy link
Owner Author

c-bata commented Apr 22, 2020

refs CyberAgentAILab/cmaes#31

@c-bata c-bata merged commit 66a64f4 into master Apr 22, 2020
@c-bata c-bata deleted the fix-cmaes-numerical-overflow branch April 22, 2020 12:26
@c-bata
Copy link
Owner Author

c-bata commented Apr 22, 2020

I confirmed the error won't be raised even if I updated 1M generations (and sampled 6M trials):

package main

import (
	"fmt"
	"math"

	"github.com/c-bata/goptuna/cmaes"
)

func objective(x1, x2 float64) float64 {
	return math.Pow(x1-3, 2) + math.Pow(10*(x2+2), 2)
}

func main() {
	mean := []float64{1, 2}
	sigma0 := 1.3
	optimizer, err := cmaes.NewOptimizer(
		mean, sigma0,
		cmaes.OptimizerOptionSeed(0),
	)
	if err != nil {
		panic(err)
	}

	n := 0

	solutions := make([]*cmaes.Solution, optimizer.PopulationSize())
	for generation := 0; generation < 1000000; generation++ {
		for i := 0; i < optimizer.PopulationSize(); i++ {
			x, err := optimizer.Ask()
			if err != nil {
				panic(err)
			}
			n++
			x1, x2 := x[0], x[1]
			v := objective(x1, x2)
			solutions[i] = &cmaes.Solution{
				Params: x,
				Value:  v,
			}
			fmt.Printf("generation %d: %f (x1=%f, x2=%f)\n",
				generation, v, x1, x2)
		}

		err = optimizer.Tell(solutions)
		if err != nil {
			panic(err)
		}
	}
	fmt.Println("n:", n)
}
(venv) $ go run _examples/cmaes/lowlevel/main.go 
...
generation 999999: 0.000000 (x1=3.000000, x2=-2.000000)
generation 999999: 0.000000 (x1=3.000000, x2=-2.000000)
generation 999999: 0.000000 (x1=3.000000, x2=-2.000000)
generation 999999: 0.000000 (x1=3.000000, x2=-2.000000)
n: 6000000

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@c-bata