Rename study options without breaking changes

[c-bata]

StudyOptionSet... is redundant and not consistent.

• sile/kurobako-go doesn't use these options.
• kubeflow/katib uses these options.

$ git grep StudyOptionSet ./pkg/suggestion/
pkg/suggestion/v1alpha3/goptuna/converter.go:   studyOpts = append(studyOpts, goptuna.StudyOptionSetDirection(direction))
pkg/suggestion/v1beta1/goptuna/converter.go:    studyOpts = append(studyOpts, goptuna.StudyOptionSetDirection(direction))

[github-actions]

Benchmark result of sigopt/evalset/Ackley problem

• Report ID: a1cd9cad1615247e6d930b1263037812a6ef192a4c2112656d530e884836c9b0
• Kurobako Version: 0.1.10
• Number of Solvers: 6
• Number of Problems: 1
• Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

1. Overall Results
2. Individual Results
3. Solvers
4. Problems
5. Studies

Overall Results

Solver	Borda	Firsts
Goptuna (BIPOP-CMA-ES)	0	1
Goptuna (CMA-ES)	0	1
Goptuna (IPOP-CMA-ES)	0	1
Goptuna (TPE)	0	1
Optuna (TPE)	0	1
Random	0	1

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=10)

Ranking	Solver	Best (avg +- sd)	AUC (avg +- sd)	Elapsed (avg +- sd)
1	Goptuna (TPE) (study)	5.116923 +- 0.561424	8145.139 +- 274.914	42.879 +- 6.544
1	Optuna (TPE) (study)	6.326946 +- 1.945140	8739.853 +- 1732.156	113.138 +- 31.210
1	Goptuna (BIPOP-CMA-ES) (study)	0.050740 +- 0.039694	3875.009 +- 339.454	1.333 +- 0.263
1	Goptuna (IPOP-CMA-ES) (study)	0.050740 +- 0.039694	3875.009 +- 339.454	1.216 +- 0.203
1	Goptuna (CMA-ES) (study)	0.050740 +- 0.039694	3875.009 +- 339.454	1.300 +- 0.278
1	Random (study)	15.476572 +- 0.916833	16117.231 +- 457.140	0.005 +- 0.004

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "bipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (BIPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "tpe"
    ]
  }
}

specification:

{
  "name": "Optuna (TPE)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 93492282fbba0e83fba906eaf579809327a6e263940fb4fd42cc3abc9a155beb

recipe:

{
  "sigopt": {
    "name": "ACKLEY",
    "dim": 10
  }
}

specification:

{
  "name": "sigopt/evalset/Ackley(dim=10)",
  "attrs": {
    "github": "https://github.com/sigopt/evalset",
    "paper": "Dewancker, Ian, et al. \"A strategy for ranking optimization methods using multiple criteria.\" Workshop on Automatic Machine Learning. 2016.",
    "version": "kurobako_problems=0.1.7"
  },
  "params_domain": [
    {
      "name": "p0",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p1",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p2",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p3",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p4",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p5",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p6",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p7",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p8",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p9",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Objective Value",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: f0fa2ceb80845ca22ffa2479f45c1ff82f4870c1bf20f556b5155edd6387cda8

• problem: sigopt/evalset/Ackley(dim=10)
• solver: Goptuna (BIPOP-CMA-ES)
• budget: 1000
• repeats: 5
• concurrency: 1

ID: 1d5f43ce7d06f0ecc20f649b9a660458e0e55148ceb2ff28705e0a123f802638

• problem: sigopt/evalset/Ackley(dim=10)
• solver: Goptuna (CMA-ES)
• budget: 1000
• repeats: 5
• concurrency: 1

ID: 6f8c38733e62ecf7d89a3669fe8739e12eee2d4828c035728ff6e509aac633af

• problem: sigopt/evalset/Ackley(dim=10)
• solver: Goptuna (IPOP-CMA-ES)
• budget: 1000
• repeats: 5
• concurrency: 1

ID: 66298c57166677e2f5d11b0595998342167947d2100a4d34abe76b2d73bdbd0a

• problem: sigopt/evalset/Ackley(dim=10)
• solver: Goptuna (TPE)
• budget: 1000
• repeats: 5
• concurrency: 1

ID: 135c5259d0a821bd89b8eb14637fe9ff1e1028162a8a083ac07acff879672620

• problem: sigopt/evalset/Ackley(dim=10)
• solver: Optuna (TPE)
• budget: 1000
• repeats: 5
• concurrency: 1

ID: ac65437313c4ab7ce99a7ecb88d98d40e124c8e950a546486d06df3b17d4e0ec

• problem: sigopt/evalset/Ackley(dim=10)
• solver: Random
• budget: 1000
• repeats: 5
• concurrency: 1