Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Calculate logLikeLihoodsAbove from samplesBelow #151

Merged
merged 1 commit into from
Aug 6, 2020
Merged

Calculate logLikeLihoodsAbove from samplesBelow #151

merged 1 commit into from
Aug 6, 2020

Conversation

c-bata
Copy link
Owner

@c-bata c-bata commented Aug 6, 2020

@c-bata c-bata added the bug Something isn't working label Aug 6, 2020
@github-actions
Copy link

github-actions bot commented Aug 6, 2020

Benchmark result of Ackley function

plot curve image

  • Report ID: b3139c02dca21d5fe7d20a97e3bd2c17632b51280bab9425780a6631d9116f4d
  • Kurobako Version: 0.2.0
  • Number of Solvers: 5
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (CMA-ES) 3 1
Goptuna (TPE) 1 0
Optuna-CMAES 3 1
Optuna-TPE 1 0
Random 0 0

Individual Results

(1) Problem: sigopt/evalset/Ackley(dim=2)

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Goptuna (CMA-ES) (study) 0.720523 +- 0.584595 448.157 +- 182.932 0.036 +- 0.015
1 Optuna-CMAES (study) 1.164011 +- 0.905332 482.710 +- 149.598 0.526 +- 0.073
3 Optuna-TPE (study) 3.074214 +- 1.116972 585.226 +- 126.096 2.011 +- 0.472
3 Goptuna (TPE) (study) 3.467162 +- 0.626148 628.950 +- 121.013 0.158 +- 0.032
5 Random (study) 7.067197 +- 1.265246 871.279 +- 125.875 0.000 +- 0.000

Solvers

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: fd78b69a4424cea6bdf87dc5e79a74a8c6d0aff8371badd7c1ba11c5389b799a

recipe:

{
  "name": "Optuna-CMAES",
  "optuna": {
    "loglevel": "error",
    "sampler": "CmaEsSampler"
  }
}

specification:

{
  "name": "Optuna-CMAES",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.0.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 3071ba4560e75197b0d9e85d0117b988af460904c42b15fb6ccf579d166a4e9a

recipe:

{
  "name": "Optuna-TPE",
  "optuna": {
    "loglevel": "error",
    "sampler": "TPESampler"
  }
}

specification:

{
  "name": "Optuna-TPE",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.0.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: adba9490e43fe66d939f067852b174c90815eb5f7d736b45282d07bd324579d4

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.0"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 17966ef7eccd1a5b5e29f23659e90202e5d228948069153cce3b1f1a03751803

recipe:

{
  "sigopt": {
    "name": "ACKLEY",
    "dim": 2
  }
}

specification:

{
  "name": "sigopt/evalset/Ackley(dim=2)",
  "attrs": {
    "github": "https://github.com/sigopt/evalset",
    "paper": "Dewancker, Ian, et al. \"A strategy for ranking optimization methods using multiple criteria.\" Workshop on Automatic Machine Learning. 2016.",
    "version": "kurobako_problems=0.1.8"
  },
  "params_domain": [
    {
      "name": "p0",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "p1",
      "range": {
        "type": "CONTINUOUS",
        "low": -10.0,
        "high": 30.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Objective Value",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 1976426ff91e4113bb730f4fba67724f4bd21fdaac5cc2166fb9d3e057458b7d

ID: 23c993067b5293e699061e27627e38dcabe267f63caf9773842c9562e4e88b8f

ID: d162d78007748fa9bda3dce403be5c63ed9110740c59be258ea73c0e9ebf708b

ID: 7bcc41ef32dc0cca32d7c01b844ec32abd6fece5f863ab7fc7ec330ba63913d6

ID: fe994927342992c810fcb1f862b16d7d8759ece4e09b05b0c6e742969acbfd14

@github-actions
Copy link

github-actions bot commented Aug 6, 2020

Benchmark result of Rastrigin problem

plot curve image

  • Report ID: d356f012b6d8d34591a7fd2771672f33bbbc3d0da8f8dcaf4194442ebd8eaad6
  • Kurobako Version: 0.2.0
  • Number of Solvers: 4
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (BIPOP-CMA-ES) 0 1
Goptuna (CMA-ES) 0 1
Goptuna (IPOP-CMA-ES) 0 1
Random 0 1

Individual Results

(1) Problem: Rastrigin function (dim=2)

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Goptuna (BIPOP-CMA-ES) (study) 0.628570 +- 0.621310 3004.628 +- 1954.645 9.849 +- 0.958
1 Random (study) 1.392878 +- 0.512078 5576.501 +- 1551.925 0.005 +- 0.002
1 Goptuna (IPOP-CMA-ES) (study) 0.583980 +- 0.569993 3036.752 +- 1822.481 9.909 +- 0.829
1 Goptuna (CMA-ES) (study) 1.341307 +- 1.118099 3757.611 +- 2586.556 10.084 +- 0.570

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "bipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (BIPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: adba9490e43fe66d939f067852b174c90815eb5f7d736b45282d07bd324579d4

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.0"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 0091bc29d1a812f56db93aa64502974e93cc18283ec26b6c5c99b085b81529b8

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/rastrigin_problem",
    "args": [
      "2"
    ]
  }
}

specification:

{
  "name": "Rastrigin function (dim=2)",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rastrigin",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 8c50e86759933a0dbaf04c154ae3cebfbbf6b397a49a1db1b47ac5dae7365a89

ID: 5d4812251fac7d96cd6bcb683c134e553741f955e2379673731a49deb201616d

ID: af7aa97997c9e5ca6c80b9a7adc66f4979f841b5abba604f9329d12e88fd05fe

ID: aeab515b58a5d09dd615e3f47cad686f03c0f24f168a34ff5c3a667d9b408078

@github-actions
Copy link

github-actions bot commented Aug 6, 2020

Benchmark result of HPOBench-Naval problem

plot curve image

  • Report ID: 6a239ba15e4dc4175337926a7db941b28344c9e843031940cb2410826522908a
  • Kurobako Version: 0.2.0
  • Number of Solvers: 5
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Optuna-RANDOM-ASHA 0 1
Optuna-RANDOM-MEDIAN 0 1
Optuna-TPE-ASHA 0 1
Optuna-TPE-MEDIAN 0 1
Random 0 1

Individual Results

(1) Problem: HPO-Bench-Naval

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Optuna-TPE-ASHA (study) 0.000483 +- 0.000580 0.069 +- 0.070 29.515 +- 10.524
1 Optuna-RANDOM-MEDIAN (study) 0.000035 +- 0.000004 0.027 +- 0.020 24.923 +- 2.506
1 Random (study) 0.000152 +- 0.000045 0.044 +- 0.037 0.000 +- 0.000
1 Optuna-TPE-MEDIAN (study) 0.000068 +- 0.000024 0.029 +- 0.030 25.916 +- 6.364
1 Optuna-RANDOM-ASHA (study) 0.000053 +- 0.000011 0.010 +- 0.006 6.950 +- 0.946

Solvers

ID: d7f35c18c3d80bc06651efa2be60cd39448525dac4938682f161f8721dc6eda3

recipe:

{
  "name": "Optuna-RANDOM-ASHA",
  "optuna": {
    "loglevel": "error",
    "sampler": "RandomSampler",
    "pruner": "SuccessiveHalvingPruner"
  }
}

specification:

{
  "name": "Optuna-RANDOM-ASHA",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.0.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 581be156c7d4cd5286f9f0ae7f9cf4555d39923045b738cb2fcf233491871f0b

recipe:

{
  "name": "Optuna-RANDOM-MEDIAN",
  "optuna": {
    "loglevel": "error",
    "sampler": "RandomSampler",
    "pruner": "MedianPruner"
  }
}

specification:

{
  "name": "Optuna-RANDOM-MEDIAN",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.0.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 00b6d16a8f2934ddb046c283a5520fed93476954acf55a12814a9bdac7e278f2

recipe:

{
  "name": "Optuna-TPE-ASHA",
  "optuna": {
    "loglevel": "error",
    "sampler": "TPESampler",
    "pruner": "SuccessiveHalvingPruner"
  }
}

specification:

{
  "name": "Optuna-TPE-ASHA",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.0.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d70996cc6ff2f96bf615d9623f0325d60a1012c914f2a7383d6f9059106411cb

recipe:

{
  "name": "Optuna-TPE-MEDIAN",
  "optuna": {
    "loglevel": "error",
    "sampler": "TPESampler",
    "pruner": "MedianPruner"
  }
}

specification:

{
  "name": "Optuna-TPE-MEDIAN",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=2.0.0, kurobako-py=0.1.8"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: adba9490e43fe66d939f067852b174c90815eb5f7d736b45282d07bd324579d4

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.2.0"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 5c382dedcdf8170f494a51e00713979e4f1754e3520a61df2ab570c3a115f032

recipe:

{
  "hpobench": {
    "dataset": "/home/runner/work/goptuna/goptuna/tmp/fcnet_tabular_benchmarks/fcnet_naval_propulsion_data.hdf5"
  }
}

specification:

{
  "name": "HPO-Bench-Naval",
  "attrs": {
    "github": "https://github.com/automl/nas_benchmarks",
    "paper": "Klein, Aaron, and Frank Hutter. \"Tabular Benchmarks for Joint Architecture and Hyperparameter Optimization.\" arXiv preprint arXiv:1905.04970 (2019).",
    "version": "kurobako_problems=0.1.8"
  },
  "params_domain": [
    {
      "name": "activation_fn_1",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "activation_fn_2",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "tanh",
          "relu"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "batch_size",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 4
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "dropout_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 3
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "init_lr",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "lr_schedule",
      "range": {
        "type": "CATEGORICAL",
        "choices": [
          "cosine",
          "const"
        ]
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_1",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "n_units_2",
      "range": {
        "type": "DISCRETE",
        "low": 0,
        "high": 6
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Validation MSE",
      "range": {
        "type": "CONTINUOUS",
        "low": 0.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 100
}

Studies

ID: b98e3a6ce1e7943d7212740ed82ea19bff92a33cf48802b8e02d6fea0e1b3a19

ID: a128278035279dea09fa9c9be248ab43cf039fe1dde948657e1794e03ff89a58

ID: c911c8bc0385b87b358dcd8f3a8243be61d5fd18c82fe74015bf550ad1277f19

ID: 8bca403316c1c0aa23af7c097c7b13436b20ad4a69c7c5deac166fd577cbcda0

ID: 5aae795e88e4a0125a6e1b625f2ce3c64cccc045eb33dd7ab6f7e2c445207879

@c-bata c-bata merged commit 146588a into master Aug 6, 2020
@c-bata c-bata deleted the fix-log-likelihoods branch August 6, 2020 10:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@c-bata