Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix a bug of successive halving pruner and alter pruner API more Go-ish. #120

Merged
merged 8 commits into from
Jul 28, 2020
Merged

Fix a bug of successive halving pruner and alter pruner API more Go-ish. #120

merged 8 commits into from
Jul 28, 2020

Conversation

c-bata
Copy link
Owner

@c-bata c-bata commented Apr 19, 2020 •
edited
Loading

Summary

In this PR, I added an example which uses successive halving pruner, then I found a lot of issues like pruners haven't worked ever. This PR contains a few breaking changes around pruner API. But I merge this without backward compatibility because I'm sure that nobody uses pruning functionalities.

  • Use successive halving pruner on gorgonia_iris.
  • Alter pruner API more Go-ish.
  • Fix a typo of successive halving pruner option.
  • Fix a bug of successive halving pruner.

Checklist

  • I confirmed that Kubeflow/Katib does not uses pruner API.
  • I confirmed that sile/kurobako-go does not uses pruner API.

Example

$ go run _examples/gorgonia_iris/main.go -dataset ./_examples/gorgonia_iris/iris.csv 
2020/07/29 04:13:59 [INFO] Trial finished: trialID=0 state=Complete evaluation=0.000000
2020/07/29 04:14:01 [INFO] Trial finished: trialID=1 state=Complete evaluation=0.666667
2020/07/29 04:14:03 [INFO] Trial finished: trialID=2 state=Complete evaluation=0.960000
2020/07/29 04:14:03 [INFO] Trial finished: trialID=3 state=Pruned evaluation=0.000000
2020/07/29 04:14:03 [INFO] Trial finished: trialID=4 state=Pruned evaluation=0.093333
2020/07/29 04:14:03 [INFO] Trial finished: trialID=5 state=Pruned evaluation=0.700000
2020/07/29 04:14:03 [INFO] Trial finished: trialID=6 state=Pruned evaluation=0.306667
2020/07/29 04:14:03 [INFO] Trial finished: trialID=7 state=Pruned evaluation=0.260000
2020/07/29 04:14:03 [INFO] Trial finished: trialID=8 state=Pruned evaluation=0.253333
2020/07/29 04:14:03 [INFO] Trial finished: trialID=9 state=Pruned evaluation=0.953333
2020/07/29 04:14:03 [INFO] Trial finished: trialID=10 state=Pruned evaluation=0.000000
2020/07/29 04:14:03 [INFO] Trial finished: trialID=11 state=Pruned evaluation=0.833333
2020/07/29 04:14:03 [INFO] Trial finished: trialID=12 state=Pruned evaluation=0.000000
2020/07/29 04:14:03 [INFO] Trial finished: trialID=13 state=Pruned evaluation=0.233333
2020/07/29 04:14:03 [INFO] Trial finished: trialID=14 state=Pruned evaluation=0.266667
2020/07/29 04:14:03 [INFO] Trial finished: trialID=15 state=Pruned evaluation=0.580000
2020/07/29 04:14:05 [INFO] Trial finished: trialID=16 state=Complete evaluation=0.966667
2020/07/29 04:14:05 [INFO] Trial finished: trialID=17 state=Pruned evaluation=0.693333
2020/07/29 04:14:05 [INFO] Trial finished: trialID=18 state=Pruned evaluation=0.253333
2020/07/29 04:14:05 [INFO] Trial finished: trialID=19 state=Pruned evaluation=0.746667
2020/07/29 04:14:05 [INFO] Trial finished: trialID=20 state=Pruned evaluation=0.433333
2020/07/29 04:14:05 [INFO] Trial finished: trialID=21 state=Pruned evaluation=0.553333
2020/07/29 04:14:07 [INFO] Trial finished: trialID=22 state=Complete evaluation=0.966667
2020/07/29 04:14:07 [INFO] Trial finished: trialID=23 state=Pruned evaluation=0.306667
2020/07/29 04:14:07 [INFO] Trial finished: trialID=24 state=Pruned evaluation=0.000000
2020/07/29 04:14:07 [INFO] Trial finished: trialID=25 state=Pruned evaluation=0.000000
2020/07/29 04:14:07 [INFO] Trial finished: trialID=26 state=Pruned evaluation=0.313333
2020/07/29 04:14:07 [INFO] Trial finished: trialID=27 state=Pruned evaluation=0.153333
2020/07/29 04:14:07 [INFO] Trial finished: trialID=28 state=Pruned evaluation=0.000000
2020/07/29 04:14:07 [INFO] Trial finished: trialID=29 state=Pruned evaluation=0.233333
2020/07/29 04:14:07 [INFO] Trial finished: trialID=30 state=Pruned evaluation=0.000000
2020/07/29 04:14:07 [INFO] Trial finished: trialID=31 state=Pruned evaluation=0.706667
2020/07/29 04:14:07 [INFO] Trial finished: trialID=32 state=Pruned evaluation=0.333333
2020/07/29 04:14:07 [INFO] Trial finished: trialID=33 state=Pruned evaluation=0.000000
2020/07/29 04:14:08 [INFO] Trial finished: trialID=34 state=Pruned evaluation=0.733333
2020/07/29 04:14:08 [INFO] Trial finished: trialID=35 state=Pruned evaluation=0.793333
2020/07/29 04:14:08 [INFO] Trial finished: trialID=36 state=Pruned evaluation=0.960000
2020/07/29 04:14:08 [INFO] Trial finished: trialID=37 state=Pruned evaluation=0.506667
2020/07/29 04:14:08 [INFO] Trial finished: trialID=38 state=Pruned evaluation=0.873333
2020/07/29 04:14:08 [INFO] Trial finished: trialID=39 state=Pruned evaluation=0.726667
2020/07/29 04:14:08 [INFO] Trial finished: trialID=40 state=Pruned evaluation=0.100000
2020/07/29 04:14:08 [INFO] Trial finished: trialID=41 state=Pruned evaluation=0.000000
2020/07/29 04:14:08 [INFO] Trial finished: trialID=42 state=Pruned evaluation=0.073333
2020/07/29 04:14:08 [INFO] Trial finished: trialID=43 state=Pruned evaluation=0.000000
2020/07/29 04:14:08 [INFO] Trial finished: trialID=44 state=Pruned evaluation=0.926667
2020/07/29 04:14:08 [INFO] Trial finished: trialID=45 state=Pruned evaluation=0.033333
2020/07/29 04:14:08 [INFO] Trial finished: trialID=46 state=Pruned evaluation=0.000000
2020/07/29 04:14:08 [INFO] Trial finished: trialID=47 state=Pruned evaluation=0.000000
2020/07/29 04:14:08 [INFO] Trial finished: trialID=48 state=Pruned evaluation=0.000000
2020/07/29 04:14:08 [INFO] Trial finished: trialID=49 state=Pruned evaluation=0.000000
2020/07/29 04:14:10 [INFO] Trial finished: trialID=50 state=Complete evaluation=0.966667
2020/07/29 04:14:10 [INFO] Trial finished: trialID=51 state=Pruned evaluation=0.426667
2020/07/29 04:14:10 [INFO] Trial finished: trialID=52 state=Pruned evaluation=0.000000
2020/07/29 04:14:10 [INFO] Trial finished: trialID=53 state=Pruned evaluation=0.000000
2020/07/29 04:14:10 [INFO] Trial finished: trialID=54 state=Pruned evaluation=0.426667
2020/07/29 04:14:10 [INFO] Trial finished: trialID=55 state=Pruned evaluation=0.406667
2020/07/29 04:14:10 [INFO] Trial finished: trialID=56 state=Pruned evaluation=0.313333
2020/07/29 04:14:10 [INFO] Trial finished: trialID=57 state=Pruned evaluation=0.946667
2020/07/29 04:14:10 [INFO] Trial finished: trialID=58 state=Pruned evaluation=0.000000
2020/07/29 04:14:10 [INFO] Trial finished: trialID=59 state=Pruned evaluation=0.000000
2020/07/29 04:14:10 [INFO] Trial finished: trialID=60 state=Pruned evaluation=0.220000
2020/07/29 04:14:10 [INFO] Trial finished: trialID=61 state=Pruned evaluation=0.393333
2020/07/29 04:14:10 [INFO] Trial finished: trialID=62 state=Pruned evaluation=0.646667
2020/07/29 04:14:10 [INFO] Trial finished: trialID=63 state=Pruned evaluation=0.000000
2020/07/29 04:14:10 [INFO] Trial finished: trialID=64 state=Pruned evaluation=0.853333
2020/07/29 04:14:10 [INFO] Trial finished: trialID=65 state=Pruned evaluation=0.000000
2020/07/29 04:14:10 [INFO] Trial finished: trialID=66 state=Pruned evaluation=0.313333
2020/07/29 04:14:10 [INFO] Trial finished: trialID=67 state=Pruned evaluation=0.693333
2020/07/29 04:14:10 [INFO] Trial finished: trialID=68 state=Pruned evaluation=0.000000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=69 state=Pruned evaluation=0.900000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=70 state=Pruned evaluation=0.280000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=71 state=Pruned evaluation=0.000000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=72 state=Pruned evaluation=0.000000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=73 state=Pruned evaluation=0.560000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=74 state=Pruned evaluation=0.400000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=75 state=Pruned evaluation=0.000000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=76 state=Pruned evaluation=0.686667
2020/07/29 04:14:11 [INFO] Trial finished: trialID=77 state=Pruned evaluation=0.746667
2020/07/29 04:14:11 [INFO] Trial finished: trialID=78 state=Pruned evaluation=0.306667
2020/07/29 04:14:11 [INFO] Trial finished: trialID=79 state=Pruned evaluation=0.000000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=80 state=Pruned evaluation=0.000000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=81 state=Pruned evaluation=0.746667
2020/07/29 04:14:11 [INFO] Trial finished: trialID=82 state=Pruned evaluation=0.346667
2020/07/29 04:14:11 [INFO] Trial finished: trialID=83 state=Pruned evaluation=0.160000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=84 state=Pruned evaluation=0.780000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=85 state=Pruned evaluation=0.000000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=86 state=Pruned evaluation=0.306667
2020/07/29 04:14:11 [INFO] Trial finished: trialID=87 state=Pruned evaluation=0.000000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=88 state=Pruned evaluation=0.060000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=89 state=Pruned evaluation=0.906667
2020/07/29 04:14:11 [INFO] Trial finished: trialID=90 state=Pruned evaluation=0.333333
2020/07/29 04:14:11 [INFO] Trial finished: trialID=91 state=Pruned evaluation=0.333333
2020/07/29 04:14:11 [INFO] Trial finished: trialID=92 state=Pruned evaluation=0.960000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=93 state=Pruned evaluation=0.000000
2020/07/29 04:14:11 [INFO] Trial finished: trialID=94 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=95 state=Complete evaluation=0.973333
2020/07/29 04:14:14 [INFO] Trial finished: trialID=96 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=97 state=Pruned evaluation=0.713333
2020/07/29 04:14:14 [INFO] Trial finished: trialID=98 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=99 state=Pruned evaluation=0.400000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=100 state=Pruned evaluation=0.846667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=101 state=Pruned evaluation=0.946667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=102 state=Pruned evaluation=0.013333
2020/07/29 04:14:14 [INFO] Trial finished: trialID=103 state=Pruned evaluation=0.586667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=104 state=Pruned evaluation=0.173333
2020/07/29 04:14:14 [INFO] Trial finished: trialID=105 state=Pruned evaluation=0.713333
2020/07/29 04:14:14 [INFO] Trial finished: trialID=106 state=Pruned evaluation=0.520000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=107 state=Pruned evaluation=0.173333
2020/07/29 04:14:14 [INFO] Trial finished: trialID=108 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=109 state=Pruned evaluation=0.806667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=110 state=Pruned evaluation=0.820000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=111 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=112 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=113 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=114 state=Pruned evaluation=0.526667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=115 state=Pruned evaluation=0.260000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=116 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=117 state=Pruned evaluation=0.946667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=118 state=Pruned evaluation=0.273333
2020/07/29 04:14:14 [INFO] Trial finished: trialID=119 state=Pruned evaluation=0.860000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=120 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=121 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=122 state=Pruned evaluation=0.506667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=123 state=Pruned evaluation=0.326667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=124 state=Pruned evaluation=0.720000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=125 state=Pruned evaluation=0.280000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=126 state=Pruned evaluation=0.266667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=127 state=Pruned evaluation=0.000000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=128 state=Pruned evaluation=0.080000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=129 state=Pruned evaluation=0.426667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=130 state=Pruned evaluation=0.793333
2020/07/29 04:14:14 [INFO] Trial finished: trialID=131 state=Pruned evaluation=0.026667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=132 state=Pruned evaluation=0.946667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=133 state=Pruned evaluation=0.793333
2020/07/29 04:14:14 [INFO] Trial finished: trialID=134 state=Pruned evaluation=0.926667
2020/07/29 04:14:14 [INFO] Trial finished: trialID=135 state=Pruned evaluation=0.800000
2020/07/29 04:14:14 [INFO] Trial finished: trialID=136 state=Pruned evaluation=0.766667
2020/07/29 04:14:15 [INFO] Trial finished: trialID=137 state=Pruned evaluation=0.726667
2020/07/29 04:14:15 [INFO] Trial finished: trialID=138 state=Pruned evaluation=0.900000
2020/07/29 04:14:15 [INFO] Trial finished: trialID=139 state=Pruned evaluation=0.153333
2020/07/29 04:14:15 [INFO] Trial finished: trialID=140 state=Pruned evaluation=0.900000
2020/07/29 04:14:15 [INFO] Trial finished: trialID=141 state=Pruned evaluation=0.893333
2020/07/29 04:14:15 [INFO] Trial finished: trialID=142 state=Pruned evaluation=0.766667
2020/07/29 04:14:15 [INFO] Trial finished: trialID=143 state=Pruned evaluation=0.373333
2020/07/29 04:14:15 [INFO] Trial finished: trialID=144 state=Pruned evaluation=0.513333
2020/07/29 04:14:15 [INFO] Trial finished: trialID=145 state=Pruned evaluation=0.580000
2020/07/29 04:14:15 [INFO] Trial finished: trialID=146 state=Pruned evaluation=0.733333
2020/07/29 04:14:15 [INFO] Trial finished: trialID=147 state=Pruned evaluation=0.866667
2020/07/29 04:14:15 [INFO] Trial finished: trialID=148 state=Pruned evaluation=0.000000
2020/07/29 04:14:15 [INFO] Trial finished: trialID=149 state=Pruned evaluation=0.960000
2020/07/29 04:14:18 [INFO] Trial finished: trialID=150 state=Complete evaluation=0.973333
2020/07/29 04:14:18 [INFO] Trial finished: trialID=151 state=Pruned evaluation=0.120000
2020/07/29 04:14:18 [INFO] Trial finished: trialID=152 state=Pruned evaluation=0.833333
2020/07/29 04:14:18 [INFO] Trial finished: trialID=153 state=Pruned evaluation=0.860000
2020/07/29 04:14:18 [INFO] Trial finished: trialID=154 state=Pruned evaluation=0.266667
2020/07/29 04:14:18 [INFO] Trial finished: trialID=155 state=Pruned evaluation=0.420000
2020/07/29 04:14:18 [INFO] Trial finished: trialID=156 state=Pruned evaluation=0.366667
2020/07/29 04:14:18 [INFO] Trial finished: trialID=157 state=Pruned evaluation=0.600000
2020/07/29 04:14:18 [INFO] Trial finished: trialID=158 state=Pruned evaluation=0.960000
2020/07/29 04:14:18 [INFO] Trial finished: trialID=159 state=Pruned evaluation=0.280000
2020/07/29 04:14:18 [INFO] Trial finished: trialID=160 state=Pruned evaluation=0.886667
2020/07/29 04:14:18 [INFO] Trial finished: trialID=161 state=Pruned evaluation=0.393333
2020/07/29 04:14:18 [INFO] Trial finished: trialID=162 state=Pruned evaluation=0.826667
2020/07/29 04:14:18 [INFO] Trial finished: trialID=163 state=Pruned evaluation=0.840000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=164 state=Pruned evaluation=0.940000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=165 state=Pruned evaluation=0.000000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=166 state=Pruned evaluation=0.313333
2020/07/29 04:14:19 [INFO] Trial finished: trialID=167 state=Pruned evaluation=0.120000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=168 state=Pruned evaluation=0.000000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=169 state=Pruned evaluation=0.000000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=170 state=Pruned evaluation=0.686667
2020/07/29 04:14:19 [INFO] Trial finished: trialID=171 state=Pruned evaluation=0.353333
2020/07/29 04:14:19 [INFO] Trial finished: trialID=172 state=Pruned evaluation=0.113333
2020/07/29 04:14:19 [INFO] Trial finished: trialID=173 state=Pruned evaluation=0.233333
2020/07/29 04:14:19 [INFO] Trial finished: trialID=174 state=Pruned evaluation=0.440000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=175 state=Pruned evaluation=0.873333
2020/07/29 04:14:19 [INFO] Trial finished: trialID=176 state=Pruned evaluation=0.906667
2020/07/29 04:14:19 [INFO] Trial finished: trialID=177 state=Pruned evaluation=0.806667
2020/07/29 04:14:19 [INFO] Trial finished: trialID=178 state=Pruned evaluation=0.173333
2020/07/29 04:14:19 [INFO] Trial finished: trialID=179 state=Pruned evaluation=0.960000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=180 state=Pruned evaluation=0.680000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=181 state=Pruned evaluation=0.000000
2020/07/29 04:14:19 [INFO] Trial finished: trialID=182 state=Pruned evaluation=0.593333
2020/07/29 04:14:20 [INFO] Trial finished: trialID=183 state=Pruned evaluation=0.960000
2020/07/29 04:14:20 [INFO] Trial finished: trialID=184 state=Pruned evaluation=0.620000
2020/07/29 04:14:20 [INFO] Trial finished: trialID=185 state=Pruned evaluation=0.333333
2020/07/29 04:14:20 [INFO] Trial finished: trialID=186 state=Pruned evaluation=0.953333
2020/07/29 04:14:22 [INFO] Trial finished: trialID=187 state=Complete evaluation=0.973333
2020/07/29 04:14:22 [INFO] Trial finished: trialID=188 state=Pruned evaluation=0.373333
2020/07/29 04:14:22 [INFO] Trial finished: trialID=189 state=Pruned evaluation=0.533333
2020/07/29 04:14:22 [INFO] Trial finished: trialID=190 state=Pruned evaluation=0.886667
2020/07/29 04:14:22 [INFO] Trial finished: trialID=191 state=Pruned evaluation=0.826667
2020/07/29 04:14:22 [INFO] Trial finished: trialID=192 state=Pruned evaluation=0.686667
2020/07/29 04:14:22 [INFO] Trial finished: trialID=193 state=Pruned evaluation=0.806667
2020/07/29 04:14:23 [INFO] Trial finished: trialID=194 state=Pruned evaluation=0.960000
2020/07/29 04:14:23 [INFO] Trial finished: trialID=195 state=Pruned evaluation=0.000000
2020/07/29 04:14:23 [INFO] Trial finished: trialID=196 state=Pruned evaluation=0.500000
2020/07/29 04:14:23 [INFO] Trial finished: trialID=197 state=Pruned evaluation=0.866667
2020/07/29 04:14:23 [INFO] Trial finished: trialID=198 state=Pruned evaluation=0.940000
2020/07/29 04:14:23 [INFO] Trial finished: trialID=199 state=Pruned evaluation=0.000000
2020/07/29 04:14:23 Best evaluation=0.973333
2020/07/29 04:14:23 Solver: Vanilla
2020/07/29 04:14:23 Learning rate (vanilla): 0.001816

@github-actions

This comment has been minimized.

Sign in to view
@c-bata c-bata force-pushed the more-go-ish-pruner branch from 920aeae to d8986d0 Compare April 19, 2020 19:25
c-bata
c-bata commented Apr 19, 2020
View reviewed changes
successivehalving/pruner.go
}
competingValues = append(competingValues, value)
Copy link
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/optuna/optuna/blob/93a55d73ef0f89739605c773fbd6c904ea5b3661/optuna/pruners/successive_halving.py#L222

@c-bata c-bata force-pushed the more-go-ish-pruner branch from 1a2c711 to d79fbaa Compare April 19, 2020 19:33
@c-bata c-bata changed the title More Go-ish pruner API Fix a bug of pruners and make more Go-ish pruner API Jul 28, 2020
@c-bata c-bata added bug Something isn't working examples labels Jul 28, 2020
@c-bata c-bata changed the title Fix a bug of pruners and make more Go-ish pruner API More Go-ish pruner API Jul 28, 2020
@c-bata c-bata changed the title More Go-ish pruner API Fix a bug of successive halving pruner and alter pruner API more Go-ish. Jul 28, 2020
@c-bata c-bata force-pushed the more-go-ish-pruner branch from d79fbaa to 03f7d79 Compare July 28, 2020 19:14
@github-actions
Copy link

Benchmark result of Rosenbrock function

plot curve image

  • Report ID: b72e0dcc60537bdae2317af01701baf9414433db563867a021098679c5f89a88
  • Kurobako Version: 0.1.10
  • Number of Solvers: 7
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (BIPOP-CMA-ES) 1 1
Goptuna (CMA-ES) 1 1
Goptuna (IPOP-CMA-ES) 1 1
Goptuna (TPE) 0 1
Optuna (CMA-ES) 1 1
Optuna (TPE) 0 1
Random 0 0

Individual Results

(1) Problem: Rosenbrock Function

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Goptuna (TPE) (study) 2.326988 +- 2.557454 332103.306 +- 433971.716 0.182 +- 0.038
1 Optuna (TPE) (study) 1.444340 +- 2.328256 132001.686 +- 216733.334 3.392 +- 0.583
1 Optuna (CMA-ES) (study) 0.617863 +- 1.143378 107558.525 +- 205615.855 0.779 +- 0.059
1 Goptuna (BIPOP-CMA-ES) (study) 0.836955 +- 2.786545 36906.435 +- 47729.791 0.045 +- 0.029
1 Goptuna (IPOP-CMA-ES) (study) 0.836955 +- 2.786545 36906.435 +- 47729.791 0.042 +- 0.021
1 Goptuna (CMA-ES) (study) 0.836955 +- 2.786545 36906.435 +- 47729.791 0.038 +- 0.018
5 Random (study) 5.724383 +- 6.688278 140688.449 +- 232034.768 0.000 +- 0.000

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "bipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (BIPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Optuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "tpe"
    ]
  }
}

specification:

{
  "name": "Optuna (TPE)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 01f15f09812e2d814a26d1219a981765c157b45100698158c37abe239cea997b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/rosenbrock_problem",
    "args": []
  }
}

specification:

{
  "name": "Rosenbrock Function",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.0,
        "high": 10.0
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rosenbrock",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 448a4fa37c2c00cd2de71c65834d73154693960539a6aab5ea721d8e87cebf27

ID: 61d0e750fd0ffa044e7c517592e06f5a752aac50eb791f8194f11ca88afc650f

ID: 0ea83c6798372e5bb6cc28f133c9f43d8fc3cc2fc35bac6c736d0b173932eae5

ID: 9f7d1940842a6b2263038ddb1e94170e969f33e73dfa4fdb0f4302d1ca147ec4

ID: fac4aa5c1cb91e3a4b7b06481bd8bca69d9e363c718dae984eaf369c9d95c73a

ID: 2dabcbaaca241ae2723f9c504aa6f2e6fbbd25022b83f7b7b69b8aca4dec9f64

ID: f853cdb8bf8b30946ab222443e8ea4105b3e4aa0dde8d0a1f14b8310b870a195

@github-actions
Copy link

Benchmark result of Rastrigin problem

plot curve image

  • Report ID: 9953c41290ec142fa09b40f2a330cd7e58294182152fbd594c9103e7078e9db0
  • Kurobako Version: 0.1.10
  • Number of Solvers: 7
  • Number of Problems: 1
  • Metrics Precedence: best value -> AUC

Please refer to "A Strategy for Ranking Optimizers using Multiple Criteria" for the ranking strategy used in this report.

Please expand here for more details.

Table of Contents

  1. Overall Results
  2. Individual Results
  3. Solvers
  4. Problems
  5. Studies

Overall Results

Solver Borda Firsts
Goptuna (BIPOP-CMA-ES) 0 1
Goptuna (CMA-ES) 0 1
Goptuna (IPOP-CMA-ES) 0 1
Goptuna (TPE) 1 1
Optuna (CMA-ES) 0 1
Optuna (TPE) 1 1
Random 0 0

Individual Results

(1) Problem: Rastrigin function (dim=2)

Ranking Solver Best (avg +- sd) AUC (avg +- sd) Elapsed (avg +- sd)
1 Goptuna (TPE) (study) 0.056682 +- 0.057501 2342.317 +- 504.722 99.964 +- 10.269
1 Optuna (TPE) (study) 0.255531 +- 0.374109 1705.209 +- 951.156 204.397 +- 56.801
1 Optuna (CMA-ES) (study) 1.079281 +- 0.660871 3170.515 +- 1659.740 32.751 +- 4.611
1 Goptuna (BIPOP-CMA-ES) (study) 0.613861 +- 0.634039 3099.438 +- 2761.768 8.720 +- 1.585
1 Goptuna (IPOP-CMA-ES) (study) 0.637754 +- 0.772753 2887.578 +- 2731.140 9.112 +- 1.416
1 Goptuna (CMA-ES) (study) 1.189730 +- 1.362081 3442.373 +- 3291.051 9.233 +- 1.752
3 Random (study) 1.342591 +- 0.675203 5484.908 +- 2427.214 0.005 +- 0.001

Solvers

ID: 9b2ad76978c9cab636e881f48d36cb398e7812c07cf0cf044ad74b88ba37f902

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "bipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (BIPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: bcb361930b088ad765b33edfe444986095c910402687ed162e8f6c11a5351b43

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: b40e4010fb9c8506d051f50c41db99f67e5d52d585d04ba4ef88e2d6490b6e15

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "ipop-cmaes"
    ]
  }
}

specification:

{
  "name": "Goptuna (IPOP-CMA-ES)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 5c2f3ce0f48edaa415f646290c199434d68ef4ad4638bf963c13f9c1a5d1bd2b

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/goptuna_solver",
    "args": [
      "tpe"
    ]
  }
}

specification:

{
  "name": "Goptuna (TPE)",
  "attrs": {
    "github": "https://github.com/c-bata/goptuna"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 8931843d684313fcaad663dbaa143cbb7bea514bc200c5c8593e10ad7d8d446c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "cmaes"
    ]
  }
}

specification:

{
  "name": "Optuna (CMA-ES)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: 86646e95541bf74caec8d04822a0bafa84c876b38544bee3573e40097daf2e6c

recipe:

{
  "command": {
    "path": "python",
    "args": [
      "/home/runner/work/goptuna/goptuna/_benchmarks/optuna_solver.py",
      "tpe"
    ]
  }
}

specification:

{
  "name": "Optuna (TPE)",
  "attrs": {
    "github": "https://github.com/optuna/optuna",
    "paper": "Akiba, Takuya, et al. \"Optuna: A next-generation hyperparameter optimization framework.\" Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. ACM, 2019.",
    "version": "optuna=1.5.0, kurobako-py=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "CATEGORICAL",
    "CONDITIONAL",
    "CONCURRENT"
  ]
}

ID: d68b081af9fa6cddfbb0253616526b338f391dc7050393134faec93c510a22a2

recipe:

{
  "random": {}
}

specification:

{
  "name": "Random",
  "attrs": {
    "version": "kurobako_solvers=0.1.7"
  },
  "capabilities": [
    "UNIFORM_CONTINUOUS",
    "UNIFORM_DISCRETE",
    "LOG_UNIFORM_CONTINUOUS",
    "LOG_UNIFORM_DISCRETE",
    "CATEGORICAL",
    "CONDITIONAL",
    "MULTI_OBJECTIVE",
    "CONCURRENT"
  ]
}

Problems

ID: 0091bc29d1a812f56db93aa64502974e93cc18283ec26b6c5c99b085b81529b8

recipe:

{
  "command": {
    "path": "/home/runner/work/goptuna/goptuna/bin/rastrigin_problem",
    "args": [
      "2"
    ]
  }
}

specification:

{
  "name": "Rastrigin function (dim=2)",
  "attrs": {},
  "params_domain": [
    {
      "name": "x1",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    },
    {
      "name": "x2",
      "range": {
        "type": "CONTINUOUS",
        "low": -5.12,
        "high": 5.12
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "values_domain": [
    {
      "name": "Rastrigin",
      "range": {
        "type": "CONTINUOUS"
      },
      "distribution": "UNIFORM",
      "constraint": null
    }
  ],
  "steps": 1
}

Studies

ID: 8c50e86759933a0dbaf04c154ae3cebfbbf6b397a49a1db1b47ac5dae7365a89

ID: 5d4812251fac7d96cd6bcb683c134e553741f955e2379673731a49deb201616d

ID: af7aa97997c9e5ca6c80b9a7adc66f4979f841b5abba604f9329d12e88fd05fe

ID: ad5dfb8e8e8259b0545508a7f9bb4794a20418603980eec1b360a0c1109a2024

ID: f156615c68518cb4353f04dea3c364c1df46918484a4b84e612a63733075fecf

ID: fa9d2c5ebf27912d80d2ed5427827541df0332cf56973bcbb8cc878af37ae53c

ID: 235bc03468a4b2491baa33c1956bb754d38a1022f20005381e0557818c7d759f

@c-bata c-bata merged commit 8d81696 into master Jul 28, 2020
@c-bata c-bata deleted the more-go-ish-pruner branch July 28, 2020 19:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
bug Something isn't working examples
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@c-bata