Java: expose specialized primitive collections (IntList) to avoid auto-boxing

[Krumpet]

fastutil offers data structures for primitive Java types that are smaller in size and faster for reading and mutating. If protoc used e.g. IntArrayList rather than ArrayList<Integer> and provided getInt() then boxing and unboxing could be avoided.

What language does this apply to?
generated Java code

Describe the problem you are trying to solve.
improve read/write performance and size of proto messages

Describe the solution you'd like
utilize data structures provided in fastutil (or another package providing similar benefits) and provide the richer get API so client code can work with unboxed primitives as much as possible.

Describe alternatives you've considered
For smaller collections the difference is negligible, for larger ones we convert them from the source proto object so the penalty of using the regular ArrayList<Integer> is minimized

[esorot]

Are you able to perform the experiment and share the details of your finding at this time?

[Krumpet]

One example I found of intSet performance vs. HashSet:

https://www.baeldung.com/fastutil#1-performance

I'll try to update with some tests for primitive lists as well.

[Krumpet]

I created this repo to test things:

https://github.com/Krumpet/protobuf-performance

initial results testing insertion into fastutil's IntArrayList vs an ArrayList<Integer> (throughput, higher is better):

Benchmark                             (listSize)   Mode  Cnt         Score        Error  Units
IntListTest.fastutilIntListInsertion         100  thrpt    5  17714739.171 ± 596148.258  ops/s
IntListTest.fastutilIntListInsertion       10000  thrpt    5    123036.544 ±   5199.245  ops/s
IntListTest.fastutilIntListInsertion     1000000  thrpt    5      1164.040 ±     53.725  ops/s
IntListTest.genericIntListInsertion          100  thrpt    5   5128286.873 ±  61191.414  ops/s
IntListTest.genericIntListInsertion        10000  thrpt    5     37667.105 ±   1977.311  ops/s
IntListTest.genericIntListInsertion      1000000  thrpt    5        97.854 ±     19.564  ops/s

[Krumpet]

I tried testing with Protobuf's internal IntList but it's implementation isn't public so I couldn't test it from outside.

[Krumpet]

Is there any interest in this? This could be as simple as exposing the internal IntList which is a List<Integer> so existing users aren't affected.

Then, users who care about this performance gain can use the IntList and avoid auto-boxing.

[Krumpet]

@esorot pinging to make sure this didn't fall by the wayside.

[Krumpet]

Another ping :)

[ennerf]

fastutil is a 22.3MB dependency, so it's ~14x (!) larger than the already-enormous protobuf-java library.

I'm not affiliated with protobuf, but IMO public libraries should strife for zero dependencies, and adding something that large is an absolute no-go. Exposing the internal IntList would be more reasonable, but I think there are already several issues for that.

[Krumpet]

There's the fastutil-core jar at 6MB with only some of the data structures (for ints, longs, and doubles). Exposing IntList (and other specialized primitive collections, if protobuf has them) is probably the better thing to do in this case, though. Can you link to these open issues? I'll see if I can find them myself as well.

[ennerf]

This seems like the biggest one #3316