I was reading the The Book of JOSH and saw the following statement:
“Json delivers on what XML promised. Simple to understand, effective data markup accessible and usable by human and computer alike. Serialization/Deserialization is on par with or faster then XML, Thrift and Protocol Buffers.”
That seemed a bit too definite for my taste. There are so many variables that can affect the results that I was interested in more information, so I asked for it and eventually got an answer.
I had a brief look at the benchmark referenced and that was enough to come up with some talking points. To make it easier to follow, I will just compare protocol buffers and json (jackson). I started by running the benchmark in my machine (java 1.6.0_14-ea-b03):
Object create | Serialization | Deserialization | Serialized Size | |
---|---|---|---|---|
protobuf | 312.95730 | 3052.26500 | 2340.84600 | 217 |
json | 182.64535 | 2284.88300 | 3362.31850 | 310 |
Ok, so json doesn’t seem to be faster on deserialization and the size is almost 50% bigger (a big deal if the network is the bottleneck as is often the case). Why is serialization of protobuf so slow though? Let’s see the code:
public byte[] serialize(MediaContent content, ByteArrayOutputStream baos) throws IOException { content.writeTo(baos); return baos.toByteArray(); }
How about we replace that with content.toByteArray()?
Object create | Serialization | Deserialization | Serialized Size | |
---|---|---|---|---|
protobuf | 298.89330 | 2087.79800 | 2339.44450 | 217 |
json (jackson) | 174.49190 | 2482.53350 | 3599.90800 | 310 |
That’s more like it. Let’s try something a bit more exotic just for fun and add XX:+DoEscapeAnalysis:
Object create | Serialization | Deserialization | Serialized Size | |
---|---|---|---|---|
protobuf | 260.51330 | 1925.32300 | 2302.74250 | 217 |
json (jackson) | 176.20370 | 2385.99750 | 3647.01700 | 310 |
That reduces some of the cost of object creation for protobuf, but it’s still substantially slower than json. This is not hard to believe because of the builder pattern employed by the Java classes generated by protocol buffers, but I haven’t investigated it in more detail. In any case, protocol buffers is better in 3 of the measures for this particular benchmark.
What does this mean? Not a lot. As usual, where performance is important, you should create benchmarks that mirror your application and environment. I just couldn’t let the blanket “json is on par with or faster than…” statement pass without a bit of scrutiny. ;)
Hi there! You might want to contribute the PB patch to the benchmark project — I think they’d be happy to take it. That’s how other codecs have been improved.
Hi Cowtowncoder,
Yes, I try to be a good open-source citizen. :) I had filed an issue with a patch earlier today:
http://code.google.com/p/thrift-protobuf-compare/issues/detail?id=2
Best,
Ismael
I was about to commit the fix when realizing that it may not by so fair to the other serializers. There is a hidden cost in the loop where we do ByteArrayOutputStream::reset(), I’ll try to take it out of the equation.
Hi,
I added the fix, but to be fair to the other serializors I also removed the ByteArrayOutputStream reset out of the time measurements. Here is a snippet from the results (didn’t publish them yet):
Object create,Serialization,Deserialization
protobuf 492.90000,325.60000,444.75000
json (jackson) 289.05000,324.55000,429.00000
Hi Eishay,
Interesting, I updated from SVN, but I get different results:
Object create, Serialization, Deserialization
protobuf 302.05880, 209.97715, 244.49875
json(jackson) 141.42370, 225.06970, 344.70990
What JDK are you using? Also, are the results repeatable?
Ismael
Regarding the ByteArrayOutputStream.reset, I am not convinced that the latest change not to include it in the final time is correct.
In normal usage, you’d have to reset it or create one, right? The fact that protocol buffers does this efficiently internally should be rewarded (or a similar approach should be used in the various serializers).
Ismael
I changed the benchmarking to have the serializer create its own stream if it needs to. It doesn’t make mush of a difference. The other thing I did was to create a new object to serialize every time (not reuse the same one). The latter dramatically reduce the performance of protobuf serialization. I’ll write about it more once I’ll have time. The code it checked in, if you have a chance please let me know if you think its fair for all serializers.
I replied in Eishay’s post.
Eishay: I wish I had seen comments earlier.
Please note that ByteArrayOutputStream.reset() is practically free: all it does it does it set pointer variable to 0. This is much cheaper than allocating a new stream.
I know that many other JDK classes do some cleanup, like ArrayList clearing up entries when size is set to 0. But BOS doesn’t (I checked JDK sources to ensure that).
Of course, it probably does not make that much difference like you say, so maybe it’s all irrelevant. :-)
I saw it too, but it is not the same thing as reusing an existing memory block over and over (as you do in rest) instead of allocating new memory each time and throwing it to the gc (as protobuf toByteArray does). Not arguing about the efficiencies of the modern gc with young objects in eden etc, just saying it is not the same cpu/memory usage profile.
ByteArrayOutputStream.toByteArray() also does make a virgin copy. So I think it is doing about the same as what PB does in the end.
But if BAOS is re-created each time, it is doing allocations twice. This may be exactly what PB does too.
In the end though I agree in that it doesn’t matter too much, either way, as long as BAOS is not created way too small (which it isn’t with current test code).
Hi, this is really nice post i like it so much
thank very much