At some point during my journey I needed to start storing my 3D objects into some kind of persistent store. Because I’m dealing with a lot of data I quickly found that the file system (keeping each model item in a separate file) was just too slow. File IO is slow and even buffered reading proved sluggish on a per-object basis.
Next choice was a hand-rolled paged file system which crammed multiple objects into a single page based file. This was many times faster since the file was opened once per session, and stayed open. However it was a bit primitive.
Lastly I settled on SQL Server Compact Edition which is an in-process database which is vaguely SQL like, but probably based underneath it all on the ancient Microsoft Jet engine that they pilfered from FoxPro when they bought it way back … The API is simple and the speed is very good – as it should be considering I’m just reading and wring objects by primary key.
A future alternative may be MongoDB or perhaps a hand rolled spatial index store, but for now the SQL CE will do … but they’ve gone and deprecated it now, so I’ll have to keep an eye out for a replacement.
As to the technology of choice to transform my .net classes into byte streams for storage there are many choices for serializing and de-serializing for in-application use;
1) The built in BinarySerializer. Rock solid but slow.
2) Various kinds of XML Serializer. Ok but very verbose.
3) The “Gold Standard” of ProtoBuf.net (http://code.google.com/p/protobuf-net/) which is both the fastest and the smallest of all the serializers. However it is very selective in terms of what is will handle and requires an up-front investment in a data type catalog for it to run efficiently. Still – its very, very quick.
4) My current choice is the popular ServiceStack Text Serializer which will serialize all .net classes (but not structs – you have to do those yourself) into a JSON string format, which is only slightly slower than Protobuf and has the benefit of being able to be debugged (its all just text).
Find the authoritative performance comparison here.
