Serialization of 3D Models

At some point during my journey I needed to start storing my 3D objects into some kind of persistent store. Because I’m dealing with a lot of data I quickly found that the file system (keeping each model item in a separate file) was just too slow. File IO is slow and even buffered reading proved sluggish on a per-object basis.

Next choice was a hand-rolled paged file system which crammed multiple objects into a single page based file. This was many times faster since the file was opened once per session, and stayed open. However it was a bit primitive.

Lastly I settled on SQL Server Compact Edition which is an in-process database which is vaguely SQL like, but probably based underneath it all on the ancient Microsoft Jet engine that they pilfered from FoxPro when they bought it way back … The API is simple and the speed is very good – as it should be considering I’m just reading and wring objects by primary key.

A future alternative may be MongoDB or perhaps a hand rolled spatial index store, but for now the SQL CE will do … but they’ve gone and deprecated it now, so I’ll have to keep an eye out for a replacement.

As to the technology of choice to transform my .net classes into byte streams for storage there are many choices for serializing and de-serializing for in-application use;

1) The built in BinarySerializer. Rock solid but slow.

2) Various kinds of XML Serializer. Ok but very verbose.

3) The “Gold Standard” of ProtoBuf.net (http://code.google.com/p/protobuf-net/) which is both the fastest and the smallest of all the serializers. However it is very selective in terms of what is will handle and requires an up-front investment in a data type catalog for it to run efficiently. Still – its very, very quick.

4) My current choice is the popular ServiceStack Text Serializer which will serialize all .net classes (but not structs – you have to do those yourself) into a JSON string format, which is only slightly slower than Protobuf and has the benefit of being able to be debugged (its all just text).

Find the authoritative performance comparison here.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s