Introducing SAFE EventStore (event sourcing database)

I don’t remember the exact phrasing, but David Irvine mentioned some boxing analogy about the patience being the important part, winning the long game.

So, we have another update :slight_smile:

Actually, there will come a day in a not too distant future, when I can actually spend more than a day here and there a month on SAFENetwork related coding. We will get there.

The big news for this update, is that SAFE.EventStore is now upgraded to be used with safe_app_0.6.0 and the latest csharp bindings!
This Saturday I refactored the interaction with the bindings and did some final cleanup and testing yesterday.

(SAFE.EventStore repo is here)

There’s been a lot of nice simplification and elegancy introduced with the new csharp bindings so it’s getting easier to work with. Very nice! I like that there are now less static classes and more instantiation, like I had suggested once earlier (I don’t know if my call was heeded or if it was already thought of).

Additionally for the EventStore, I have increased the capacity of the db. I wasn’t happy with 1k dbs, 1k stream categories and 1k stream instances per categoy.
So I introduced some sharding by non-cryptographic consistent hashing.
Each category will have almost 1k shards (minus a couple reserved entries per MD for other stuff) which each hold 1000 streamkeys, instead of each category holding the streamkeys directly.
So far, I have just multiplied stream instance capacity by 1000, but I will do the same to the events.
In effect, we will then get a total of 1000 trillion events per database id, and 1 million trillion events per safe account (so, with total of 1k dbs).

I will have to see how the performance is when having any substantial number of events in there though… I mean, the sharding still give fast key value access, but if I would ever need a full scan, then that would probably be completely insane. But we’ll get there :slight_smile:

So if you have an app (1 database id) with expected life time of 10 years, you could fill it with more than 3 million events per second, and still not fill the quota during its lifetime. So… it will take a lot of devices to be able to push that many events to the network per second :upside_down_face:

(I guess, eventually, I’ll have it expanding to be “unlimited” by some nifty way.)

There’s still garbage collection of delegates in the csharp bindings, which causes NullReferenceExceptions. This happens inevitably after doing some writing or reading of data. But I think the major bug that was causing me problems with safe_app_0.4.0 is now gone, and that is good, because it was an inexplicable one. Had no idea what was causing it. The garbage collection of delegates is well defined, even though I don’t currently know exactly how we should solve it.

So, that’s that for now. As soon as the garbage collection of delegates is solved, I think we have a functional (albeit very simple) eventstore for SAFENetwork :slight_smile:

32 Likes