IMO, having some high level diagrams and architectural overview documents goes a long way. Much of this is provided in the primer, which is an excellent resource.
Of course, documenting interface points is critical and the last time I was writing code for safe network, there were some good docs already for this.
In short, devs need a ‘way in’. They need to understand the design at a conceptual level and they need to be able to connect with the interface points. From there, they can start figuring out how things work.
I don’t want to distract the core team, but I also want set expectations of where we are and where we are going. Which even in present tense is still a lot of changing and new concepts being introduced but only documented by being stated in a dev update blurb.
Every time I try to talk about SAFE I will ask: So how does X accomplish Y. Then I get “well we solve it by doing Z and T”. And I will be curious where the solution is in code that implements Z and T. And then I try to find in code or docs etc and its not easy to know where everything lives. Or the concepts that are almost talked about as if they are commonplace and complete, such as malice are actually still TODO and have not been dealt with, see:
for example. It seems lots of core concepts have onions to peel back in dialog to get to the core of how it works and in many cases I struggle to see the high level concept 100% fleshed out all the way to the bone in implementation. I guess that is where my frustrations lie. We have too many high level concepts and I can’t see full v1 implementations in many of the components(UI components are usually self-evident but more talking backend + network components). Hence I want an SLD and tech specs that hopefully link out to source and also gives an idea of whats actually left to implement vs whats been fully baked and doesn’t need touching anymore.
Can you imagine asking how a part of a brain achieves X or how part of an ant allows it to do Y? What I mean is decentralised systems like SAFE will use a lot of interrelated modules to achieve “stuff”. Then somebody says I want to know how it does X and then we have to say well X is done by … which means you need to explain almost the whole system to see each part, but folk want a part as a stand alone module with a sentence to describe that module. Many questions are like this (in my mind), “how do you do (this buzzword)?” and to be honest they are almost impossible to answer in a single sentence succinctly. Kinda like why is the earth the 3rd rock from the sun? then all hell breaks loose as there is no simple answer without a huge amount of info.
As for SLD, I don’t know what that is, I could google, but I am almost certain it’s the latest, inked docs. xml, UML or some similar, " we can describe complex stuff simply". Well IMO we cannot.
OK so saying all of that, rust has a truly brilliant feature and that is the documentation system. Each crate has a docs repo, all published ones are on docs.rs AFAIK, that is where we document each code piece. So I think if questions are not able to be answered there then we perhaps should look at updating that. It’s a start, but I have spent thousands of man hours documenting stuff, our patents for example are on occasion 120 pages long (as opposed to normal 4-5 pages) and I bet nobody has read them, certainly nobody who wants to know how/why (I include our own Engineers), then papers, wow a load, and again same, then wikie, again, primer, again and so on.
IMO folk want docs in their own personal preferred style and then I bet they still don’t read them, especially for large complex systems
Better safe then sorry, unless you want to be responsible for an official release. Might be fine if maidsafe does it in the future… but safer if user downloads an official install disk with checksums from a supported distro like debian or redhat etc and installs themselves… For now.
https://docs.maidsafe.net/ has had no updates for 2 tears. Could/should we help by raising PRs to update that with links to all readme.md and docs.rs files for starters?
Or perhaps link to all docs.rs crates etc. I recon code documentation is best. Then we can focus on adding example usage etc. in those docs. They are then tested as part of CI, so always up to date.
No offense taken, it takes a lot to irritate or anger me best to not let things bother yah, happy to hear the perspective from the other side. Just want to ensure that there is enough clarity to go the distance, I can’t really see it all on the outside looking in, but that is not what really matters for the time being. If you and the full engineering team have a good grip on all the bits then I am content to wait it out and try to put the pieces together in my head and begin combing code when its ready for eyes on it come launch time.
I have designed a bash script file that analyzes the main Maidsafe components and generates a SVG diagram that shows the inter-dependencies between them:
This is a really good start, next steps would be explaining the relationships and concise descriptions in some sort of companion markdown with the auto-gen. Then a further bonus would be breaking out the component features and where the gaps are that need further work. Thanks for putting this together!
Would you be able to put the generation script in a cron, so that it will auto-regenerate eg daily or weekly? Hopefully it would still work if/when dependencies change? Ideally, the team would integrate this or something like it into the build/release process, but I’m not sure if anyone has bandwidth for that right now.
Basically, I’d like to see this become a standard part of our docs, but less useful if the diagram is just going to become stale and is not maintained up-to-date…
I reckon we can get this in CI and as @jeremyjpj0916 says we can add more detail to the descriptions, possibly link to readme.md and docs.rs for each repo? Let’s ask @StephenC tomorrow. He likes stuff like this.
Definitely agree, this sort of thing provides highly valuable marketing (to nerds).
I think the RFCs have served really well in this respect and I’d love to see them continue to be worked on, especially with the latest developments in AT2 and CRDT.
Thank you @tfa. I started working on something like this about a year ago, but put it on the back burner when I ran into some difficulty and then forgot about it. Great Job!
Dependencies for node repositories (safe_browser and safe_nodejs) are hard coded. But dependencies of all other repositories are dynamically evaluated, based on their Cargo.toml files.
The list of repositories to include is defined at the beginning of the script together with the branch to select (currently master branch for all, except for routing which is fleming).
I can add other maidsafe repositories if you like. Or remove some (I am thinking about safe-nd which is used in many places and so creates many arrows).
Or you can fork it to adapt it to your exact needs.
best to not let things bother yah, happy to hear the perspective from the other side. Just want to ensure that there is enough clarity to go the distance, I can’t really see it all on the outside looking in, but that is not what really matters for the time being. If you and the full engineering team have a good grip on all the bits then I am content to wait it out and try to put the pieces together in my head and begin combing code when its ready for eyes on it come launch time.