SAFEpress: A Web App that builds websites/blogs on SAFE Network

I’ve fired in some code this morning. And I’ve got some questions re: how we might structure this stuff (and I guess more will follow once we realise what’s possible).

I made an issue on the repo for discussion there Decentralised CMS structure · Issue #3 · happybeing/safepress · GitHub

Feel free to chime in over there, or here if this is perhaps a simpler / better platform for this sort of discussion. (I wasn’t sure where would be best).

I’m not the man to feedback on client side framework choices, but :slight_smile:

If you are going offlinefirst, checkout the offlinefirst frameworks (and nobackend.org), not least PouchDB which we’ve been discussing.

My plan though is to store directly on SAFE if at all possible, using offlinefirst only as a secondary “bonus” optional feature: ability to work without internet, and then sync when internet available.

I realise this may not be achievable, but finding out is the priority, because only then do we know what we’re building, and can look at which tools to build it with.

This is what my first target of a simple hacked together blog is about:

  • it’s based on REST UI (in CouchDB for now, but drop in replaceable with SAFE REST as soon as available)
  • it doesn’t matter what we build it with, we learn what data structures we need and what to do with them, and how this can (or cannot) be done with SAFE REST. Maybe we learn SAFE REST is not enough, and need to extend the plugin to get at more of the SAFE API, I don’t know.

EDIT: This might be based on klerks or another CouchApp blog, which is why I was interested in evaluating those. Did you get around to looking at klerks or the other CouchApps I sent you?

I think this provides the answers we’ll need in order to decide what we’re going to build, so it should come before choosing the actual frameworks. Investigating options is valuable though, so I’m not saying don’t discuss, or don’t try then out. If that’s if interest to you, by all means follow your interest.

I just want to make sure I’ve explained what my thinking is adequately, and that i think it is too soon to say what we can or can’t store on SAFE. We’ve barely started that discussion!

Props (still don’t know what that means so hope it’s appropriate!) to @joshuef for the:

[applause]

Guys, I am so happy that you are building it. Your work is both very important as it will demonstrate that the SAFE network is far from just being a storing system and because it’s something completely new that has never been done before.

Thanks a lot!

3 Likes

That’s great feedback @gmason. It means a lot to get this, especially from the SAFE communitiy which myself and our embryo team know so well, and respect. :hand: (That’s a high five not “stop” :-))

@happybeing, I totally agree offline first is the way to go.

Pouch/Couch etc are great, but in the end these frameworks are all built around the same, sync to server thinking. We can for sure use an in browser DB (or abstraction: Pouch), and save this locally for offline first, and then sync this via data retrieved, but the bigger questions for me are how we manage that data on a distributed network; who own’s what, and how is that updated / managed /synced without a server?

Kleks: I couldnt get running locally quickly. In the end I figured from reading around: it’s another CouchDB CMS, it needs to sync. I also found keystone.js etc which looks more promising and has a better ecosystem. Either way, hacking something like this to work with a distributed system would take longer than just rebuilding a UI, IMO (We can start small and get more complex as we have a working platform).

I agree something as RESTful as possible is desirable. It keeps things simple. And we can do that with our SAFEpress for ease of interaction, but it’s not our main stumbling block.

The bigger questions around ownership of data/posts; and synchronising that around SAFE / any filesystem, and how people interact with and view the final thing are the hurdles that I think we need to tackle.

So to clarify my thinking: My first aim isn’t so much testing out these various ways of syncing data or an API for adding posts. I’d like to get a basic something in place to test out interacting with a filesystem (halfway there) / what I imagine SAFE to be; and then we can see how many people could interact with and update such a site.

re: nobackend stuff; right now I’m looking at Firebase as placholder for SAFE itself. It offers free online DBs and many ways to interact with them, which could allow us to set up a couple of DBs and see about having an instance of POC-Safepress and see if we can get it to load all posts, no matter how many different Firebase users are involved.

This project is very exciting to watch unfold. Thanks to all involved. @happybeing your work and enthusiasm from day one has been a joy to observe. My skill set does not include development so I am grateful to all who are moving this vision forward. Everywhere I’ve explored anything maidsafe I’ve seen your fingerprints. So, thanks again @happybeing

1 Like

Seems to me that the fascinating part about this project compared to Wordpress is that in essence we can build one installation for the entire world to use and one database to hold the all of the world’s SafePress posts. Each user will be logged into SAFE, and thus they will be logged into SafePress-- Although we will probably need to manage aliases for them… As in some cases people may want to remain nearly entirely anonymous, and in other cases they want to be quite verified and unforgeable. If Satoshi wants to comment on the Block size debate we want to know it was indeed Satoshi even if we don’t know who Satoshi is… If Satoshi wants to post on such a debate without revealing his identity, it ought to be his right to be whomever he wants to be without the identities being traceable to one another. I suspect this function would have to live at the SAFE level, not the SAFEpresss level?

I would think that comments would belong to the comment author, but the owner of the blog that the comment was attached to would have the option to display the comment or orphan it?

1 Like

Hi @joshuef, we agree on the imporant issue, but I’m still not sure I’ve adequately explained my thinking so let me know if there’s anything puzzling about my reasoning.

I do agree that any existing CouchApp will assume extra functionally that is not in SAFE REST making it a significant task to port at least the first one, but that’s not the only reason I’m thinking CouchDB is a good choice.

I don’t yet see an advantage in using Firebase over CouchDB, though this may be the case, so I’d like us to examine the pros and cons before you home in on Firebase.

And if anyone reading this thinks there’s another, even better way to go for what I’m outlining here, please say cos right now it’s easy to switch.

Aims

Also, I agree that the priority, and biggest challenge, is to work out what data structures to store and how operate on them with multiple users.

Frameworks

Firebase and CouchDB are roughly equivalent in some areas we care about: both can easily substitute for SAFE REST, both can be run locally for development, and both are easily available as services if we want to open that out to multiple users (although I would expect SAFE REST to be operational before we are in a position to do that, so I’m not sure that will matter).

I don’t know Firebase at all, so there may be things that mean it’s a better choice for our needs, but for now, I’m only aware of plus points for CouchDB.

Server Side Features = Simpler is Best

Our architecture is essentially noBackend = very simple server side functionality.

So for playing with and adapting, we want something as close to this as possible, which means a DB with very little cleverness in the server, and a client app framework that expects little from the server.

CouchDB is very simple on the DB side - it’s server functionality is very limited (because it’s main selling point is doing sync better than anything else). So it’s web app framework is close to what we need, or at least closer than a framework built to talk to a cleverer database server.

So if Firebase is cleverer on the server side than CouchDB, that doesn’t help us, and would mean any client side stuff will expect more from the server and be harder for us to make use of.

So we need to understand how Firebase/CouchDB compare in terms of what each of their client side frameworks/apps expect from the server - as little as possible being the ideal.

CouchApp

The other aspect of using CouchDB that attracted me was, as you know, CouchApp.

CouchApp is a command line interface that helps build and deploy web apps. With one command you can create a template web app that can be deployed to a CouchDB with another single command.

We could fork this to do the same for a SAFE web app framework. So with two “SafeApp” commands a web dev could have a test web app deployed and working on SAFE!

EDIT: And to work locally, they just install CouchDB and deploy to that.

Also, there are many simple CouchApps that we could use for inspiration, or port to SAFE to create a nice suite of examples to inspire other devs.

Firebase Question

I haven’t looked at Firebase, so maybe it can do these or other things that give it advantages we should consider. I know @Viv suggested it would be useful for this kind of project, so maybe you and he could say more about this if you think it has advantages for what we have planned for SAFEpress?

So let’s have a discussion of this, why Firebase rather than CouchDB? before proceeding.

BTW Firebase is Mozilla, and CouchDB / CouchApp is Apache. I’m not sure if this has any bearing.

@janeannford thank you very much for saying so. [big grin]

Morning!

CouchApp seems to be to have the drawback of needing this install step. I see what you’re saying about just a command to get something going, but it’s another command.

I think Firebase and something like IrishCouch (https://www.iriscouch.com/service) are broadly similar. Firebase still has a rest API and syncing, which is good.

Firebase also offers a lot of functionality to us as well as REST, it’s got AUTH methods and deeper javascript APIs too (although I’ll be avoiding this in favour of KISS at this juncture).

Thus far the main selling point is that it’s super easy to sign up and free for a development account, which means that we could get a couple of them up and running easily and actually test our distributed site out (I hope! I’m planning to have a little play now so lets see).

In the end, I’d hope setting up SAFEpress could be as simple as clonging the SAFEpress app and running our file in the browser. This would then ask for your DB/SAFE url to start with. If it exists, it should sync site info (if you’re authorised), if not, it’ll set you up a new site ‘DB’ (/manifest).

Anywho, at this stage I’m not planning to go deep into anything, so we can switch out the technology stack to something else at any time. I reckon I’ll add a couple of calls to REST and that’s it so that our options stay open and the only difference between Firebase/CouchDB would be the URL. so the system we’re using wouldn’t matter at all (nobeckend! :smiley:).

Hmmm. And then I wonder: Maybe a more accurate SAFE approach is to do data writes to Dropbox and share links to those public folders.

@joshuef This is a misunderstanding of what I’ve said about CouchApp. There’s no need for an install step for SAFEpress. There I was referring to a developer publishing a new app they built using tools we spin out of the SAFEpress project into a SAFE web developer toolkit. I’m sorry if that was not clear above. Let me clarify.

This project can deliver two products, which I was outlining above, but not clearly enough:

1) SAFEpress: an app we build, we deploy, and which prospective website builders access and use to create websites. As you say, there’s no need to install. BTW My intent is to use CouchDB/Firebase purely as a stand-in for SAFE REST. If we have SAFE REST fully working, and we might very soon, there’s no need for a Firebase/CouchDB instance at all. However, right now we’re choosing something to work with in case it’s not ready, and I favour CouchDB because of 2), below.

2.) SafeApp: a web application developer framework. For example: Front-end web developer “Dreamcode” API for general purpose web app development (as built for, and used by us, to build SAFEpress). Think something like Hoodie. Plus a tool based on the CouchApp command line tool I described, that a dev (not a SAFEpress website owner) can use to help rapid development and deployment of his own web apps.

I don’t think you’ve understood this, and so aren’t getting why i am still advocating CouchDB. I get that you like Firebase, but in the context of 1 and 2 you haven’t given me a reason why Firebase is preferable. My point is that i think CouchApp/CouchDB are particularly suited to doing both 1 & 2, I’m not just considering SAFEpress.

Firebase v CouchApp/CouchDB

Note, CouchDB can also be cloud hosted, see IrisCouch, so this is not a differentiator. But in all probability we won’t need that anyway. We’ll likely have SAFE REST on a live public network by then. If not, then either would suffice as a stand in for SAFE REST if all we do is build SAFEpress. However, I want us to build 2 as well!

You mentioned extra features in Firebase, but I’ve explained why I regard extra features in the backend server as not useful, but in fact a disadvantage because of 2.

It’s Not Just SAFEpress!

This is what i think i have not yet conveyed - hopefully this post will do it.

My thinking is that by using CouchDB/CouchApp as a starting point, or for inspiration by studying and/or adapting any useful bits, we can do not just SAFEpress, but also create a way for other web devs to be creating their own apps without any need to learn about SAFE REST. They can just use the dreamcode style javascript API we’ll be creating (with front end devs in mind) while we build SAFEpress, and we give them a handy tool to create their ready to customize template app, and to deploy to SAFE (or to a local CouchDB if they want that option during testing).

Why Create An API For Others Too?

If you would like to install and try out CouchApp, I hope it will become apparent how good a similar toolkit would be for SAFE Network. It will empower any and every web app / front end dev with the ability to get something they can test, tweak, and use to build brand new apps on SAFE, with existing HTML/CSS/Javascript skills, and no need to consider SAFE REST or any backend. It’ll be bloody brilliant!

For SAFE Network, it’s as big as SAFEpress itself, so we could get two amazing projects for the price of one and a bit! :slight_smile:

Please check out CouchApp and make sure you understand what I’m saying here, because i think there’s been an ongoing misunderstanding about what I’ve been proposing over quite a few posts now, and I have so far not managed to clear it up. We really need to get this clear before proceeding or it’s just going to be a waste of your time building something different from what I’m envisaging.

Thanks for your patience and persistence. I bet you just like to get into some code :-). Soon!

@happybeing Okay, I’m hearing you :smiley:

I think I’m not being clear. So I’ll reply and then clarify my intent with current testing:

  1. Yup! Firebase/CouchDB/Hoodie. Everything is a placeholder. As such, anything I’m doing is only loosely using the service and could/would be stripped out easily. (Or we’d rebuild using any learnings once we know some safe architecture)

  2. Something to enable easy app generation! Yes. This is also a great thing (but no, I hadn’t understood you were also talking about this as a part of the project’s scope).

I have to say I haven’t checked CouchApp as yet as when I go to github (GitHub - couchapp/couchapp: Utilities to make standalone CouchDB application development simple) and try and access tutorials or FAQs to get a better idea everything is 404ing, so for now I left it out of my endeavours in favour of testing some ideas.

As I haven’t tried it yet, I can’t comment or see exactly what it’s benefits are (but I will try and get to trying it out soon).

And to be clear about what I’m currently exploring: I’m not looking for a specific methodology or setup for SAFEpress. Firebase/couch etc, would all be stripped out anyway. What I want is to explore how a distributed site could interact / be built such that everyone owns their own content. Any development I’m doing is fast and loose, and I’m not intending us to commit to anything structure / API wise. Anything I’m using is just for ease of getting setup to explore.

I’m doing it as a learning exercise in distributing the content more than anything concrete (and yep: doing is learning for me :smiley: as you noted :P), so there will be no time wasted even if we throw it all away and rebuild! as at the end I’ll have a better understanding of how this stuff can be handled and how we can apply that to SAFE.

And just to sign off: I’m totally down for helping with both 1 & 2! I agree that they’re both important. And I hope I cleared up a bit what I’m hoping to achieve with current tests. (also if you think it’s better to keep these out of the repo, then that’s fine, i can keep this stuff forked)

Thanks!

1 Like

Data Structures/Ownership:

So in my testing around this, I’m beginning to think that even if we’re on about a curated blog/site etc, if (as noted by a lot of people above) we’re working in that you own your content etc, that perhaps SAFEpress should first and foremost manage your ‘SAFE’ site presence and contributions.

Imagine that everything you do in the realm of SAFEpress is a post (an idea lifted from WP). Be it writing a post, commenting, uploading a picture etc. So my default you’d have a collection of your content stored locally, which youd choose what to do with. Which means, by default you’d have a blog / newsFeed of your online SAFE activity (posts, pictures, comments etc… essentially your facebook page; whether you put this online in any guise would be up to you).

This would allow

mySAFEpressData: :[ /*array of posts you made*]

Each ‘post’ (or comment etc), could be standalone for you, or could reference a location or ‘site’ on the network (which would also be updated on ‘save’):

posts: [ 
    post1: {
        title: 'MyAwesomePost',
        linkedPresses: [ /* Array of SAFEpress sites that use this post */ ]
        other things : 'any properties we want'
    }
]

This would allow for a lot of flexibility. And when you write to your own storage area, you’d simply declare what site you’d like to write to if not your own (and assuming you’re an authorised contributor), and the reference is written both to your storage, and The Other Site.

This Other Site, would have a manifest of contributors, and all ‘posts’ for the site would be requested from the requisite SAFE folders on the network.

We’d have to build out functionality for site admins as well, so posts can be edited/checked etc. And then there’s the question about whether the site page, as it is in the end, would do these requests on the fly (slow over a traditional network with a lot of contributors), or if we’d generate static output etc, or a mix of the two (for commenting and so forth).

I’m aware people have mentioned these ideas above, I’m trying to get a bit deeper into how this might actually be split up and be useable. I am beginning to think that an individual approach first, with site curation and editing areas being an extension of this might be the most flexible approach…

@happybeing @mvanzyl thoughts?

(I think a basic facebook like ‘site’, that can be written to my many contributors / you update other peoples’ sites would not be insanely complex in this method… does anyone know: is that how diaspora etc work? )

Anyway. Semi-specific thought splurge over. If anyone has thoughts here-in just chime in below!

1 Like

I love following your development and really appreciate you continuing the open spirit of MAIDSAFE. It is already providing a great road map to what direction I need to go. Since there are lots of discussion on db’s, I wanted to see if you guys had any experience with NoSQL M/GT.M/MUMPS? It has been around before the internet (much different forms), but it was originally designed for a decentralized health record program.

My question is partly selfish because if you guys figured out how to get GT.M running on SAFE then my world just got a whole lot easier.

This is what I am working on:


That seems like a great baseline to start with. I was thinking in the same vein there, having the option to have the data saved to either a local device, a user’s private storage on SAFE, or their public storage on SAFE would be great. Depending on how social media crazy some people get, having locally stored data may get burdensome as far as storage goes (How am I supposed to store my bazillion selfies locally!?!..lol!) and if you are pushing the data that you want to post publicly to public SAFE storage, that would be free anyway… (am I right on this?) Plus users could earn a little coin on the data as opposed to keeping it locally…just a thought… Anywho, great stuff @joshuef !!!

Here is an outline of some different database options to debate…may be a very helpful reference.
There is also a reference at the end of this article to some fantastic reading on database structures (MongoDB, etc) along with full text search functionalities/engines/options to consider. I reached out to the author to see if he would be interested in collaborating as well!!

1 Like

@joshuef / @mvanzyl

I’m commenting after we’ve clarified that my aim (at this time) is not decentralised ownership and creation, but the current model: a website curated and owned by someone, who actively curates the content. We are not building a decentralised facebook/twitter/Youtube or diaspora, though they would be great projects I’d love to see. My reason for keeping to a centrally owned and curated website model is: 1) I think it is needed on SAFE alongside decentralised systems, and 2) I have not figured out a way to do decentralised ownership, while automatically integrating it into a single website (like diaspora), and I don’t want to try at this point. My model for SAFEpress is WordPress

So this is not essential, but I think it still makes sense. I see people who use the SAFEpress UI to create content (whoever they are: visitor/subscriber, author, admin and owner) will create content that is stored and edited in their own space and remains 100% under their control forever.

When they want to submit this, they send a request to the desired website (using the SAFEpress UI) which will arrive as a message. At this point nothing has been changed on the website.

The owner/admins of the website receive and review these request messages (in the SAFEpress UI), and if they choose to accept the content to the site it is published, causing a copy of the content to be stored in the site owners SAFE storage, and for the indexes that are used to display the public website to be updated. So anyone visiting the website from this point on will see the updated content. This can be applied to posts, pages and comments (using blog terminology). It gets around the problem of how to have a commenter post a comment to a website when there is no server to accept it and update the site index. This now happens only when the owner/admins review and accept a post/edit/delete request.

It means everything has to be reviewed and ok’d for now, but that’s fine until we figure out a way to do this with decentralised ownership. If anyone solves that one, let me know! :slight_smile: I think it will be possible, but not until the full magic of Structured Data has been revealed, or maybe some new feature added, I’m not sure.

I think if we sketch out the basic data structures (say pages, posts, comments), the queries needed to generate a typical blog with additional pages, and the indexes that would be needed to generate these queries, we can then build a mockup to show what’s possible. This could be HTML & Javascript to query the indexes, get the content and generate the pages, with the content put together manually, or even automatically transferred from a WordPress SQL database into the database structures we’ve designed. That can live inside CouchDB initially, and once available we move it over to SAFE REST.

I don’t know how to solve this one, and I don’t believe we should try at this point because we don’t need to. We can always use our learning to build something like this later, although I think producing the WordPress / CouchApp of SAFE Network might be a big enough task for us :-).

(BTW diaspora doesn’t give users control - look for remoteStorage apps for that. On diaspora there is a central database owned and controlled by the site admin. Diaspora let’s people control their content, but they have to trust the admin, they don’t actually look after their own data. remoteStorage is about storing data on a user owned service, typically a cloud service rented by the user, or a server they own for the technically adept.)

@mvanzyl Thanks for the NoSQL article. I will study it and hopefully something will be learned!

@MrAnderson I’m sorry, that’s something I don’t know anything about. SAFE is my first NoSQL and I didn’t know that much about SQL either.

2 Likes