SAFE Wiki... Wikipedia on the SAFE Network


#1

Please see my original forum post for more context. :smiley:

Without a doubt in my mind, the most important website in the world is Wikipedia. The importance of having free access to educational material cannot be overstated. For many people around the world, it is a luxury that they can only dream of. In the SAFE Network I saw an opportunity to help bring this luxury to those unfortunate few. Since October of last year, I have been working away on developing my application SAFE Wiki. SAFE Wiki is an Electron powered adaptation of the WebExtension Kiwix-JS that can use the SAFE Network to both store and read ZIM files.

ZIM files are magical in that they provide an “offline storage [solution] for content coming from the Web.” Specifically, ZIM files can be used to store Wikimedia and StackExchange content.

Kiwix provide a number of tools/programs to easily browse these ZIM files. Kiwix-JS is just one of many projects that Kiwix maintain. If you wish to know more I highly suggest you visit their website.

SAFE-Wiki brings the functionality of the SAFE Network to Kiwix-JS. Once you use SAFE-Wiki to upload a ZIM file, it is publicly available to any other user of SAFE-Wiki. This means that we can easily publish decentralised ‘archives’ of popular websites like Wikipedia.

When you upload a ZIM file to the network, it get’s stored in what I am calling a ‘ZIM folder’ that lives in your _public container. When you create a ZIM folder, you specify a name. The name is important as it is the ‘name’ of the MD on the network and is how other users will find the ZIM files you upload. This ‘ZIM Folder’ is stored in your _public container against the key ‘zim’. To store and read ZIM files, a ‘ZIM folder’ is accessed through NFS emulation. For example… If you are Wikimedia, you might have a ‘ZIM folder’ called Wikimedia. Within that MD you store a ZIM file called Wikipedia. For another user to access Wikipedia, all they need to do is type in Wikimedia as the target ‘ZIM Folder’, and enter Wikipedia as the filename. It is as simple as that!

The main features of the application work. You can upload ZIM files and anyone can browse them! The functionality to delete a file that you have uploaded has also been implemented. I will be introducing further ‘house keeping’ functionality in due course. Please note there are issues with ZIM files that contain images… A fix for this is my top priority at the moment so I should have it sorted soon.

As some as you will remember, last year at University can be rather busy. The application doesn’t have nearly as much polish as I would like. Since the application is now useable and I have overcome many of the big technical challenges, I thought it best to share it with you all. Ugly bugs and all. My project is due near the end of March, so you can expect to see lots of activity over the coming weeks/months as I fix bugs and improve functionality. There is still much to be done! If circumstances allow it, I will continue to work on this project after I have finished my degree. I truly feel that SAFE-Wiki could be a useful tool to many people around the world.

I STRONGLY recommend that you use mock routing to test the application if you so desire. I have done very limited testing with the Alpha-2 test network and I don’t feel the application is fully ready to be used on the public network. Completely up to yourself though if you wish to live dangerously :wink:. You can download ZIM files from Kiwix (http://wiki.kiwix.org/wiki/Content), They have a vast array of different ZIM files for you to choose from. Please note that without any configuration of the SAFE Browser you have a file size “limit” of ~1GB, so please use the smaller files if you wish to avoid the trouble. If you are going to download ZIM files, please try the BitTorrent links first… Reduces load and cost on their end. I have been working with the SAFE Browser so far, not sure how things function with Peruse so please let me know how you get on if you go down that route. Development has been entirely on macOS, so if you have OS specific bugs then please let me know and I will have it fixed before I hand in my project to the Uni! As this is a final year project at University, I have to be careful about seeking help from outside sources. So if anyone submits any pull requests for features or things like that, please don’t be offended if I don’t do anything with them until after I have submitted my project/dissertation.

Please find the source here.

Full instructions on how to run the program are in the README, if you have any trouble then please don’t hesitate to ask. I did put the instructions through the ‘rubber ducky test’ but I still might have left some details out. Very excited to hear what you all think. :grinning:


A ZIM file being uploaded.


Browsing Wikivoyage. Note there are issues with ZIM files that contain images, hence why this doesn’t look very pretty. Fix coming soon!


Wikispecies. This ZIM file has no images so it displays perfectly.


Glasgow University Students to Explore the Future of Internet Technology
#2

The apps are coming thick and fast now! :sunglasses:

I haven’t tried this yet, but it sounds like the basis for getting Wikipedia on SAFENetwork. That would be super cool! :slight_smile:


#3

I’ve managed to get it working on Windows without a hitch which is something. Nothing ever compiles on Windows so I count that as a victory in itself. The app works fine. I can read zim files stored to on my hard drive but can’t work out how to use it with the mock network - not helped by the fact the web hosting manager doesn’t seem to be talking to the mock browser for some reason. Can you give us an idiot’s guide? :clown_face:

Nice work by the way - keep us updated on your progress!


#4

Seems to be hanging at the authentication stage although there are no obvious errors.


#5

Absolutely! This is brilliant @DaBrown95. This app is one of the most anticipated, so good on you for getting this done. It’s looking great already imo, I’m going to try and give it a go this weekend. Can’t wait to see what it will look like polished :smiley: Bravo man, bravo :clap:t3: I’ll be sharing on Twitter


#6

Very nice! That momentum rise we all spoke of a few months back is here. With data chains coming into place the flocks are going to storm this forum. Can’t wait! :grinning:

Q: any idea of how large the current wikipedia repository is on the clearnet?


#7

Hello !!

That is an amazing project, one of which give the Safe network a true meaning.
At some point it could be an idea to tell Sci-Hub people about it ?

Build and launch went flawlessly on linux x64 fedora 26.

I can open zim files and browse them without issues.
When trying to connect to the mock network, the app fails with :

Requesting auth

network.js? [sm]:60 Uncaught (in promise) 
E {message: "Failed to load native libraries: Error: Dynamic Sy…libsafe_app.so: undefined symbol: test_create_app", stack: "1↵    at module.exports (/home/d2r2/archives/maids…-wiki/node_modules/jquery/dist/jquery.js:4991:28)"}

If I try to connect to the real alpha2 network, it just hangs there with :

Requesting auth

#8

Awesome! I have kiwix on my phone! For starters bringing snapshots of Wikipedia to safe (and with them free knowledge to all the world) that would be an awesome start before Wikipedia itself decides to utilize the safe network

Ps: @Stark German Wikipedia with pictures is roughly 25gb … English will be a little bit larger I guess xD


#9

Thank you so much for trying it out! :grinning: If it is Windows you are using it looks like it is an OS specific bug. What happens is SAFE-Wiki sits and waits for a response from SAFE Browser to say, “You are authenticated now!” That response isn’t being heard so it will be an issue with the way I listen to the ‘auth-response’ on Windows. Will fix that soon. Thanks for discovering it.:grinning:


#10

Thank you very much for the kind words. :smiley: Let me know how you get on!


#11

The ZIM file for the full English Wikipedia is >75GB last time I checked! The ‘real’ Wikipedia is obviously much larger when you start to include revision data etc.


#12

I had never even head of SciHub before… Interesting suggestion! :smiley:

It is great to hear that builds are running smoothly!

Hmm okay, that is a weird error! I’ll try it on Linux soon and see if I get the same error. Might be an OS related bug. I have seen errors like that before being caused by the version of Node, any idea what version you are using?

Thanks for trying it out!


#13

Kiwix on mobile is awesome! Yeah that was a thought that I had. Hopefully can bring SAFE-Wiki to mobile once the mobile authenticator is up to scratch :wink:


#14

Yes - not needing to have 75gb of free space for using Wikipedia on your phone would be awesome xD


#15

sure, this is v8.5.0


#16

+1 for Sci-hub, would love to see that sort of thing blossoming early on on Safe


#17

Just thought I would post a small update… I have fixed the issue with ZIM files containing images.


Here is Wikivoyage displaying correctly.

The Ayr page displaying correctly… You can check above to see how this originally looked.


A page from the Coffee Stack Overflow.

I now successfully have my application running against a local SAFE Network instead of using Mock Routing! It was amazingly simple to get working. I am using four vaults on my local machine at the moment, but I intend to scale this up to use multiple machines on my local network. It is very exciting to see it working for ‘real’ instead of using Mock Routing!

I will be addressing the problems you all have been having very soon. I haven’t forgotten. :grinning:


#18

@nice @JPL
After the fixes in both…


The application should work fine on both Windows and Linux! I have just tested the application running on: macOS, Windows and Ubuntu. Both packaged and running through the command line, all works fine on my end.

Please don’t hesitate to let me know if there are any more issues.:grinning:


#19

Your latest commit fixed connecting to the live alpha2 network, thank you !

On the other hand, when using mock, I still get :

Requesting auth

network.js? [sm]:60 Uncaught (in promise) 
E {message: "Failed to load native libraries: Error: Dynamic Sy…libsafe_app.so: undefined symbol: test_create_app", stack: "1↵    at module.exports (/home/d2r2/archives/maids…-wiki/node_modules/jquery/dist/jquery.js:4991:28)"}

Did someone upload something on alpha2 ? I tried my luck at a few combinations for collection name and filename, but just got some ‘requested data not found’ so far :slight_smile:


#20

That is what I like to hear. :slightly_smiling_face:

Hmm that is strange… Try a combination of deleting your node_modules and running npm rebuild. I am using Node v8.9.0 and having no issues so maybe try updating your Node version.
An alternative if you are so inclined is to run your own local SAFE Network, if you can connect to Alpha2 you will be able to connect to that. :smiley: