Idea for how a YouTube-like app could deal with different formats


#1

I think this idea isn’t the best solution, and will never happen because of that. But just in case I’m wrong…

What if everyone using a YouTube-like app to upload content only uploaded one version, like with actual YouTube? This version, say, a 1920x1080 VP9 at 60fps with 96 kHz FLAC, would be the definitive version of the video.

If someone wanted to watch videos in 1366x768 AV1 at 30fps with 48 kHz Opus, they could set this as their preference, and the app could automatically transcode any video they find not in that format and upload it.

This would mean that, assuming the viewer has a mainstream preference of codecs, most of the videos they watch would already by transcoded suitably by another user with similar settings, but occasionally they will be the first transcoder. This first transcoder would unavoidably have to bear the PUT cost of uploading it so that the new video file could be put into the app’s database.

Some users could set the app to transcode even when their preferred codecs are present, and submit the new video file to the app’s database. Assuming the transcoding and upload requirements are strict enough, this new video file would be bit identical to previous users’ uploads, and therefore not incur a PUT cost. When the user submits this video to the app’s database, the database would confirm that the user does (co-)own the video file, and add 1 to the number of verifications. If a user attempted to upload a version with embedded adverts or some other malicious addition, that version would never get verified by other uploaders.

A transcoded video with ads could be mass verified by a bot network, which is the point at which my idea fails. Maybe a Web of Trust could sort that problem out? Or, maybe the app could accept safecoin payment to transcode the video for the first user requesting the different format, which it would achieve by using that safecoin to pay a verifiable distributed computing network running within SAFE?

As it stands, I think the inevitable YouTube-like app will just default to uploading the original file along with several popular formats.


#2

i like that you think about the different formats and that there might be reasons to use them =)

…to me personally … i wouldn’t see why i should transcode a video (for which i would need to download the best quality file first) to then watch the potentially lower quality video clip :thinking: i could imagine i might upload a video in a lower quality for mobile as well - but not with the same machine that should play it in the end :thinking:

did you have a look at
https://safenetforum.org/t/which-safe-website-project-should-i-work-on/21482
and how the concept is there …? maybe thinking one or two moments more about how to achieve the common goal the best way saves some time later on in the process :slight_smile:


#3
i wouldn’t see why i should transcode a video (for which i would need to download the best quality file first) to then watch the potentially lower quality video clip

Here are a few use cases:

  • A video that you might want to watch many times. Say it’s too big for your screen — why not convert it once and be able to watch it forever, and let anyone else with a similar configuration watch it in the new resolution?
  • You may be logged in to the app on both a locked-down mobile and a powerful desktop. If you’re in the middle of nowhere with rubbish internet, why not get the app to remotely transcode from the desktop? (Then let everyone else benefit from that.)

did you have a look at
https://safenetforum.org/t/which-safe-website-project-should-i-work-on/21482

I see no mention of transcoding in this link…


#4

And also the use case that it would be a nice thing to do for potentially millions of users :​)


#6

I believe the best way to go about this (only for large scale demographic catering for multiple devices and OSs) would be for the content creator to make it available for different formats.

Anytime you upload a file there would always be a PUT cost , no matter how small the difference.

I am not too sure how you would go about implementing this as the network (without computation anyways) wouldn’t be able to verify the video is what it says it is, the owner could verify it but there isn’t much of an advantage there for the creator. If the user (not creator or uploader/transcoder) wanted to verify the video was authentic they would need to compare the output of transcoded original file to file saying it is transcoded, but there isn’t much of an advantage there for them either . Another way to go about is have list of people who thought the transcoded/modified was reputable/good/better (a rating system) and then the user or website would be able to make a decision based on whether they’ll use it in place of the original considering the context, however in this particular case I wouldn’t use a rating system like this as I believe having the creator/uploader doing the transcoding makes more sense and I suspect it would be easier to implement.


#7

Ideally people would be able to take a video on a phone and upload it without having to transcode it.

Anytime you upload a file there would always be a PUT cost , no matter how small the difference.

Do you have a reference for this? If so, that’s concerning, but I can see why they might do things that way for privacy reasons.

I am not too sure how you would go about implementing this as the network (without computation anyways) wouldn’t be able to verify the video is what it says it is

The users who set transcode to ‘always on’ would be the verification. It relies on being spam-resistant (Web of Trust?), and users choosing to set transcode to ‘always on’. It wouldn’t even have to be users — a search engine company might be motivated to create a service that automatically re-transcodes random videos.


#8

If there is an already existing Immutable Data then there is no charge as far as I understand it. If this isn’t the case I’d like to know too. Assuming not, the user uploading an already existing Imuttable Data just gets the data map for the file free of charge (thanks to self encryption). So I believe you could have a user upload a video as a Immutable Data object and then name it with a Mutable Data and tie the two together by storing the data map to the Immutable Data inside the Mutable Data. So the user only pays for the MD unless this is a new video.

I wish I knew about transcoding. Not sure what to think there. So I was trying to fully understand what the end goal was and all I could think of with my pea brain was for the app to be a stand-alone app (not a web app) allow for downloads and have the transcoding and codecs happen client side. Sounds like you want to do a web app though. Having the download feature would be great for those who go in and out of poor internet connections though.


#9

No, there is always a charge. An application could work around that by using a GET before sending each chunk but this has to be coded and this will slow down uploads. I don’t think that current applications on safe network do this.


#10
 the app to be a stand-alone app (not a web app) allow for downloads and have the transcoding and codecs happen client side

This would be for a decentralised app, so basically YouTube without the server farms.


#11

Thanks for the clarification. Future apps will surely send GETs first when it comes to expensive files like large videos. As long as a user can take co-ownership of a file without paying (​:


#12

Wow surprised I went so long thinking otherwise. Kind of a huge bummer but I understand charging for network operations mitigates spamming.


#13

You have to do the GET on each chunk as someone could have only uploaded the first part of a file/video.

The cost equation is more like expense of waiting vs the expense of upload.

For some the delay (maybe upto 3 times the upload time) wll exceed their desire to save that number of “PUT” charge.

Yes, if you request a “PUT” to be done then you pay for it.


#14
You have to do the GET on each chunk as someone could have only uploaded the first part of a file/video.

But if a file is split before uploading, the first chunk will be completely different anyway. So shouldn’t a GET on the first chunk be enough to take co-ownership?


#15

No, self encryption says that if the first 3 chunks are the same then the encrypted version of the first chunk will be the same.

It is possible to upload only the first 3 chunks of any file and if you only check the first chunk then you will think the file is intact and fully uploaded, but infact it is not and you end up with the same broken file as the person who only uploaded the first 3 chunks.

And if you think it will be rare then think again. For instance if all the apps just check the first chunk then some idiots will take all the new files/vids they can find and upload the first 3 chunks just for giggles. Then all those apps uploading the latest videos/files will think those files are there and the uploaders will be sorely disappointed.


#16

I thought the first chunk was encrypted with the hash of the last chunk or something like that. Is that not the case?


#17

Nope, but even if it was then consider this

A new file becomes available on the old internet. Mr Giggle Bones downloads it and only uploads the first 3 chunks stopping the upload there. The file is the same as you decide to upload and the self encryption has the same input as the file Mr Giggle used.

So then the 1st 3 chunks are exactly the same since they were created from the same input file.

Other reasons for incomplete uploads is power failures, internet outages (for the user) etc etc.

So to ensure you upload the complete file you must GET each chunk to ensure it was uploaded. And there lies the tradeoff of speed vs cost


#18

That sounds like a serious problem with the network. I’ll look it up sometime soon.


#19

Its only a problem if you want to only do 1 GET to see if the whole file is uploaded.

Otherwise the intention is to use the intrinsic de-dup of immutable data to handle duplicate chunks. i.e. you just upload the chunks you want uploaded.

To be as flexible as possible then you need to allow people to upload single chunks. For instance I have a file (long document with images - 200MB in size). I then add to the file (edit) and only the last 10 MB are changed and 20 MB added. The save of the file only requires that the changed chunks be uploaded, so the network has to allow parts of files to be uploaded.


#20

self encryption

And for the case of chunk 1 see the video on this page https://maidsafe.net/features.html#self-encryption

From these we see that the first chunk is only reliant on chunks 2 & 3

To do otherwise would mean the whole file has to be split, stored and process together. The self encryption allows the file to be self encrypted as it is read off disk. This allows for low power (low RAM/Disk) devices to self encrypt without excessive overheads.


#21

So the changed chunk would necessarily have to be the last to be encrypted, otherwise its changed hash would affect the hashes of all the subsequent chunks in the self-encryption process. Is that documented somewhere?