Self_encryption and compression, questions and thoughts

That’s sort of what the entropy test (that David mentioned above) would do I think - basically looking for anything with non-randomness.

Nodes could disable that sort of check though for efficiency and risk holding something illegal (maybe not a large risk anyway) … but if that happens then these nodes get an advantage over others … probably no perfect solution here in any case.

Edit: I stand corrected - see below.

1 Like

Ah, this check would be Elder based so the Adult nodes wont even get a chance to see that data.

4 Likes

The quote below left me thinking that jpg’s could still be an issue, but I’m not sure how it should be read. So I was thinking that maybe we could filter out some specific file types. But yeah, I don’t know.

1 Like

jpg and similar can have entropy close to apparently encrypted. It’s down to better compression.

I have had a few interesting convos with crypto experts and it’s not obvious but here is a thing I note and it seems surprising to a lot of people. One of those good stories to tell as it spreads info that should be obvious, but is not, so it is a good thing to spread.

A good encryption algorithm will output non recurring (or random) data. Highly non recurring. The same is true for a good compression algorithm (this one is more obvious).

3 Likes

That’s what I thought was implied in there. But I guess jpg also contains some kind of marker that it is jpg and can thus be rejected? Of course this is not capture all and future proof solution, but could some kind of simple block list work?

1 Like

Not one we can trust :wink: , but also it’s not in chunks either if the person chunked a jpeg (as he has to as we don’t store more than 1mb per chunk).

1 Like

Ok, but isn’t the problem really only a problem with small files, like under 1MB jpg? Would “one chunk jpg” contain a “jpg marker”?

And If 10MB jpg is chunked without SE, then individual chunk does not form a viewable image, or does it?

yes, kinda. You can read a part of some image files and see the image. So no markers. In any any case we cannot trust file markers (when considering malice).

It can and that’s a potential problem. It is a much-reduced problem though, but still a possible issue. As I say though much reduced, but it would be nice to close that door altogether if possible. I think with entropy checks and small chunks we are in a good simple to achieve place.

You can make a valid image in about 150bytes but likely not good enough to be bad, but I bet there will be ways to do this kind of attack? We just need to shut them down. It will be several steps to do that I think.

2 Likes

Bitcoin had problems with blockchain data triggering antivirus, so they built in a layer of local obfuscation:

6 Likes