[ZendTo] ANNOUNCE: Version 5.10-1 "production" released

Sat Jun 30 13:44:32 BST 2018

John,

With the arrival of the European Union's General Data Protection 
Regulation (GDPR), people have the right to ask any organisation for a 
copy of all the data held about them.

So everyone is suddenly realising they need an easy way to send all this 
"personal information" to any who asks for it. Because it is personal 
information, it needs to be encrypted so that only the correct person 
has access to it, and there is no other way of accessing the information.

The first request for it came from our own University's Legal Services 
team, for precisely the reason above. However, they also send quite a 
lot of stuff encrypted over ZendTo anyway, but they had always done the 
encryption by hand using 7-Zip until now. That was okay when they were 
sending files to other lawyers and so on who knew what 7-Zip was, had IT 
support to install it for them, and so on. Now they are having to send 
encrypted files to random members of the public who have no idea about 
7-Zip, how to install it or anything.

So they needed a much easier solution that anyone could work without 
assistance.

I have since had other requests from EU companies and organisations 
needing exactly the same thing, largely for the same reasons.

This is how it works:

In the "new drop-off" form, at some point the user chooses to tick 
"Encrypt every file" (it can be mandated in preferences.php).
They are prompted to enter a passphrase (minimum length set in 
preferences.php).
Provided the drop-off isn't really big (set in preferences.php) it will 
let you do that.
User creates the new drop-off as normal, in every other respect.

Their new passphrase is sent to the ZendTo server over https, along with 
the files being uploaded.
All encryption and decryption is done on the ZendTo server itself, none 
of it happens in the browser.

At the end of the upload, just after the virus-scanning and optional 
checksumming of the files...
ZendTo creates a secure key derived from the user's passphrase (lots of 
hashing, basically).
ZendTo creates some extra (very) random bytes of data as the 
"initialisation vector" (IV) for the encryption algorithm.
ZendTo then encrypts the file, 64kbytes at a time so it does not need as 
much RAM as the size of the file, using the key and the IV.
ZendTo replaces the unencrypted file with the encrypted one.
ZendTo stores the initialisation vector in its database.
ZendTo carefully wipes the memory that is storing the passphrase, the 
key, the IV and any data derived from them.

At this point, the only copy of the user's files are encrypted. 
Absolutely no-one can decrypt the files without knowing the passphrase 
used to generate the key, *and* the IV. I assume that the bad guys have 
read all the source code of ZendTo, there is nothing hidden. Not even 
the sysadmins can decrypt the files.

When a recipient comes to download the files, they are presented with 
the same download page as normal.
When they first click on a link (or the "Download All Files" button), 
they are now prompted to enter the passphrase if the drop-off was encrypted.

That passphrase is passed to ZendTo over https (in "POST" data not "GET" 
so it doesn't appear in browser URL history).
ZendTo re-generates the key from the passphrase, and retrieves the IV 
from its database.
It then attempts to decrypt the file, and download it to the user as it 
decrypts it (so you don't need RAM or disk space for a temporary 
decrypted copy of the files).
If that works (i.e. the recipient entered the correct passphrase) the 
file download (and subsequent file downloads provided they don't leave 
that web page) process appears to work just as it did before.
If the decryption failed (incorrect passphrase, for example) the user 
gets shown the download page again, but showing an error message saying 
the passphrase was probably wrong.

The net impact on the user experience is this:

1. Those not wanting to use it just don't tick the new extra checkbox in 
the "new drop-off" form.
Other than that there is *no* change to their user experience at all, 
nor to the recipients of their drop-off.

2. Those wanting to use it just tick 1 checkbox and enter a passphrase. 
Otherwise it behaves entirely as normal.
When a recipient tries to download a file, if they haven't already 
entered the passphrase they are prompted to do so. Subsequent file 
downloads just work exactly as before, as the JavaScript on the download 
page remembers the passphrase they entered *until* they leave the page 
or a decryption fails. So they are only prompted once, provided they 
entered the passphrase correctly first time.

The point of using the extra (but not secret) IV data is to provide 
"authenticated encryption".
Without authenticated encryption (often called "AEAD"), someone could 
substitute the original (encrypted) file with a different file that was 
encrypted with the same key.
Yes, that would require hacking into the ZendTo server well enough to be 
able to find the exact right file in the right drop-off, and be able to 
replace it with a new file. And of course the hacker would have to know 
the original passphrase used, and how to derive the key from it and so on.
The recipient wouldn't notice anything was wrong, but would not be 
downloading the file they were supposed to get.

With the addition of the IV data, the hacker also has to be able to read 
the IV data itself, before attempting to produce a new file. As that 
data is stored in ZendTo's database, it could well be on a totally 
different machine (the MySQL server), which makes their job a *lot* harder.

If they don't use the correct IV data (which is generated randomly — 
very carefully — for each encrypted drop-off), then even if they manage 
to replace the file stored on the ZendTo server, the decryption process 
will still immediately fail when the recipient tries to download it.

The encryption and decryption is done with "libsodium" which is a fork 
of Dan Bernstein's "NaCl" (pronounced "salt") library, which is a 
relatively new (i.e. not totally ancient) cryptography library which is 
designed to be very secure and significantly faster than other such 
libraries as it is carefully optimised for various different CPU 
architectures.
https://libsodium.org/
http://nacl.cr.yp.to

Libsodium is a standard part of PHP from PHP 7.2 onwards, and is 
available as a "pecl" extension in PHP 7.0 and 7.1.
So the ZendTo Installer will now upgrade you to the best version it can 
find automatically, regardless of what PHP version you may have 
installed at the moment.

Dan Bernstein is just the right person to produce work like this. I have 
worked on his designs and code before (notably "qmail" when writing 
MailScanner), and he is the most fussy OCD designer and developer you 
will ever find. I mean that as a compliment to him, because I know quite 
how careful he will have been in the design and choice of the 
algorithms, and the implementation.

I hope that answers some of your questions. To find the crypto code 
easily, search for "sodium" in /opt/zendto/lib/*.php.

Any other questions, please do ask!

Cheers,
Jules.

On 29/06/2018 5:33 pm, John Thurston via ZendTo wrote:
>
> On 6/29/2018 8:03 AM, Jules Field via ZendTo wrote:
>> I have just done the "production" release of ZendTo version 5.10.
>
> Thank you, again, for your time an effort.
>
>>
>>  1. Optional secure encryption and decryption of drop-offs, using a
>>     passphrase set by the user.
>
> Do you have the time to describe this to us?
>
> Why was this added?
> What business problem is it solving?
> When in the flow are the bytes en/decrypted?
>
> I'm going to go look at the code and see if I can answer some of those 
> questions, but would like to know the author's view.
>
> -- 
>    Do things because you should, not just because you can.
>
> John Thurston    907-465-8591
> John.Thurston at alaska.gov
> Department of Administration
> State of Alaska
>
> _______________________________________________
> ZendTo mailing list
> ZendTo at zend.to
> http://jul.es/mailman/listinfo/zendto

Jules

-- 
Julian Field MEng CEng CITP MBCS MIEEE MACM

'I've heard that it's possible to grow up. I've just never met
  anyone who's actually done it.' - Meredith Grey, Grey's Anatomy

www.Zend.To
Twitter: @JulesFM
PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654