[ZendTo] Re: Unable to Upload large files

Fri Mar 9 16:06:28 GMT 2012

On Fri, Mar 09, 2012 at 09:21:02AM +0000, Jules wrote:
> Brian,
> 

Hey Jules, Thanks for the reply I did some more testing with the same
results. 

> On 08/03/2012 20:16, Brian Ott wrote:
> > Hey Jules!
> >
> > We have done some testing on zendto and came across some issues, we
> > would appreciate any help you can provide. It seems we are not able to
> > upload large files around 4G. As explained below our test setup I
> > should also mention we got the same results by accessing the host
> > directly without any load balancers and only using one host.
> Yes, the load balancers shouldn't affect this, as they are just seeing a 
> packet stream.
> 
> Read on...
> >
> > Cheers!
> >
> >
> >
> > Here is our setup:
> >
> > (Names and IPs replaced from real ones)
> >
> > We have our zendto setup on two servers (identical) running centos5.
> 64-bit I hope! :-)
> >   I
> > tried running Debian and Cento6 but as mentioned in earlier posts I
> > experienced issues while trying to modify php to large file uploads.
> >
> > I follow the instructions on the website, nothing was done outside of
> > that.
> >
> > Our two servers (zendto-www1&  zendto-ww2) have zendto placed on NFS,
> > and share the mount.
> NFS often doesn't handle files over 4GB very well *at all*. In fact most 
> NFS installations I have seen are totally incapable of handling 4GB 
> files. So that is certainly a starting point, I would make up some large 
> files by hand with something like
>      yes | dd of=BIGFILE bs=1024000 count=5000
> which should give you 5GB of the letter "y" (better test than just 
> zeros) and copy that file to, from and around the NFS mount to check 
> that it can actually handle 5GB files intact. Use "cksum" to quickly 
> checksum them against the original.

Alright I Moved zendto on one server off NFS, everything is now local
I just kept the DATA dir on nfs so the complete downloads would be put
there. However 'incomplete' go to /var/zendto/incoming. 

> >   We use MySQL as the database and its on another
> > host.
> That's fine.
> >
> > here are some of our php.ini settings:
> >
> > post_max_size = 50000M
> > file_uploads = On
> > upload_tmp_dir = /nfs/zendto/zendto_files/incoming
> > upload_max_filesize = 50G
> > default_socket_timeout = 6000
> > max_execution_time = 12000     ; Maximum execution time of each
> > script, in seconds
> > max_input_time = 12000  ; Maximum amount of time each script may spend
> > parsing request data
> > memory_limit = 1000M      ; Maximum amount of memory a script may
> > consume (16MB)
> That looks okay. Also make sure your apc.ini has the relevant settings 
> for apc.max_file_size=50G, apc.slam_defense=off and such like as 
> documented on zend.to.
> >
Verified they are as such. 

> >
> > The server itself has 4G of ram (each of them).
> >
> > Problem
> > ----------
> >
> > Before we wanted to roll this out here into production our final step
> > was to do some testing, here is what we found.
> >
> > - Uploading files under 2G works fine under all Browsers
> Correct.
> > - Upload/Download speed is a little slow, transferring from the box
> >    using SCP is almost double the speed.
> That's down to the browsers mostly. But it will also be caused by your 
> NFS as that doubles the amount of network traffic having to happen for 
> the upload.
> > - Uploading 2.5G-3G files fails on IE, Safari&  Firefox.
> Yes, it will, certainly on 32-bit Windows.
> > - Chrome uploads 3G+ files fine in Linux but not Windows
> Yes, certainly on 32-bit Windows. You don't say whether you're using 
> 32-bit or 64-bit Windows. IE9 64-bit on Windows 7 64-bit does work, as 
> does Chrome. But IE9 is the *only* 64-bit browser, even on 64-bit 
> Windows. If you look in Task Manager on Windows 64-bit you will see that 
> Chrome is only a 32-bit process.
> 
> And 2^32 (max representable in an unsigned int) is 4GB. Most coders use 
> ints (which are signed) through laziness, so giving a max of 2^31 = 2GB. 
> That's where the limits come from.
> 
> >
> > What seems to happen is the following:
> >
> > 90% of the time for IE/Firefox users the connection gets reset after a
> > few seconds (maybe 30 seconds?) then takes them from verify.php to
> > dropoff.php. The user gets prompted with the popup saying "This page
> > is asking you to confirm that you want to leave etc" With two
> > selections of either "Leave Page" or "Stay on Page".
> >
> > Leave page obviously takes them to dropoff.php with a "Connection
> > Reset" page and the file upload fails. If they choose "Stay on Page"
> > they stay on "verify.php" but nothing happens and the file never
> > uploads.
> >
> > The other issue is for Chrome users on Windows, and sometimes on Linux
> > where at any percentage of a large file upload the page will have a
> > popup saying the page is not responding and if you want to "Kill it"
> > or "wait" if you select wait sometimes it will resume the upload till
> > complete. However we have had some tests where it did not on Windows
> > and just ended up having to be killed.
> >
> > I can enable any logging required or provide any information in
> > private. We would really like your help with this as we do like this
> > product and want to roll it out into Production.
> You need 64-bit Windows with 64-bit-IE9 or Chrome to reliably do >4GB 
> uploads.
> This is entirely caused by lots of bugs in lots of browsers. As it's not 
> a big marketing feature, the vendors put no effort into this area of 
> their code, so it gets better *very* slowly.
> 
> And unfortunately there's nothing I can do about that. You have to be 
> using 64-bit Windows if you want to use Windows for any >4GB uploads to 
> any website.
> 
> The Mac browsers fare far better, as OS X has been 64-bit-only for so 
> much longer. I use a Mac and everything, as they say, "just works". 
> 32-bit apps rarely exist on OS X any more.
> 
> Linux is somewhere in between right now.
> 
> I hope that helps, even though I can't necessarily give you a "quick 
> fix" setting change to apply.
> 

All out tests were done in 64bit OSES and Browsers. 

Test Enviroment: 

5 Windows 7 64 Bit OSES using Firefox/Chrome/IE9 
4 Linux Systems 64 using Chrome/Firefox
2 OSX Systems 64 using Safari

Also our CentOS servers are 64bit 
Linux dropoff-www1 2.6.18-274.12.1.el5 #1 SMP Tue Nov 29 13:37:46 EST
2011 x86_64 x86_64 x86_64 GNU/Linux

I did notice that when I do start an upload in firefox then check the
'incoming' directories they are empty. If I then check the log I see I
logged in but nothing else. I then check the 'dropoffs' directory and
it contains no upload. 

I ran TCPDUMP and when I do try to do an upload I am seeing lots of
traffic until I get that prompt saying: 

"This page is asking you to confirm that you want to leave - data you
have entered may not be saved" "Stay on page" or "Leave Page"

Then all the traffic stops. I can get this error within a few seconds
of starting the upload (maybe 20-30 seconds?) 

> Cheers,
> 
> Jules
> 
> -- 
> Julian Field MEng CITP CEng
> www.Zend.To
> 
> Follow me at twitter.com/JulesFM
> PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
> 
> 'It's okay to live without all the answers' - Charlie Eppes, 2011
> 'All programs have a desire to be useful' - Tron, 1982
> 'That is the land of lost content,
>   I see it shining plain,
>   The happy highways where I went,
>   And cannot come again.' - A.E. Houseman
> 
> _______________________________________________
> ZendTo mailing list
> ZendTo at zend.to
> http://mailman.ecs.soton.ac.uk/mailman/listinfo/zendto

-- 
Brian Ott
Unix System Administrator

Ontario Institute for Cancer Research
MaRS Centre, South Tower
101 College Street, Suite 800
Toronto, Ontario, Canada M5G 0A3

Telephone:	647-260-7977
Email:		brian.ott at oicr.on.ca
www.oicr.on.ca

This message and any attachments may contain confidential and/or privileged information for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this message in error, please contact the sender and delete all copies. Opinions, conclusions or other information contained in this message may not be that of the organization.