[ZendTo] Re: Unable to Upload large files

Fri Mar 9 17:02:44 GMT 2012

On 09/03/2012 16:06, Brian Ott wrote:
> On Fri, Mar 09, 2012 at 09:21:02AM +0000, Jules wrote:
>> Brian,
>>
> Hey Jules, Thanks for the reply I did some more testing with the same
> results.
>
>> On 08/03/2012 20:16, Brian Ott wrote:
>>> Hey Jules!
>>>
>>> We have done some testing on zendto and came across some issues, we
>>> would appreciate any help you can provide. It seems we are not able to
>>> upload large files around 4G. As explained below our test setup I
>>> should also mention we got the same results by accessing the host
>>> directly without any load balancers and only using one host.
>> Yes, the load balancers shouldn't affect this, as they are just seeing a
>> packet stream.
>>
>> Read on...
>>> Cheers!
>>>
>>>
>>>
>>> Here is our setup:
>>>
>>> (Names and IPs replaced from real ones)
>>>
>>> We have our zendto setup on two servers (identical) running centos5.
>> 64-bit I hope! :-)
>>>    I
>>> tried running Debian and Cento6 but as mentioned in earlier posts I
>>> experienced issues while trying to modify php to large file uploads.
>>>
>>> I follow the instructions on the website, nothing was done outside of
>>> that.
>>>
>>> Our two servers (zendto-www1&   zendto-ww2) have zendto placed on NFS,
>>> and share the mount.
>> NFS often doesn't handle files over 4GB very well *at all*. In fact most
>> NFS installations I have seen are totally incapable of handling 4GB
>> files. So that is certainly a starting point, I would make up some large
>> files by hand with something like
>>       yes | dd of=BIGFILE bs=1024000 count=5000
>> which should give you 5GB of the letter "y" (better test than just
>> zeros) and copy that file to, from and around the NFS mount to check
>> that it can actually handle 5GB files intact. Use "cksum" to quickly
>> checksum them against the original.
> Alright I Moved zendto on one server off NFS, everything is now local
> I just kept the DATA dir on nfs so the complete downloads would be put
> there. However 'incomplete' go to /var/zendto/incoming.
>
>>>    We use MySQL as the database and its on another
>>> host.
>> That's fine.
>>> here are some of our php.ini settings:
>>>
>>> post_max_size = 50000M
>>> file_uploads = On
>>> upload_tmp_dir = /nfs/zendto/zendto_files/incoming
>>> upload_max_filesize = 50G
>>> default_socket_timeout = 6000
>>> max_execution_time = 12000     ; Maximum execution time of each
>>> script, in seconds
>>> max_input_time = 12000  ; Maximum amount of time each script may spend
>>> parsing request data
>>> memory_limit = 1000M      ; Maximum amount of memory a script may
>>> consume (16MB)
>> That looks okay. Also make sure your apc.ini has the relevant settings
>> for apc.max_file_size=50G, apc.slam_defense=off and such like as
>> documented on zend.to.
> Verified they are as such.
>
>>> The server itself has 4G of ram (each of them).
>>>
>>> Problem
>>> ----------
>>>
>>> Before we wanted to roll this out here into production our final step
>>> was to do some testing, here is what we found.
>>>
>>> - Uploading files under 2G works fine under all Browsers
>> Correct.
>>> - Upload/Download speed is a little slow, transferring from the box
>>>     using SCP is almost double the speed.
>> That's down to the browsers mostly. But it will also be caused by your
>> NFS as that doubles the amount of network traffic having to happen for
>> the upload.
>>> - Uploading 2.5G-3G files fails on IE, Safari&   Firefox.
>> Yes, it will, certainly on 32-bit Windows.
>>> - Chrome uploads 3G+ files fine in Linux but not Windows
>> Yes, certainly on 32-bit Windows. You don't say whether you're using
>> 32-bit or 64-bit Windows. IE9 64-bit on Windows 7 64-bit does work, as
>> does Chrome. But IE9 is the *only* 64-bit browser, even on 64-bit
>> Windows. If you look in Task Manager on Windows 64-bit you will see that
>> Chrome is only a 32-bit process.
>>
>> And 2^32 (max representable in an unsigned int) is 4GB. Most coders use
>> ints (which are signed) through laziness, so giving a max of 2^31 = 2GB.
>> That's where the limits come from.
>>
>>> What seems to happen is the following:
>>>
>>> 90% of the time for IE/Firefox users the connection gets reset after a
>>> few seconds (maybe 30 seconds?) then takes them from verify.php to
>>> dropoff.php. The user gets prompted with the popup saying "This page
>>> is asking you to confirm that you want to leave etc" With two
>>> selections of either "Leave Page" or "Stay on Page".
>>>
>>> Leave page obviously takes them to dropoff.php with a "Connection
>>> Reset" page and the file upload fails. If they choose "Stay on Page"
>>> they stay on "verify.php" but nothing happens and the file never
>>> uploads.
>>>
>>> The other issue is for Chrome users on Windows, and sometimes on Linux
>>> where at any percentage of a large file upload the page will have a
>>> popup saying the page is not responding and if you want to "Kill it"
>>> or "wait" if you select wait sometimes it will resume the upload till
>>> complete. However we have had some tests where it did not on Windows
>>> and just ended up having to be killed.
>>>
>>> I can enable any logging required or provide any information in
>>> private. We would really like your help with this as we do like this
>>> product and want to roll it out into Production.
>> You need 64-bit Windows with 64-bit-IE9 or Chrome to reliably do>4GB
>> uploads.
>> This is entirely caused by lots of bugs in lots of browsers. As it's not
>> a big marketing feature, the vendors put no effort into this area of
>> their code, so it gets better *very* slowly.
>>
>> And unfortunately there's nothing I can do about that. You have to be
>> using 64-bit Windows if you want to use Windows for any>4GB uploads to
>> any website.
>>
>> The Mac browsers fare far better, as OS X has been 64-bit-only for so
>> much longer. I use a Mac and everything, as they say, "just works".
>> 32-bit apps rarely exist on OS X any more.
>>
>> Linux is somewhere in between right now.
>>
>> I hope that helps, even though I can't necessarily give you a "quick
>> fix" setting change to apply.
>>
> All out tests were done in 64bit OSES and Browsers.
>
> Test Enviroment:
>
> 5 Windows 7 64 Bit OSES using Firefox/Chrome/IE9
Using the IE9 64-bit or the normal (default) one which is 32-bit?
> 4 Linux Systems 64 using Chrome/Firefox
> 2 OSX Systems 64 using Safari
>
> Also our CentOS servers are 64bit
> Linux dropoff-www1 2.6.18-274.12.1.el5 #1 SMP Tue Nov 29 13:37:46 EST
> 2011 x86_64 x86_64 x86_64 GNU/Linux
>
> I did notice that when I do start an upload in firefox then check the
> 'incoming' directories they are empty. If I then check the log I see I
> logged in but nothing else. I then check the 'dropoffs' directory and
> it contains no upload.
>
> I ran TCPDUMP and when I do try to do an upload I am seeing lots of
> traffic until I get that prompt saying:
>
> "This page is asking you to confirm that you want to leave - data you
> have entered may not be saved" "Stay on page" or "Leave Page"
>
> Then all the traffic stops. I can get this error within a few seconds
> of starting the upload (maybe 20-30 seconds?)
This puzzles me too. I might be apc-related problems, over which I have 
no control. Though there definitely are some bugs in it. Unfortunately 
it's the only technology (other than resorting to Flash) that can 
achieve this, so I've got no choice. It seems to work with most PHP-apc 
builds with a variety of 64-bit browsers, but I have a feeling it does 
vary between builds. Also the browsers, partic Chrome of course, change 
pretty fast.

I've never had the time to analyse 2GB of TCP packets to try to produce 
a reason as to why a browser might suddenly think you're leaving the 
page. It would take me ages to track down unfortunately, and there might 
not be an answer at the end of the rainbow anyway.

Jules.

>
>> Cheers,
>>
>> Jules
>>
>> -- 
>> Julian Field MEng CITP CEng
>> www.Zend.To
>>
>> Follow me at twitter.com/JulesFM
>> PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654
>>
>> 'It's okay to live without all the answers' - Charlie Eppes, 2011
>> 'All programs have a desire to be useful' - Tron, 1982
>> 'That is the land of lost content,
>>    I see it shining plain,
>>    The happy highways where I went,
>>    And cannot come again.' - A.E. Houseman
>>
>> _______________________________________________
>> ZendTo mailing list
>> ZendTo at zend.to
>> http://mailman.ecs.soton.ac.uk/mailman/listinfo/zendto

Jules

-- 
Julian Field MEng CITP CEng
www.Zend.To

Follow me at twitter.com/JulesFM
PGP footprint: EE81 D763 3DB0 0BFD E1DC 7222 11F6 5947 1415 B654

'It's okay to live without all the answers' - Charlie Eppes, 2011
'All programs have a desire to be useful' - Tron, 1982
'That is the land of lost content,
  I see it shining plain,
  The happy highways where I went,
  And cannot come again.' - A.E. Houseman