FAQ:Uploads

From Relay

Jump to: navigation, search

Contents

[edit] Questions

  • The only thing perl brings to the party is the upload progress status meter thingy? Make it optional. I don't want to install perl just for that, and I can live without it.
    • It can be optional. Put the code from the uploadFiles() function (currently commented out) into its own file (may take some work with setting paths, etc. Then change the FC.UPLOADURL in relay.js to point to the PHP file's location. You'll need to comment out the uploadStatus stuff in relay.js too. All that to say, it is possible. ;) -- TheIdeaMan
      • Taking your idea, man, I got it working. It's quick&dirty, but at least I can use it without perl. In relay.php uncomment the uploadFiles function and change the present databaseAdd call to: 
 databaseAdd($userpath,$uploadfile,$path);

in relay.js change the whole content of sendUpload() to the following 

 $('uploadForm').relay = 'fileUpload';
 $('uploadForm').path =  this.path;
 $('uploadForm').action = FC.URL;
 $('uploadForm').submit();
 Element.toggle('uploadAdd');
 uploadFinish();

Update after upload is not working - I don't mind, but waiting for the switch in the official code ;) -- Tom

  • Show two upload bars - one for the current file beimg uploaded, and the other for the overall progress.
  • hawks5999: Enable directory uploads
  • hawks5999: Enable drag and drop of uploads (like the drag to cart or drag to directory within the page)
  • keith: I see minimum requirement for Apache is 2.0. Can it work in Apache 1.3.33 work?
  • sturgis: with a java applet it would be possible to upload with drag and drop "any" amount and size. I would strongly recommend to take a look to the http://gallery.menalto.com/ upload applet and to the http://www.afian.com upload manager, a great product (shareware). A suggestion would be to integrate http://www.postlet.com/ (opensorce) as a way to upload files. What do you think?
  • Has there been any progress on getting Relay to upload large files (10MB+)? I'm not exactly sure what is preventing the script from moving the temp file to its respective virtual directory (the dreaded 99% loop). Unless it's my webhost limiting the size of uploads. Let me check... My webhost limits PHP uploads to 7MB. With Relay, I was able to upload a 16MB file, but nothing bigger than that... Dang, a 65MB upload worked flawlessly on the ecosmear demo.
    • Has a fix been found for this/ or the problem identified? I watch the file upload perfectly (i.e. I can watch the '%'stats, the estimated time, and the speed) but as soon as it hits 99% the whole upload freezes. Filesize of 13.4MB works fine, but attempting 33MB fails.
  • The 99% problem appears to be server related--as you'll see, the DEMO works with large files. I suspect it's a memory limitation imposed by hosting servers. I've been trying to get my hosting company to give me some answers--nothing yet.
    • I have just noticed that the files which get stuck at 99% are showing up as temp files in /relaydir/uploads. when renamed they will work (if viewed from outside relay) but when downloaded from within relay (as .zip) the files report as being corrupted. as far as errors in relay go, the only one i can find is when running 'perl ./mysite.com/relaydir/upload.pl' which instead of reporting back nothing, gives me the following error 
 "Can't locate conf.uploader in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.4 /usr/local/share/perl/5.8.4 /usr/lib/perl5
/usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at ./mysite.com/relaydir/upload.pl line 8."

i have no idea if this is related to the 99% problems myself and others are finding?


[edit] Rewritten Perl Script

A while ago I found this rewritten perl script on this wiki that solved my problems concerning the 99% upload bug. But because of the immense SPAM that time I think it go lost. I just can't find the script anymore, so here is a repost. The script deals with the problems by chopping up the file in chunks while uploading so that the server memory never exceeds its limits. I believe the script was written by lucy [at] triffid.org so credits to him/here. You can find the script just below here, just try it: 

 #!/usr/bin/perl 
#use strict;
#use POSIX 'SEEK_SET';
#use vars qw/ $uploadsFolder /;

# These were never used?! 
use Fcntl qw(:DEFAULT :flock);
use File::Temp qw/ tempfile tempdir /;
use CGI;

my $path = $ENV{'PATH_TRANSLATED'} || '';
if($path eq ''){$path = $ENV{'SCRIPT_FILENAME'};}
$path =~s/upload.pl/conf.uploader/;
if($path eq ''){$path = "conf.uploader";}

my $DEBUG = 0;

print "Content-Type: text/plain\n\n";
warn "we required $path" if $DEBUG;
require $path;

my $content_type = $ENV{'CONTENT_TYPE'};
my $len = $ENV{'CONTENT_LENGTH'};
my $bRead = 0;
my $sessionid;

my $starttime = time;
my @qstring=split(/&/,$ENV{'QUERY_STRING'});
$sessionid = $qstring[0];

if($sessionid eq 'test'){
	print "ok";
	exit;
}

my $tmpfile   = "$uploadsFolder\\temp_$sessionid";
my $statsfile = "$uploadsFolder\\stats_$sessionid.txt";
my $tmpfilepre= "$uploadsFolder\\$sessionid\_";

$tmpfile =~s/\\/\//g;
$statsfile =~s/\\/\//g;
$tmpfilepre =~s/\\/\//g;

open(TMP,"+>","$tmpfile") or die "can't open temp file $tmpfile: $!";
open(STATS,">","$statsfile") or die "can't open stats file $statsfile: $!";

my $rLen = 4096;
my $LINE;

# Values for dl progression
my ($now,$elapsed,$percent,$estimate,$estTime,$percentEst);

if($len - $bRead < $rLen){
	$rLen = $len-$bRead;
}

while ($bRead < $len && read (STDIN ,$LINE, $rLen)) {
	$bRead += length $LINE;
	if($len - $bRead < $rLen){
		$rLen = $len-$bRead;
	}
	
	print TMP $LINE;
	
	$now = time;
	$elapsed = $now-$starttime;
	$percent = ($bRead/$len) * 100;

	if($elapsed >= 1){
		$estimate = $bRead / $elapsed; # in kb
	}else{
		$estimate = $bRead / .5;
	}

	$estTime = ($len-$bRead)/$estimate; #in sec
	$percentEst = $estimate/$len * 100; #in percent

	# % , % per second, kb estimate, est time remaining
	if($percent >= 99){
		$percent = 99;
	}

	seek(STATS,0,SEEK_SET);
	print STATS "$percent\n$percentEst\n$estimate\n$estTime\n$len\n\n";
}


if($bRead != $len){
	close(TMP);
	unlink $tmpfile or warn "Failed to unlink $tmpfile: $!";
	unlink $statsfile or warn "Failed to unlink $statsfile: $!";
	exit;
}

## lucy at triffid.org 01/08/2007 Main edit starts here.
my ($junk,$boundary);
($junk,$boundary) = split /=/, $ENV{CONTENT_TYPE}, 2;
$boundary =~ s/\n//;
$boundary = "--$boundary";


seek(TMP,0,SEEK_SET);

my $i = 0;
my $expect_junk = 0;
my @fileCount;
my ($filename,$line,$headers,$unwanted);


LINE: while (<TMP>) {
	$line = $_;
	# If it's a boundary
	if (($line eq "$boundary\r\n") && ($fileCount[$i] eq '')){
	close FILE;
	$i++;
	# register it's a new file
	print "got new filename" if $DEBUG;
	$fileCount[$i] = 'new';
	next LINE;
	# find the new filename
	} elsif ($fileCount[$i] eq 'new'){
		print "<br> suspect new filename on file ".$i if $DEBUG;
		($headers,$unwanted) = split(/\n\r\n/,$line,2);
		($headers,$unwanted) = split(/\n\r\n/,$line,2);
		$filename = $headers;
		$filename =~s/.*?\n?.+?filename=\"(.+(\\|\/))?(.+?)\".*\n.*/file-$3/g;
		if(substr($filename,0,5) eq "file-"){
			$filename =~s/file-//;
			print "<br>filename was $filename" if $DEBUG;
			open(FILE,"+>","$tmpfilepre$filename") or  warn "Can't open temp file: $tmpfilepre$filename: $!";
			binmode(FILE);
			$fileCount[$i] = '';
		} else { 
			# .. it's junk data and not the filename so ignore
			if ($DEBUG) {warn "something wrong\n"; print "<br>something is wrong"; }
		}
	# Are we getting the tail end of the header? ignore this too.
	} elsif ($expect_junk == 1) {
		$expect_junk = 0;
		next LINE;
	} else {
		if ($line =~ m/Content-Type/) {
		$expect_junk = 1;
		next LINE;
		}
	# Real file data here, write it.
	print FILE $line or die "Failed to write file: $!";
	}
}

close FILE;
close TMP;
seek(STATS,0,SEEK_SET);
print STATS "100\n$percentEst\n$estimate\n$estTime\n$len\n";
close STATS;

The upload.conf file remains the same and needs to be placed in the same directory as the upload.pl file. 

 $uploadsFolder = "Here you're complete server upload url";
    true;

For security reasons I also placed my upload.pl and upload.conf in the cgi-bin folder of my server. To get it to work you need to change the 'relay.js' file in the 'js' folder of relay. On line 12 you have to change 

 UPLOADURL: 'upload.pl',

into 

 UPLOADURL: 'cgi-bin/upload.pl',

This works when you your perl folder is cgi-bin and is placed in the same directory as relay, please change the last if your folder is located somewhere else or has another name.

[edit] Response

Lots of people wonder why we have to require PERL for the upload progress bar. The reason behind that is that PHP does not provide HTTP Header sizes to the rest of the script while files are being uploaded (streamed) to the process. PERL however, does have the ability to get the total header size of the HTTP request while its uploading. To get the progress bar, the total has to be a known variable. We are working on an upload.php file so that PERL is not required but those that choose that route will loose upload progress (sorry).

[edit] Dual Upload Bars

Even with PERL, dual upload bars was not in our plans. Getting the total size of the HTTP header from PERL and then getting it to PHP and pushing it through ajax was a small miracle in its own. I can see that its most likely possible, but a lot more work probably. I'd like to see someone do it, and you'd get some serious respect by us.

[edit] Directory Uploads

There is a limitation imposed by all browsers in that only one file can be added to a file input at a time. I suppose that Java could come into play here... Ideas?

  • D4Skunk @ hotmail.com : add ability to upload zips that get extracted ?
  • Chris Iufer: we've wanted to do that since we got server side zipping included. Im sure that feature will popup very soon.

[edit] Drag and Drop Uploads

Again, were at the mercy of the browsers. They just dont have the ability to do something like that (without the use of a Java applet).

Retrieved from "http://www.ecosmear.com/relay/wiki/index.php/FAQ:Uploads"
Personal tools
support