User:File Upload Bot (Kernigh)

This is a bot operated by User:Kernigh. It now has a bot flag, so to see it in Special:Recentchanges, you need to click "show bots".

This bot is now uploading the files for MapleStory. See Community Issues. Only the monsters are left now, but there are many of them.

Since the bot seems to be working okay, I will decrease the delay between uploads from 2 minutes to 20 seconds.

This bot runs a modified version of the Commons:File upload service/Script. Kernigh added two hacks to the script: one to upload to MediaWikis other than Commons, and to download images instead of uploading them. This way, it can copy images between wikis.

25 August 2006... This bot is filling up Category:MapleStory images and subcategories with images from Wikibooks (Wikibooks Import List/MapleStory). At 21:46 today, Echelon gave it a bot flag.

20 August 2006... I have produced a version of the perl scripts without my username and password for view and download by others.

1 May 2006... This bot copied the Category:NetHack images from Wikimedia Commons, as part of moving the NetHack guide from Wikibooks. The bot waited 2 minutes between uploaded, yet still owned most of Special:Newimages and Special:Recentchanges.

The scripts
WARNING Someone might have edited these scripts. If you want to run them, download them from http://kernigh.pbwiki.com/DownloadFile#MediaWikibot instead!

mwdown.pl

 * 1) !/usr/bin/perl
 * 2) mwup.pl - MediaWiki file DOWNLOAD script
 * 3)   by Kernigh - xkernigh AT netscape DOT net
 * 4)   version 2006-05-01 - this script is in the public domain
 * 5) Derived from:
 * 6) Upload script by Erik Möller - moeller AT scireview DOT de - public domain
 * 7) Developed for the Wikimedia Commons
 * 8) Note: Before usage, create an account on the source MediaWiki
 * 9) for the bot. On Wikimedia Commons, the convention is
 * 10) "File Upload Bot (Username)", for example, File Upload Bot (Kernigh).
 * 11) Set the username and password below:
 * 1) "File Upload Bot (Username)", for example, File Upload Bot (Kernigh).
 * 2) Set the username and password below:
 * 1) Set the username and password below:

$username = "USERNAME"; $password = "PASSWORD";

$pause = 1;
 * 1) Set the pause in seconds after each download

%wiki_php = ( 'commons',      'http://commons.wikimedia.org/w/index.php',  'wb',           'http://en.wikibooks.org/w/index.php',  'wikibooks',    'http://en.wikibooks.org/w/index.php',  'sw',           'http://strategywiki.net/w/index.php',  'strategywiki', 'http://strategywiki.net/w/index.php', );
 * 1) List the wiki PHP scripts where you use the above username and password


 * 1) Then run the script on the command line using
 * 2) $ perl mwdown.pl wiki dirname < filenames
 * 3) where wiki is one of the wikis from the wiki_php list above,
 * 4) and dirname/ is the name of a directory which will contain the files to
 * 5) be downloaded, and filenames is a list of files, one per line,
 * 6) to download.
 * 7) Don't edit below unless you know what you're doing.
 * 1) to download.
 * 2) Don't edit below unless you know what you're doing.
 * 1) Don't edit below unless you know what you're doing.

use LWP::Simple; use LWP::UserAgent; use HTTP::Request; use HTTP::Response; use HTTP::Cookies; use HTML::Parser; use Encode qw(encode); use warnings;
 * 1) We need these libraries. They should be part of a standard Perl
 * 2) distribution.

$ignore_login_error=0; $docstring="Please read mwup.pl for documentation.\n"; my $wiki=$ARGV[0] or die "Syntax: perl mwdown.pl wiki directory\n$docstring"; my $dir=$ARGV[1] or die "Syntax: perl mwdown.pl wiki directory\n$docstring";

$cgi = $wiki_php{$wiki} or die "Unknown wiki: $wiki\n$docstring";
 * 1) Find the wiki PHP script

if( $cgi =~ m|http://([^/]+)/| ) { $cgi_host = $1; } else { die "Unable to extract hostname from:\n$cgi\n"; }
 * 1) Find the hostname

$dir=~s|\\|/|gi;
 * 1) Make Unix style path

$sep=$/; $/="/"; chomp($dir); $/=$sep;
 * 1) Remove trailing slashes

my $browser=LWP::UserAgent->new; my @ns_headers = (  'User-Agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20041107 Firefox/1.0',   'Accept' => 'image/gif, image/x-xbitmap, image/jpeg,        image/pjpeg, image/png, */*',   'Accept-Charset' => 'iso-8859-1,*,utf-8',   'Accept-Language' => 'en-US',  );

$browser->cookie_jar( {} );

$response=$browser->post("$cgi?title=Special:Userlogin&action=submitlogin", @ns_headers, Content=>[wpName=>$username,wpPassword=>$password,wpRemember=>"1",wpLoginAttempt=>"Log in"]);

if($response->code!=302 && !$ignore_login_error) { print "We weren't able to login. This could have the following causes:
 * 1) After logging in, we should be redirected to another page.
 * 2) If we aren't, something is wrong.

Solution: Edit upload.pl and change them. Solution: Go to (where?) and get a new version of the script. The wiki has cookie check disabled. Solution: Try setting \$ignore_login_error to 1.
 * The username ($username) or password may be incorrect.
 * The MediaWiki software has been upgraded.
 * You are trying to hack this script for other wikis.

Regardless, we will now try to write the output from the server to $dir/debug.txt....\n\n";       open(DEBUG,">$dir/debug.txt") or die "Could not write file.\n";        print DEBUG $response->as_string;        print "This seems to have worked. Take a look at the file for further information.\n";       close(DEBUG);        exit 1; }

mkdir $dir; open(DESC, ">$dir/files.txt") or die "Could not write file.\n";


 * 1) HTML parser callbacks: the while() loop first attaches the
 * 2) div_handler to the image description page. The div_handler attaches
 * 3) an div_a_handler which downloads the image (the first link after

sub div_a_handler { # tagname,attr,self return if shift ne "a"; my $attrs = shift; return unless $$attrs{"href"};
 * 1) This handles the  by downloading the linked image and
 * 2) terminating the parser.

shift->eof; # terminate all parsing after this element

my $url = $$attrs{"href"}; print "Url: $url\n";

# if url starts with slash, prepend host $url = "http://$cgi_host". $url if( $url =~ m|^/| );

if( $url =~ m|.*/([^/]+)| ) { $filename = $1;

# download image print DESC ">$filename\n"; $browser->get( $url, ":content_file" => "$dir/$filename" ); } else { print "FAILURE: Unable to extract filename\n"; } }

sub div_handler { # tagname,attr,self return if shift ne "div"; my $attrs = shift; return unless $$attrs{"id"}; return if $$attrs{"id"} ne "file";
 * 1) This handles the containing the image by
 * 2) attaching div_a_handler.

# If we have not returned, then we found it! Attach image handler. shift->handler( start => \&div_a_handler, "tagname,attr,self" ); }

sub text_handler { #tagname,self return if shift ne "text";
 * 1) This handles the element of the Special:Export page containing
 * 2) the image description. (Yes, using HTML::Parser on XML.)

$self = shift; # the HTML::Parser

# Attach handler to print the image description. $self->handler( text => sub { print DESC (shift) . "\n" }, "dtext" ); $self->handler( end => sub { shift->eof }, "self" ); }

while(  ) { sleep $pause;
 * 1) This parses all image description pages.

$file = $_; chomp $file; # remove line terminator from each line

next if $file =~ /^\w*#/; # skip comments in input $file =~ s/[_ ]/+/g;     # change _, spaces to +

$response=$browser->get( "$cgi?title=Image:$file" ); push @responses, $response->headers_as_string. "\n";

if( $response->code != 200 ) { print "FAILURE: No img desc page: $file\n"; next; }

print "Downloading: $file\n";

# Parse the image description page: div_handler will # attach img_handler to download the iamge. my $parser = HTML::Parser->new( api_version => 3 ); $parser->handler( start => \&div_handler, "tagname,attr,self" ); $parser->parse( ${$response->content_ref} );

# Fetch the image description. Use Special:Export to get it # (because we might be blocked from the edit screen). $response=$browser->get( "$cgi?title=Special:Export/Image:$file" ); if( $response->code != 200 ) { print "FAILURE: No img desc export: $file\n"; print DESC "FAILURE: Unable to read ". "image description from $wiki.\n"; next; }	$parser = HTML::Parser->new( api_version => 3 ); $parser->handler( start => \&text_handler, "tagname,self" ); $parser->parse( ${$response->content_ref} ); }

print "Everything seems to be OK. Log will be written to $dir/debug.txt.\n"; open(DEBUG,">$dir/debug.txt") or die "Could not write file.\n"; print DEBUG @responses;

mwup.pl

 * 1) !/usr/bin/perl
 * 2) mwup.pl - MediaWiki file UPLOAD script
 * 3)   by Kernigh - xkernigh AT netscape DOT net
 * 4)   version 2006-05-01 - this script is in the public domain
 * 5) Derived from:
 * 6) Upload script by Erik Möller - moeller AT scireview DOT de - public domain
 * 7) Developed for the Wikimedia Commons
 * 8) Note: Before usage, create an account on the destination MediaWiki
 * 9) for the bot. On Wikimedia Commons, the convention is
 * 10) "File Upload Bot (Username)", for example, File Upload Bot (Kernigh).
 * 11) Set the username and password below:
 * 1) "File Upload Bot (Username)", for example, File Upload Bot (Kernigh).
 * 2) Set the username and password below:
 * 1) Set the username and password below:

$username = "USERNAME"; $password = "PASSWORD";

$pause = 120;
 * 1) Set the pause in seconds after each upload

%wiki_php = ( 'commons',      'http://commons.wikimedia.org/w/index.php',  'wb',           'http://en.wikibooks.org/w/index.php',  'wikibooks',    'http://en.wikibooks.org/w/index.php',  'sw',           'http://strategywiki.net/w/index.php',  'strategywiki', 'http://strategywiki.net/w/index.php', );
 * 1) List the wiki PHP scripts where you have the username/password pair


 * 1) Then run the script on the command line using
 * 2) $ perl mwup.pl wiki dirname
 * 3) where wiki is one of the wikis from the wiki_php list above,
 * 4) and dirname/ is the name of a directory containing the files to
 * 5) be uploaded, and a file named files.txt in the following format
 * 6) What you write                Explanation
 * 7) @     This text is appended to every description.
 * 8) °Dog photo by Eloquence       This text is used when no description exists.
 * 9) >Dog01.jpg                    Name of a file in the specified directory.
 * 10) German shepherd dog           Description (can be multi-line).
 * 11) >Dog02.jpg                    File without a description (use default)
 * 12) The "@" and "°" lines are optional, and must be in one line. They can
 * 13) occur multiple times in a single file and are only valid until they
 * 14) are changed. As a consequence, description lines cannot start with "@"
 * 15) or "°".
 * 16) Don't edit below unless you know what you're doing.
 * 1) The "@" and "°" lines are optional, and must be in one line. They can
 * 2) occur multiple times in a single file and are only valid until they
 * 3) are changed. As a consequence, description lines cannot start with "@"
 * 4) or "°".
 * 5) Don't edit below unless you know what you're doing.
 * 1) Don't edit below unless you know what you're doing.

use LWP::Simple; use LWP::UserAgent; use HTTP::Request; use HTTP::Response; use HTTP::Cookies; use Encode qw(encode); use warnings;
 * 1) We need these libraries. They should be part of a standard Perl
 * 2) distribution.

$ignore_login_error=0; $docstring="Please read mwup.pl for documentation.\n"; my $wiki=$ARGV[0] or die "Syntax: perl mwup.pl wiki directory\n$docstring"; my $dir=$ARGV[1] or die "Syntax: perl mwup.pl wiki directory\n$docstring";

$cgi = $wiki_php{$wiki} or die "Unknown wiki: $wiki\n$docstring";
 * 1) Find the wiki PHP script

$dir=~s|\\|/|gi;
 * 1) Make Unix style path

$sep=$/; $/="/"; chomp($dir); $/=$sep;
 * 1) Remove trailing slashes

open(FILELIST,"<$dir/files.txt") or die "Could not find file list at $dir/files.txt.\n$docstring";
 * 1) Now try to get the list of files

$standard_text[0]=""; $default_text[0]=""; $stx=0; $dtx=0; while() { $line=$_; chomp($line); if($line=~m/^@/) { $line=~s/^@//; $standard_text[$stx]=$line; $stx++; $stw=1; }       elsif($line=~m/^°/) { $line=~s/^°//; $default_text[$dtx]=$line; $dtx++; $dtw=1; }       elsif($line=~m/^>/) { $line=~s/^>//;

# New file, but last one doesn't have a description yet - # add current default. if($currentfile) { # If there's been a change of the default or standard # text, we need to apply the old text to the previous # file, not the new one. $dx= $dtw? $dtx-2 : $dtx -1; $sx= $stw? $stx-2 : $stx -1; if(!$desc_added) { $file{$currentfile}.="\n".$default_text[$dx]; }                       $file{$currentfile}.="\n\n".$standard_text[$sx]; }               # Abort the whole batch if this file doesn't exist. if(!-e "$dir/$line") { die "Could not find $dir/$line. Uploading no files.\n"

}               $currentfile=$line; $desc_added=0; $dtw=0;$stw=0; }else { # If this is a header comment, # we just ignore it. Otherwise # it's a file description. if($currentfile) { $file{$currentfile}.="\n".$line; $desc_added=1; }       } }

if($currentfile) { $dx= $dtw? $dtx-2 : $dtx -1; $sx= $stw? $stx-2 : $stx -1; if(!$desc_added) { $file{$currentfile}.="\n".$default_text[$dx]; }       $file{$currentfile}.="\n\n".$standard_text[$sx]; }
 * 1) Last file needs to be processed, too

my $browser=LWP::UserAgent->new; my @ns_headers = (  'User-Agent' => 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7) Gecko/20041107 Firefox/1.0',   'Accept' => 'image/gif, image/x-xbitmap, image/jpeg,        image/pjpeg, image/png, */*',   'Accept-Charset' => 'iso-8859-1,*,utf-8',   'Accept-Language' => 'en-US',  );

$browser->cookie_jar( {} );

$response=$browser->post("$cgi?title=Special:Userlogin&action=submitlogin", @ns_headers, Content=>[wpName=>$username,wpPassword=>$password,wpRemember=>"1",wpLoginAttempt=>"Log in"]);

if($response->code!=302 && !$ignore_login_error) { print "We weren't able to login. This could have the following causes:
 * 1) After logging in, we should be redirected to another page.
 * 2) If we aren't, something is wrong.

Solution: Edit upload.pl and change them. Solution: Go to (where?) and get a new version of the upload script. are uploading to has cookie check disabled. Solution: Try setting \$ignore_login_error to 1.
 * The username ($username) or password may be incorrect.
 * The MediaWiki software has been upgraded.
 * You are trying to hack this script for other wikis. The wiki you

Regardless, we will now try to write the output from the server to $dir/debug.txt....\n\n";       open(DEBUG,">$dir/debug.txt") or die "Could not write file.\n";        print DEBUG $response->as_string;        print "This seems to have worked. Take a look at the file for further information.\n";       close(DEBUG);        exit 1; }

foreach $key(keys(%file)) { sleep $pause; print "Uploading $key to the wiki $wiki. Description:\n"; print $file{$key}."\n". "-" x 75. "\n"; uploadfile: $eckey=encode('utf8',$key); if($eckey ne $key) { symlink("$key","$dir/$eckey"); }       $response=$browser->post("$cgi?title=Special:Upload",        @ns_headers,Content_Type=>'form-data',Content=>        [                wpUploadFile=>["$dir/$eckey"],                wpUploadDescription=>encode('utf8',$file{$key}),                wpUploadAffirm=>"1",                wpUpload=>"Upload file",                wpIgnoreWarning=>"1"        ]); push @responses,$response->as_string; if($response->code!=302 && $response->code!=200) { print "Upload failed! Will try again. Output was:\n"; print $response->as_string; goto uploadfile; } else { print "Uploaded successfully.\n"; } }

print "Everything seems to be OK. Log will be written to $dir/debug.txt.\n"; open(DEBUG,">$dir/debug.txt") or die "Could not write file.\n"; print DEBUG @responses;