本帖最后由 523066680 于 2021-5-9 09:49 编辑
我打开txt文件,疑似开起了车车复制代码 就让我想起了之前抓W4B图片的脚本,Perl写的。N个常驻线程,循环获取任务下载。- sub thread_func
- {
- our (@mission, @headers, $progress, $total);
- my ($ua) = @_;
- my $target;
-
- while ( $#mission >= 0 )
- {
- $progress++;
- $target = shift @mission;
-
- printf "[%d] [%03d/%03d] %s\n", threads->tid(), $progress, $total;
- get_pics( threads->tid(), $ua, $target );
- }
- }
复制代码 不过最近又学了一下,mojo本来就支持多线程- my $ua = Mojo::UserAgent->new()->with_roles('+Queued');;
- $ua->max_active(10); #限制并发数
- for ( 1 .. 10086 )
- {
- # Non-blocking request
- $ua->get( $url => sub ($ua, $tx) { say gbk($tx->result->dom->at("title")->text); }
- }
- $loop->start unless $loop->is_running;
复制代码 ---
按理说 aria2 这么时髦的工具,应该也是内置支持多线程下载的,-j, --max-concurrent-downloads=<N>- aria2c.exe -i links.txt -j 6
复制代码
- 05/09 09:46:09 [NOTICE] Downloading 20 item(s)
- [DL:0B][#ea89d8 0B/0B][#2c34b3 0B/0B][#8e6c5a 0B/0B][#8fb920 0B/0B][#9ca066 0B/0B](+1)
- 05/09 09:46:10 [NOTICE] Download complete: D:/temp/0001-06.jpg
- 05/09 09:46:10 [NOTICE] Download complete: D:/temp/0001-01.jpg
- 05/09 09:46:10 [NOTICE] Download complete: D:/temp/0001-02.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-05.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-04.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-03.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-07.jpg
- [DL:2.4MiB][#6fe37d 112KiB/314KiB(35%)][#14a780 0B/0B][#6124fa 0B/0B][#e3781d 0B/0B][#19422e 0B/0B](+1)
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-08.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-12.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-13.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-09.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-10.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-11.jpg
- 05/09 09:46:11 [NOTICE] Download complete: D:/temp/0001-16.jpg
- 05/09 09:46:12 [NOTICE] Download complete: D:/temp/0001-14.jpg
- 05/09 09:46:12 [NOTICE] Download complete: D:/temp/0001-17.jpg
- 05/09 09:46:12 [NOTICE] Download complete: D:/temp/0001-15.jpg
- [DL:2.8MiB][#9b3b42 112KiB/227KiB(49%)][#1ae2bd 96KiB/376KiB(25%)][#ee53c9 256KiB/256KiB(99%)]
- 05/09 09:46:12 [NOTICE] Download complete: D:/temp/0001-20.jpg
- 05/09 09:46:12 [NOTICE] Download complete: D:/temp/0001-18.jpg
- 05/09 09:46:12 [NOTICE] Download complete: D:/temp/0001-19.jpg
复制代码
|