Reading files recursively in parallel in Perl -
i have 500 files read, reading recursively each file takes 2 minutes approximately. want operation in parallel using perl. how can that?
you're talking massive amount of reading if takes 2 minutes. you're spending time waiting hard drive. files on separate hard drives? if not, why think trying second file @ same time going faster? in fact, might make things slower increasing amount of seeking hard drive has make.
but if want try anyway,
use threads; use thread::queue qw( ); use constant num_workers => 4; # twiddle sub run { ($qfn) = @_; ...read file $qfn here... } $q = thread::queue->new(); @threads; (1..num_workers) { push @threads, async { while (my $job = $q->dequeue()) { run($job); } }; } $q->enqueue($_) @qfns; $q->enqueue(undef) @threads; $_->join() @threads;
Comments
Post a Comment