Laravel queues and supervisor - php runs out of memory and parallel processing issues -


i working on data importing functionality website. using laravel 5.1 on homestead, , have given machine 4gb of ram , 2 virtual cpus. importing data used laravel excel package, chuncking mechanism breaks data set in chunks , adds processing queue. workflow follows:

  • the user uploads excel file;
  • a job dispatched handle excel file;
  • this job chunks file , dispatches multiple other jobs process chunked data;
  • each of these chunks dispatches job background logic on inserted data when done.

i have setup queue named 'import' in supervisor these processes. along queue, have default 1 (for sending mails , other, less intensive, stuff) , low priority queue dispatching jobs other jobs. here supervisor configuration:

[program:laravel-worker] process_name=%(program_name)s_%(process_num)02d command=/usr/bin/php /home/vagrant/code/iris/artisan queue:work --tries=1 --daemon --queue=default,import,background1 autostart=true autorestart=true user=vagrant numprocs=8 redirect_stderr=true stdout_logfile=/home/vagrant/code/iris/storage/logs/worker.log

with smaller files works well, when attempt import ~30k row document, php processes spawned supervisor run out of memory toward end , in laravel log start seeing invalidargumentexception: no handler registered command [__php_incomplete_class] in illuminate\bus\dispatcher (this happens when run 2 imports in parallel or attempt download through websockets) , confused why happening. no 1 process exceeds 512 mb limit far can tell using memory_get_usage(). garbage collector issue? should summon manually?

and since mentioned websockets, trying create separate queue (with higher priority) handling web socket requests. tried multiple supervisor configurations (created dedicated worker in supervisor configuration files, added queue --queue= option in config, etc.) no avail. implemented report downloading through web sockets, , works fine itself, when there other things in queue, socket request gets handled after several items in lower priority 'import' queue finish, leaving me believe not understand queue priorities well. there way have separate queue socket requests responds these requests?

update

i narrowed down memory leak job dispatching @ end of import. however, not able dispatch job when attempting import large file (30k+ rows). errors out invalidargumentexception: no handler registered command [__php_incomplete_class] in illuminate\bus\dispatcher (i think it's because running out of memory) , php processes started supervisor remain active , seem consume same amount of resources did @ peak (about 450 mb each). keeps on going until manually run php artisan queue:restart. smaller files, flow works intendended: runs excel batches , @ end dispatches job queue, processed success.


Comments

Post a Comment

Popular posts from this blog

Sort a complex associative array in PHP -

this question has answer here: php sort array subarray value 5 answers i have associative array in php like: $newarray1 = array ( [producta] => array ( [link] => http://sanjosespartan.com/blog/products/agile-java-development/ [visitcount] => 4 ) [productd] => array ( [link] => http://sanjosespartan.com/blog/products/intro-to-go/ [visitcount] => 1 ) [productg] => array ( [link] => http://sanjosespartan.com/blog/products/node-js-for-developers/ [visitcount] => 1 ) [productb] => array ( [link] => http://sanjosespartan.com/blog/products/beginning-mysql/ [visitcount] => 1 ) [productc] => array ( [link] => http://sanjosespartan.com/blog/products/computer-networks/ [visitcount] => 1 ) ...
Read more

vb.net - How to ignore if a cell is empty nothing -

how use roscolor() ignore value (like empty cell) ? color row if value upper 5 when there nothing in cell, want ignore roscolor() apply, how? public sub roscolor() integer = 0 quotedatagridview1.rows.count() - 1 step +1 dim val integer val = quotedatagridview1.rows(i).cells(3).value if val = vbempty quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.white elseif val < 5 quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.red elseif val > 5 , val < 10 quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.lightyellow end if next end sub you can check empty value following public sub roscolor() integer = 0 quotedatagridview1.rows.count() - 1 step +1 dim val = quotedatagridview1.rows(i).cells(3).value if isdbnull(val) or val nothing quotedatagridview1.rows(i).defaultcellstyle.backcolor = color.white ...
Read more

recursion - Can every recursive algorithm be improved with dynamic programming? -

i first year undergraduate csc student looking competitive programming. recursion involves defining , solving sub problems. understand, top down dynamic programming (dp) involves memoizing solutions sub problems reduce time complexity of algorithm. can top down dp used improve efficiency of every recursive algorithm overlapping sub problems? dp fail work , how can identify this? the short answer is: yes. however, there constraints. obvious 1 recursive calls must overlap. i.e. during execution of algorithm, recursive function must called multiple times same parameters. lets truncate recursion tree memoization. can use memoization reduce number of calls. however, reduction of calls comes price. need store results somewhere. next obvious constraint need have enough memory. comes not-so obvious constraint. memory access requires time. first need find result stored , maybe copy location. in cases, might faster let recursion calculate result instead of loading somewher...
Read more