mongodb - Concurrent tasks workers with celery -


i have mongodb collections 20 in number using store data regarding tasks processing using cron jobs.

i have 1 worker per collection when using cron jobs. want improve arrangement , looking celery. want have @ least 4 workers per collection since have many records in each collection.

i want jobs done come , not wait 5 minutes wait happening when using cron jobs.

is possible me have 4 workers per collection in celery in way have described?.

celery workers pick tasks new task initiated , execute it, celery can use redis, or rabittmq storing tasks queue. day can scale celery running distributed on multiple machines or scaling machine , increasing number of workers. https://www.slideshare.net/nicolasgrasset/scaling-up-task-processing-with-celery

instead of using crontab, use celery beat task scheduler celery.

there no need of having collection wise celery workers.

please go through below celery documentation understanding celery.

http://docs.celeryproject.org/en/latest/getting-started/introduction.html


Comments

Popular posts from this blog

resizing Telegram inline keyboard -

command line - How can a Python program background itself? -

php - "cURL error 28: Resolving timed out" on Wordpress on Azure App Service on Linux -