mongodb - Concurrent tasks workers with celery -

i have mongodb collections 20 in number using store data regarding tasks processing using cron jobs.

i have 1 worker per collection when using cron jobs. want improve arrangement , looking celery. want have @ least 4 workers per collection since have many records in each collection.

i want jobs done come , not wait 5 minutes wait happening when using cron jobs.

is possible me have 4 workers per collection in celery in way have described?.

celery workers pick tasks new task initiated , execute it, celery can use redis, or rabittmq storing tasks queue. day can scale celery running distributed on multiple machines or scaling machine , increasing number of workers. https://www.slideshare.net/nicolasgrasset/scaling-up-task-processing-with-celery

instead of using crontab, use celery beat task scheduler celery.

there no need of having collection wise celery workers.

please go through below celery documentation understanding celery.

http://docs.celeryproject.org/en/latest/getting-started/introduction.html