使用 Celery 创建动态队列

问题描述

这是我的场景:

当用户登录我的网站时，我为给定用户排队了一堆任务(通常每个任务需要 100 毫秒，每个用户有 100 多个任务).这些任务排队到默认的 Celery Queue 中，我有 100 名工作人员正在运行.当后端任务完成时，我使用 websockets 向用户显示实时进度.如果我只有 1 或 2 个活跃用户，生活就会很美好.

When a user logs in to my website, I queue up a bunch of tasks for the given user (typically each task takes 100s of msecs and there are 100s of tasks per user). These tasks are queued to the default Celery Queue and I have 100s of workers running. I use websockets to show the user real-time progress as the tasks complete on the backend. Life is good if I have just 1 or 2 users active.

现在，如果我有几个并发用户登录到我的站点，后面的用户会排在初始用户后面，他们的任务就会饿死(因为所有任务都进入同一个队列).我的想法是为每个用户创建一个动态队列以确保公平.但是根据芹菜文档(http://docs.celeryproject.org/en/latest/userguide/routing.html#defining-queues)，看来我需要静态定义队列.

Now if I a few concurrent users log-in to my site, the latter users are queued behind the initial users and their tasks starve (since all the tasks go to the same queue). My thoughts are to create a dynamic queue per user to ensure fairness. However as per Celery documentation (http://docs.celeryproject.org/en/latest/userguide/routing.html#defining-queues), seems I need to be define the queues statically.

关于在我的场景中使用 celery 的最佳实践有什么建议吗?

Any suggestions on best practices for using celery for my scenario?

推荐答案

http://docs.celeryproject.org/en/latest/userguide/workers.html#queues

celery -A proj control add_consumer foo -d worker1.local

同样可以使用 app.control.add_consumer() 方法动态完成:

The same can be accomplished dynamically using the app.control.add_consumer() method:

app.control.add_consumer('foo', reply=True)
[{u'worker1.local': {u'ok': u"already consuming from u'foo'"}}]

app.control.add_consumer('foo', reply=True,
destination=['worker1@example.com'])

这篇关于使用 Celery 创建动态队列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持跟版网！