[{"id":"35334003585","type":"WatchEvent","actor":{"id":18518,"login":"yoppi","display_login":"yoppi","gravatar_id":"","url":"https://api.github.com/users/yoppi","avatar_url":"https://avatars.githubusercontent.com/u/18518?"},"repo":{"id":174817300,"name":"nalepae/pandarallel","url":"https://api.github.com/repos/nalepae/pandarallel"},"payload":{"action":"started"},"public":true,"created_at":"2024-02-02T02:17:30Z"},{"id":"35005169893","type":"IssueCommentEvent","actor":{"id":18518,"login":"yoppi","display_login":"yoppi","gravatar_id":"","url":"https://api.github.com/users/yoppi","avatar_url":"https://avatars.githubusercontent.com/u/18518?"},"repo":{"id":184460,"name":"celery/celery","url":"https://api.github.com/repos/celery/celery"},"payload":{"action":"created","issue":{"url":"https://api.github.com/repos/celery/celery/issues/8673","repository_url":"https://api.github.com/repos/celery/celery","labels_url":"https://api.github.com/repos/celery/celery/issues/8673/labels{/name}","comments_url":"https://api.github.com/repos/celery/celery/issues/8673/comments","events_url":"https://api.github.com/repos/celery/celery/issues/8673/events","html_url":"https://github.com/celery/celery/issues/8673","id":2011870096,"node_id":"I_kwDOAALQjM536rOQ","number":8673,"title":"queue_order_strategy = priority randomizes queue order","user":{"login":"crabhi","id":1015246,"node_id":"MDQ6VXNlcjEwMTUyNDY=","avatar_url":"https://avatars.githubusercontent.com/u/1015246?v=4","gravatar_id":"","url":"https://api.github.com/users/crabhi","html_url":"https://github.com/crabhi","followers_url":"https://api.github.com/users/crabhi/followers","following_url":"https://api.github.com/users/crabhi/following{/other_user}","gists_url":"https://api.github.com/users/crabhi/gists{/gist_id}","starred_url":"https://api.github.com/users/crabhi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/crabhi/subscriptions","organizations_url":"https://api.github.com/users/crabhi/orgs","repos_url":"https://api.github.com/users/crabhi/repos","events_url":"https://api.github.com/users/crabhi/events{/privacy}","received_events_url":"https://api.github.com/users/crabhi/received_events","type":"User","site_admin":false},"labels":[{"id":235024002,"node_id":"MDU6TGFiZWwyMzUwMjQwMDI=","url":"https://api.github.com/repos/celery/celery/labels/Issue%20Type:%20Bug%20Report","name":"Issue Type: Bug Report","color":"e11d21","default":false,"description":""}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2023-11-27T09:35:48Z","updated_at":"2024-01-22T07:34:53Z","closed_at":null,"author_association":"NONE","active_lock_reason":null,"body":"\r\n# Checklist\r\n\r\n- [x] I have verified that the issue exists against the `main` branch of Celery.\r\n- [ ] This has already been asked to the [discussions forum](https://github.com/celery/celery/discussions) first.\r\n- [x] I have read the relevant section in the\r\n [contribution guide](https://docs.celeryq.dev/en/main/contributing.html#other-bugs)\r\n on reporting bugs.\r\n- [x] I have checked the [issues list](https://github.com/celery/celery/issues?q=is%3Aissue+label%3A%22Issue+Type%3A+Bug+Report%22+-label%3A%22Category%3A+Documentation%22)\r\n for similar or identical bug reports.\r\n- [x] I have checked the [pull requests list](https://github.com/celery/celery/pulls?q=is%3Apr+label%3A%22PR+Type%3A+Bugfix%22+-label%3A%22Category%3A+Documentation%22)\r\n for existing proposed fixes.\r\n- [x] I have checked the [commit log](https://github.com/celery/celery/commits/main)\r\n to find out if the bug was already fixed in the main branch.\r\n- [x] I have included all related issues and possible duplicate issues\r\n in this issue (If there are none, check this box anyway).\r\n\r\n## Mandatory Debugging Information\r\n\r\n- [x] I have included the output of ``celery -A proj report`` in the issue.\r\n (if you are not able to do this, then at least specify the Celery\r\n version affected).\r\n- [x] I have verified that the issue exists against the `main` branch of Celery.\r\n- [x] I have included the contents of ``pip freeze`` in the issue.\r\n- [x] I have included all the versions of all the external dependencies required\r\n to reproduce this bug.\r\n\r\n## Optional Debugging Information\r\n\r\n- [ ] I have tried reproducing the issue on more than one Python version\r\n and/or implementation.\r\n- [ ] I have tried reproducing the issue on more than one message broker and/or\r\n result backend.\r\n- [ ] I have tried reproducing the issue on more than one version of the message\r\n broker and/or result backend.\r\n- [ ] I have tried reproducing the issue on more than one operating system.\r\n- [ ] I have tried reproducing the issue on more than one workers pool.\r\n- [ ] I have tried reproducing the issue with autoscaling, retries,\r\n ETA/Countdown & rate limits disabled.\r\n- [ ] I have tried reproducing the issue after downgrading\r\n and/or upgrading Celery and its dependencies.\r\n\r\n## Related Issues and Possible Duplicates\r\n\r\n\r\n#### Related Issues\r\n\r\n- #5127 \r\n\r\n#### Possible Duplicates\r\n\r\n- None\r\n\r\n## Environment & Settings\r\n\r\n**Celery version**: main (5.3.6)\r\n\r\n\r\ncelery report
Output:
\r\n\r\n\r\n```\r\n\r\nsoftware -> celery:5.3.6 (emerald-rush) kombu:5.3.4 py:3.10.12\r\n billiard:4.2.0 redis:5.0.1\r\nplatform -> system:Linux arch:64bit, ELF\r\n kernel version:6.2.0-37-generic imp:CPython\r\nloader -> celery.loaders.app.AppLoader\r\nsettings -> transport:redis results:disabled\r\n\r\nbroker_url: 'redis://127.0.0.1:6379/0'\r\ndeprecated_settings: None\r\nbroker_transport_options: \r\n 'queue_order_strategy': 'priority'}\r\n\r\n```\r\n\r\n
\r\n \r\n\r\n# Steps to Reproduce\r\n\r\n## Required Dependencies\r\n\r\n- **Minimal Python Version**: N/A or Unknown\r\n- **Minimal Celery Version**: N/A or Unknown\r\n- **Minimal Kombu Version**: N/A or Unknown\r\n- **Minimal Broker Version**: N/A or Unknown\r\n- **Minimal Result Backend Version**: N/A or Unknown\r\n- **Minimal OS and/or Kernel Version**: N/A or Unknown\r\n- **Minimal Broker Client Version**: N/A or Unknown\r\n- **Minimal Result Backend Client Version**: N/A or Unknown\r\n\r\n### Python Packages\r\n\r\n\r\npip freeze
Output:
\r\n\r\n\r\n```\r\namqp==5.2.0\r\nasync-timeout==4.0.3\r\nbilliard==4.2.0\r\ncelery @ file:///home/krab/workspace/celery\r\nclick==8.1.7\r\nclick-didyoumean==0.3.0\r\nclick-plugins==1.1.1\r\nclick-repl==0.3.0\r\nkombu==5.3.4\r\nprompt-toolkit==3.0.41\r\npython-dateutil==2.8.2\r\nredis==5.0.1\r\nsix==1.16.0\r\ntzdata==2023.3\r\nvine==5.1.0\r\nwcwidth==0.2.12\r\n\r\n```\r\n\r\n
\r\n \r\n\r\n### Other Dependencies\r\n\r\n\r\n\r\nN/A\r\n
\r\n \r\n\r\n## Minimally Reproducible Test Case\r\n\r\n\r\n\r\n\r\n\r\n`proj.py`:\r\n\r\n```python\r\nimport time\r\n\r\nfrom celery import Celery\r\n\r\napp = Celery('proj', broker='redis://127.0.0.1:6379/0')\r\napp.conf.update(broker_transport_options={\r\n 'queue_order_strategy': 'priority'\r\n})\r\n\r\n@app.task\r\ndef t(q):\r\n print(f'{q} start')\r\n time.sleep(1)\r\n\r\n```\r\n\r\nTasks scheduling:\r\n\r\n```python\r\nIn [1]: from proj import t\r\n\r\nIn [2]: for i in range(5000):\r\n ...: q = 'high' if i %2 == 0 else 'low'\r\n ...: t.apply_async([q], queue=q)\r\n```\r\n\r\nWorker CLI:\r\n\r\n```\r\ncelery -A proj worker -Q high,low --concurrency 1\r\n```\r\n\r\nStart the worker, let it process a few tasks and kill it. Repeat and observe which tasks get processed.\r\n\r\n
\r\n \r\n\r\n# Expected Behavior\r\nI expect that every time, tasks from the `high` queue are consumed first, then the tasks from the `low` queue.\r\n\r\n# Actual Behavior\r\n\r\n\r\nThe worker always consumes tasks from one queue first. However, which one it is varies between restarts. If I start `celery -A proj worker -Q high,low --concurrency 1`, it may consume tasks from `high`. When I kill it and start again, it may start picking tasks from `low` instead.\r\n\r\nI think the culprit may be that `kombu.transport.redis.Channel` uses a set for its `active_queues` property. Therefore the strategy itself receives the queues in effectively randomized order.","reactions":{"url":"https://api.github.com/repos/celery/celery/issues/8673/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0},"timeline_url":"https://api.github.com/repos/celery/celery/issues/8673/timeline","performed_via_github_app":null,"state_reason":null},"comment":{"url":"https://api.github.com/repos/celery/celery/issues/comments/1903411418","html_url":"https://github.com/celery/celery/issues/8673#issuecomment-1903411418","issue_url":"https://api.github.com/repos/celery/celery/issues/8673","id":1903411418,"node_id":"IC_kwDOAALQjM5xc8Da","user":{"login":"yoppi","id":18518,"node_id":"MDQ6VXNlcjE4NTE4","avatar_url":"https://avatars.githubusercontent.com/u/18518?v=4","gravatar_id":"","url":"https://api.github.com/users/yoppi","html_url":"https://github.com/yoppi","followers_url":"https://api.github.com/users/yoppi/followers","following_url":"https://api.github.com/users/yoppi/following{/other_user}","gists_url":"https://api.github.com/users/yoppi/gists{/gist_id}","starred_url":"https://api.github.com/users/yoppi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yoppi/subscriptions","organizations_url":"https://api.github.com/users/yoppi/orgs","repos_url":"https://api.github.com/users/yoppi/repos","events_url":"https://api.github.com/users/yoppi/events{/privacy}","received_events_url":"https://api.github.com/users/yoppi/received_events","type":"User","site_admin":false},"created_at":"2024-01-22T07:34:52Z","updated_at":"2024-01-22T07:34:52Z","author_association":"NONE","body":"@crabhi \r\nYou can specify the order of the queues by specifying options.\r\nsee. https://github.com/celery/kombu/blob/2d6d8baf0f9cf6812c70b7ddb06b242bf31c8ad0/kombu/transport/redis.py#L673-L678 \r\n","reactions":{"url":"https://api.github.com/repos/celery/celery/issues/comments/1903411418/reactions","total_count":0,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0},"performed_via_github_app":null}},"public":true,"created_at":"2024-01-22T07:34:53Z","org":{"id":319983,"login":"celery","gravatar_id":"","url":"https://api.github.com/orgs/celery","avatar_url":"https://avatars.githubusercontent.com/u/319983?"}},{"id":"34990497146","type":"WatchEvent","actor":{"id":18518,"login":"yoppi","display_login":"yoppi","gravatar_id":"","url":"https://api.github.com/users/yoppi","avatar_url":"https://avatars.githubusercontent.com/u/18518?"},"repo":{"id":288487894,"name":"docker/setup-qemu-action","url":"https://api.github.com/repos/docker/setup-qemu-action"},"payload":{"action":"started"},"public":true,"created_at":"2024-01-21T08:42:13Z","org":{"id":5429470,"login":"docker","gravatar_id":"","url":"https://api.github.com/orgs/docker","avatar_url":"https://avatars.githubusercontent.com/u/5429470?"}}]