Implement Celery/Redis for background tasks #26

New Issue

williamp · 2023-03-23T21:53:51Z

williamp commented

2023-03-23 21:53:51 +00:00

chatgpt analysis:

Using a task queue and a broker like Celery and Redis can be a more scalable and robust solution for running background tasks than using a simple scheduler like APScheduler.

With Celery and Redis, you can define tasks as Python functions and add them to a queue. Workers (which can be running on separate machines) can then consume tasks from the queue and execute them asynchronously. This allows you to distribute the workload across multiple machines and handle large numbers of tasks.

In addition, Celery provides features such as task retries, task prioritization, and task result storage, which can be very useful for certain types of applications.

If you are working with a large number of tasks or if your tasks are computationally expensive, using a task queue like Celery and Redis is definitely worth considering. However, if your tasks are relatively simple and you don't expect a large volume of them, using APScheduler may be sufficient.

In any case, I would recommend looking into Celery and Redis to see if they might be a good fit for your application.

With Celery, you can define multiple task queues and assign different priorities to each queue. This allows you to prioritize critical tasks over less important ones and ensure that your application can handle large volumes of tasks without becoming overloaded.

In addition, Celery provides support for task result storage and monitoring, so you can track the progress of your tasks and be notified if any tasks fail or take too long to complete.

Overall, Celery and Redis can provide a powerful and flexible solution for managing background tasks in your application. I would encourage you to explore this option further and see if it meets your needs.

chatgpt analysis: Using a task queue and a broker like Celery and Redis can be a more scalable and robust solution for running background tasks than using a simple scheduler like APScheduler. With Celery and Redis, you can define tasks as Python functions and add them to a queue. Workers (which can be running on separate machines) can then consume tasks from the queue and execute them asynchronously. This allows you to distribute the workload across multiple machines and handle large numbers of tasks. In addition, Celery provides features such as task retries, task prioritization, and task result storage, which can be very useful for certain types of applications. If you are working with a large number of tasks or if your tasks are computationally expensive, using a task queue like Celery and Redis is definitely worth considering. However, if your tasks are relatively simple and you don't expect a large volume of them, using APScheduler may be sufficient. In any case, I would recommend looking into Celery and Redis to see if they might be a good fit for your application. With Celery, you can define multiple task queues and assign different priorities to each queue. This allows you to prioritize critical tasks over less important ones and ensure that your application can handle large volumes of tasks without becoming overloaded. In addition, Celery provides support for task result storage and monitoring, so you can track the progress of your tasks and be notified if any tasks fail or take too long to complete. Overall, Celery and Redis can provide a powerful and flexible solution for managing background tasks in your application. I would encourage you to explore this option further and see if it meets your needs.

williamp added the

enhancement

label 2023-03-23 21:53:51 +00:00

williamp self-assigned this 2023-03-23 21:53:56 +00:00

williamp added a new dependency 2023-03-23 21:55:17 +00:00

#11 Garbage Collection - Events

williamp added a new dependency 2023-03-23 21:55:38 +00:00

#4 Task expiry/autodelete

williamp added a new dependency 2023-03-23 21:55:57 +00:00

#3 Push Notifications via Websocket

williamp added reference redis-messaging

2023-04-16 14:25:50 +00:00