Prevent 100% cpu usage while SQS long polling on an empty queue
While using moto server with a test SQS client, I noticed significant CPU usage while the client was long polling. I narrowed this down to the `receive_messages` call of the SQS service sitting in a `while True:` statement with no work to be done, thus looping forever. To produce this issue, I do: ``` $ python3 -m venv venv $ . ./venv/bin/activate (venv) $ pip install moto moto[server] boto3 Collecting moto Downloading moto-0.4.31-py2.py3-none-any.whl (303kB) --snip-- (venv) $ moto_server sqs & [1] 31727 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit) (venv) $ python3 Python 3.6.0 (default, Dec 24 2016, 08:01:42) [GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import boto3 >>> client = boto3.client('sqs', region_name='us-east-1', endpoint_url='http://127.0.0.1:5000'); >>> client.create_queue(QueueName='testing') 127.0.0.1 - - [16/Mar/2017 13:34:20] "POST / HTTP/1.1" 200 - {'QueueUrl': 'http://sqs.us-east-1.amazonaws.com/123456789012/testing', 'ResponseMetadata': {'RequestId': '7a62c49f-347e-4fc4-9331-6e8e7a96aa73', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'text/html; charset=utf-8', 'content-length': '343', 'server': 'Werkzeug/0.12.1 Python/3.6.0', 'date': 'Thu, 16 Mar 2017 20:34:20 GMT'}, 'RetryAttempts': 0}} >>> client.receive_message(QueueUrl='http://sqs.us-east-1.amazonaws.com/123456789012/testing', MaxNumberOfMessages=10, WaitTimeSeconds=10) ``` At this point the moto server will run at 100% cpu for 10 seconds until the request times out waiting for a message. If multiple clients are continuously reconnected (as in mocking a normal sqs worker setup) the server will sit at 100% cpu indefinitely. This pull request adds a simple sleep statement to the SQS `receive_messages` call when there are no messages to process. In doing so, the loop will be limited to executing once per 0.001 seconds when the queue is empty. The CPU usage is nearly 0% after this change.
This commit is contained in:
parent
a5da348fba
commit
83084bf2af
@ -289,6 +289,11 @@ class SQSBackend(BaseBackend):
|
||||
|
||||
# queue.messages only contains visible messages
|
||||
while True:
|
||||
if len(queue.messages) == 0:
|
||||
import time
|
||||
time.sleep(0.001)
|
||||
continue
|
||||
|
||||
for message in queue.messages:
|
||||
if not message.visible:
|
||||
continue
|
||||
|
Loading…
Reference in New Issue
Block a user