This implements the same MD5 hashing pattern as implemented in the Ruby
and Java AWS SDKs
Doesn't yet handle list types but if you're reading this you might be
surprised how easy that is to add. Give it a shot and if you get stuck
reach out to me for help.
While using moto server with a test SQS client, I noticed significant CPU usage while the client was long polling. I narrowed this down to the `receive_messages` call of the SQS service sitting in a `while True:` statement with no work to be done, thus looping forever.
To produce this issue, I do:
```
$ python3 -m venv venv
$ . ./venv/bin/activate
(venv) $ pip install moto moto[server] boto3
Collecting moto
Downloading moto-0.4.31-py2.py3-none-any.whl (303kB)
--snip--
(venv) $ moto_server sqs &
[1] 31727
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
(venv) $ python3
Python 3.6.0 (default, Dec 24 2016, 08:01:42)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> client = boto3.client('sqs', region_name='us-east-1', endpoint_url='http://127.0.0.1:5000');
>>> client.create_queue(QueueName='testing')
127.0.0.1 - - [16/Mar/2017 13:34:20] "POST / HTTP/1.1" 200 -
{'QueueUrl': 'http://sqs.us-east-1.amazonaws.com/123456789012/testing', 'ResponseMetadata': {'RequestId': '7a62c49f-347e-4fc4-9331-6e8e7a96aa73', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'text/html; charset=utf-8', 'content-length': '343', 'server': 'Werkzeug/0.12.1 Python/3.6.0', 'date': 'Thu, 16 Mar 2017 20:34:20 GMT'}, 'RetryAttempts': 0}}
>>> client.receive_message(QueueUrl='http://sqs.us-east-1.amazonaws.com/123456789012/testing', MaxNumberOfMessages=10, WaitTimeSeconds=10)
```
At this point the moto server will run at 100% cpu for 10 seconds until the request times out waiting for a message. If multiple clients are continuously reconnected (as in mocking a normal sqs worker setup) the server will sit at 100% cpu indefinitely.
This pull request adds a simple sleep statement to the SQS `receive_messages` call when there are no messages to process. In doing so, the loop will be limited to executing once per 0.001 seconds when the queue is empty. The CPU usage is nearly 0% after this change.
unix_time() from moto.core.utils is used as the time source through moto,
and it is identical to time.time() in output. Hence, using unix_time()
since it makes mocking easier during testing (when time is mocked out).
Previously, receive_message would always use the queue's default
visibility timeout instead of the value passed as a query parameter when
calling the receive_message method on an SQS connection.
- Support for delaying messages
- Support for visibility timeouts
- Support for actually deleting messages
- Support for message bodies longer than 27 characters
- Support for message attributes
- Support for accurate queue attributes
Caveats:
- All message attributes are returned regardless of whether or not
attributes were requested when reading messages
- I’m not sure why messages longer than 27 characters were breaking in
my tests. Boto seems to expect the body to be base64 encoded and bodies
less than 27 characters would be fine, but if I attempted to use a
larger body it would mangle the content. I now base64 encode the body
if the raw string is longer than 27 characters and all is fine.
