For our local development setup we have found that moto is using around 25% CPU constantly. Digging in with gdb it turned out that it was burning that CPU in the sleeping loop. Here i'm increasing the sleep by 10x which brings the idle CPU usage down by 10x (to ~2%).
I'm not familiar enough with the moto/sqs codebase to know if lengthening this sleep will have an adverse effect; however, in other Python dev I've noticed that (in Python 2.7 anyway..) Python threading won't context switch a thread until a sleep of at least 0.01 seconds is performed (shockingly long!). So based on this guesswork I suspect sleeping for 0.01 seconds won't cause any grief.
When a queue is created with the ReceiveMessageWaitTimeSeconds
attribute the value is never converted to an integer. When the
ReceiveMessage action is called it tries to compare the string
ReceiveMessageWaitTimeSeconds with the min and max wait times which
raises a TypeError.
The solution is to convert this value to an integer before comparing.
* Add support for FilterPolicy to sns subscription set_filter_attributes
* Add basic support for sns message filtering
This adds support for exact string value matching along with AND/OR
logic as described here:
https://docs.aws.amazon.com/sns/latest/dg/message-filtering.html
It does not provide support for:
- Anything-but string matching
- Prefix string matching
- Numeric Value Matching
The above filter policies (if configured) will not match messages.
* Return correct error code when fetching a queue that does not exist
* Improve SQS Queue get and set attributes
* Queue creation and set_attributes uses the same code path
- ensure bool/int values are cast correctly
* RedrivePolicy is handled properly with set_attributes
- _setup_dlq is called
- is json decoded, so that returned RedrivePolicy is not json
encoded twice
* As per AWS not all attributes are returned when they are not set, for
example RedrivePolicy, FifoQueue, Policy, Kms*
* WaitTimeSeconds is not a queue attribute switch to
ReceiveMessageWaitTimeSeconds
* X-Ray Client SDK patched
Fixes#1250
* Fixed flake8
* Fixed some issues
* Fixed flake8
* Fixed more typos
* Fixed python2 string
* Fixed aws-sdk patch order
* Added more test cases to test the patching
This implements the same MD5 hashing pattern as implemented in the Ruby
and Java AWS SDKs
Doesn't yet handle list types but if you're reading this you might be
surprised how easy that is to add. Give it a shot and if you get stuck
reach out to me for help.
While using moto server with a test SQS client, I noticed significant CPU usage while the client was long polling. I narrowed this down to the `receive_messages` call of the SQS service sitting in a `while True:` statement with no work to be done, thus looping forever.
To produce this issue, I do:
```
$ python3 -m venv venv
$ . ./venv/bin/activate
(venv) $ pip install moto moto[server] boto3
Collecting moto
Downloading moto-0.4.31-py2.py3-none-any.whl (303kB)
--snip--
(venv) $ moto_server sqs &
[1] 31727
* Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
(venv) $ python3
Python 3.6.0 (default, Dec 24 2016, 08:01:42)
[GCC 4.2.1 Compatible Apple LLVM 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> client = boto3.client('sqs', region_name='us-east-1', endpoint_url='http://127.0.0.1:5000');
>>> client.create_queue(QueueName='testing')
127.0.0.1 - - [16/Mar/2017 13:34:20] "POST / HTTP/1.1" 200 -
{'QueueUrl': 'http://sqs.us-east-1.amazonaws.com/123456789012/testing', 'ResponseMetadata': {'RequestId': '7a62c49f-347e-4fc4-9331-6e8e7a96aa73', 'HTTPStatusCode': 200, 'HTTPHeaders': {'content-type': 'text/html; charset=utf-8', 'content-length': '343', 'server': 'Werkzeug/0.12.1 Python/3.6.0', 'date': 'Thu, 16 Mar 2017 20:34:20 GMT'}, 'RetryAttempts': 0}}
>>> client.receive_message(QueueUrl='http://sqs.us-east-1.amazonaws.com/123456789012/testing', MaxNumberOfMessages=10, WaitTimeSeconds=10)
```
At this point the moto server will run at 100% cpu for 10 seconds until the request times out waiting for a message. If multiple clients are continuously reconnected (as in mocking a normal sqs worker setup) the server will sit at 100% cpu indefinitely.
This pull request adds a simple sleep statement to the SQS `receive_messages` call when there are no messages to process. In doing so, the loop will be limited to executing once per 0.001 seconds when the queue is empty. The CPU usage is nearly 0% after this change.
unix_time() from moto.core.utils is used as the time source through moto,
and it is identical to time.time() in output. Hence, using unix_time()
since it makes mocking easier during testing (when time is mocked out).
