The response returned for sns.get_endpoint_attributes was not in
sync with the actual response from boto.
Co-authored-by: Antillon, Alejandro <alejandro.antillon@f-secure.com>
These tests, when run, do not execute any `moto` code. They fail the
parameter validation check in `botocore`, which raises an exception
before ever sending a request. These tests do not cover or verify
any `moto` behavior and have been removed.
The `botocore` response parsers are forgiving when it comes to timestamps,
but a real AWS backend does return time zone details for this attribute.
Verified failure/fix using the Go repo included in the issue report.
Fixes#3516
* Address `boto` deprecation warnings
This commit eliminates the following warning:
../boto/ec2/connection.py:582:
PendingDeprecationWarning: The current get_all_instances implementation will be replaced with get_all_reservations.
`boto` isn't likely to ever make good on this warning, but doing the replacement will
declutter the `moto` test output.
* Remove `invoke_lambda` tracebacks from unit test logging
If an exception is encountered, the details are returned in the response payload.
Printing the traceback was just adding noise to the pytest output.
* Use known AMIs in unit tests
This commit eliminates the following warning in the pytest output:
`PendingDeprecationWarning: Could not find AMI with image-id:ami-123456, in the near future this will cause an error.`
Known, pre-loaded AMI image ids are used instead of random ids that don't actually
exist in the moto backend. The integrity of the tests is unaffected by this change.
A test has been added to provide explicit coverage of the PendingDeprecationWarning
raised when an invalid AMI image id is passed to moto.
The latest release of `botocore` (1.19.62) makes changes to the parameter
validation code, which for some reason was also covered by a couple of
`moto` tests.
These tests, when run, do not execute any `moto` code. They fail the
parameter validation check in `botocore`, which raises an exception
before ever sending a request. These tests do not cover or verify
any `moto` behavior and have been removed.
Ref: ff8ae76eccCloses#3627
Moto's implementation of autoscaling:CreateLaunchConfiguration is a little too loose,
allowing the ImageId parameter to be omitted and defaulting it to None, which results
in dozens of deprecation warnings:
`PendingDeprecationWarning: Could not find AMI with image-id:ami-123456, in the near future this will cause an error.`
This commit refactors calls to CreateLaunchConfiguration to explicitly pass in a known
AMI ImageId in order to avoid these warnings.
The integrity of the tests is unaffected by this change.
This commit eliminates the following warning (of which there are currently dozens):
../boto/ec2/connection.py:582:
PendingDeprecationWarning: The current get_all_instances implementation will be replaced with get_all_reservations.
`boto` isn't likely to ever make good on this warning, but doing the replacement will
declutter the `moto` test output.
* Added redshift.get_cluster_credentials
* Marked endpoint in list
* Removed f string from tests
* Python 2.7 compat changes
* Fixed parameter retrieval
* Formatting
* Removed try/catch in favor of if
* Changed to existing random_string util
Co-authored-by: Andrea Amorosi <aamorosi@amazon.es>
* Fix ec2 filter by empty tag value
Return `None` instead of an empty string when the tag key does not exist
and replace the falsy check with a more explicit `is None`, which allows
empty string values to correctly pass through the filter comparator.
Behavior confirmed against a real AWS backend.
Closes#3603
* Make test case more explicit
Test case now pulled directly from the issue report (#3603).
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* Initial attempt to mock AWS Media Live create_channel endpoint. Test fails.
* Completes basic implementation of Media Live create_channel endpoint
* Completes basic implementation of Media Live list_channels endpoint
* Adds skaffolds for describe_channel and delete_channel
* Adds unit test for delete_channel
* Adds unit test for describe_channel
* Reduces repetitive code by introducing a Channel model
* Implements MediaLive start_channel and stop_channel endpoints
* Fixes lack of support for the dash character in resource ARNs
* Implements MediaLive update_channel endpoint.
* Implements MediaLive create_input endpoint (and Input model).
* Implements MediaLive describe_input endpoint.
* Implements MediaLive list_inputs endpoint.
* Implements MediaLive update_input endpoint.
* Addse server tests for MediaLive
* Adds further url patterns for medialive
* Fixes url patterns
* Fixes url patterns
* Added explicit exception raise when no stack found.
Currently, any operation that uses 'get_stack' method from 'CloudFormationBackend' class
will fail with AttributeError or jinja2 exception if ran against non-existing stack(created/deleted)
To fix the issue I explicitly raised a 'ValidationError' exception.
Added tests for boto and boto3 responses.
* Moved non-existing stack tests to 'test_stack_events'
When using 'update_stack' to test raising an exception when the stack doesn't exist
test coverage dropped by 0.5%. I am using stack_events instead.
* Removed some unreachable paths
After adding the exception couple of paths in the code are unreachable as 'get_stack' doesn't return 'None' anymore.
This is the reason why coverall was reporting decreased coverage.
* Removed an unreachable path I missed
* Added couple of tests in cloudformation/models
* Added more assertions around raised exception
* Formatted document using black to fix issue with travis.
This test is flaky, but when it fails we don't get any indication as to why.
This commit ensures that the reason for failure will be part of the assertion
message.
Once we have information about why this test fails, we can troubleshoot further
and hopefully come up with a permanent fix.
Instead of modifying responses._default_mock, create our own
responses.RequestsMock object that we can modify as needed without
interfering with other users of the responses library.
Fixes#3264.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
* Add get_function_configuration support for Lambda
* remove unnesecary code from test and use _lambda_region when asserting
* rename function and skip coping configuration
* run black formatting
This handles the add-to-list case using the legacy `AttributeUpdates` parameter.
* Added test coverage.
* Verified against real AWS backend.
Closes#3561
The mocked response for ECS RegisterTaskDefinition has drifted from what
actually returns when run against a real ECS endpoint. I created a
minimal task definition for both EC2:
```
>>> ecs.register_task_definition(
family="moto",
containerDefinitions=[
{
"name": "hello_world",
"image": "hello-world:latest",
"memory": 400
}
]
)["taskDefinition"]
{'taskDefinitionArn': 'arn:aws:ecs:us-east-1:************:task-definition/moto:1',
'containerDefinitions': [{'name': 'hello_world',
'image': 'hello-world:latest',
'cpu': 0,
'memory': 400,
'portMappings': [],
'essential': True,
'environment': [],
'mountPoints': [],
'volumesFrom': []}],
'family': 'moto',
'revision': 1,
'volumes': [],
'status': 'ACTIVE',
'placementConstraints': [],
'compatibilities': ['EC2']}
```
and FARGATE:
```
>>> ecs.register_task_definition(
family="moto",
containerDefinitions=[
{
"name": "hello_world",
"image": "hello-world:latest",
"memory": 400
}
],
requiresCompatibilities=["FARGATE"],
networkMode="awsvpc",
cpu="256",
memory="512"
)["taskDefinition"]
{'taskDefinitionArn': 'arn:aws:ecs:us-east-1:************:task-definition/moto:2',
'containerDefinitions': [{'name': 'hello_world',
'image': 'hello-world:latest',
'cpu': 0,
'memory': 400,
'portMappings': [],
'essential': True,
'environment': [],
'mountPoints': [],
'volumesFrom': []}],
'family': 'moto',
'networkMode': 'awsvpc',
'revision': 2,
'volumes': [],
'status': 'ACTIVE',
'requiresAttributes': [{'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.18'},
{'name': 'ecs.capability.task-eni'}],
'placementConstraints': [],
'compatibilities': ['EC2', 'FARGATE'],
'requiresCompatibilities': ['FARGATE'],
'cpu': '256',
'memory': '512'}
```
This change adds several default keys to the task based on those two
real responses and the AWS documentation:
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RegisterTaskDefinition.html
The mock still doesn't match the real response exactly but it's much
closer than it was before.
* Properly coerce `privateDnsEnabled` to boolean value when parsing requests.
* Per AWS spec, default `privateDnsEnabled` request value to `True`.
* Properly serialize `privateDnsEnabled` as boolean value in responses.
* Add test coverage.
Ref: #3540
Applies the user credentials pattern from the ADMIN_NO_SRP_AUTH flow
to the ADMIN_USER_PASSWORD_AUTH auth flow for Cognito admin_initiate_auth
requests.
Co-authored-by: Robin Wilkins <r.wilkins@waracle.com>
* fix https://github.com/localstack/localstack/issues/3339
* fixe lint issues
* Fix review comments
- move deduplication time to constants
- make tests parameterized
- update tests as per review comments
* change variable name expectedCount => expected_count
* fix tests for python 2.7
increase deduplication mock config to account for delays
* ignore time mocking test in server mode
These tests were passing with TZ=UTC, but under a non-UTC timezone
they were failing:
E AssertionError: given
E X = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzutc()), …}, …]
E and
E Y = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzlocal()), …}, …]
E X[0]['timestamp'] != Y[0]['timestamp']
With this fix, they pass either way.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
* Pass the "default" cluster
* Mock ECS exceptions more accurately
Moto's mock ECS has drifted fairly far from the actual ECS API in terms
of which exceptions it throws. This change begins to bring mock ECS's
exceptions in line with actual ECS exceptions. Most notably:
- Several custom exceptions have been replaced with their real ECS
exception. For example, "{0} is not a cluster" has been replaced with
ClusterNotFoundException
- Tests have been added to verify (most of) these exceptions work
correctly. The test coverage was a little spotty to begin with.
- The new exceptions plus the change to pass the "default" cluster
exposed a lot of places where mock ECS was behaving incorrectly. For
example, the ListTasks action is always scoped to a single cluster in
ECS but it listed tasks for all clusters in the mock. I've minimally
updated the tests to make them pass, but there's lots of opportunity to
refactor both this method's test and its implementation.
This does not provide full coverage of exceptions. In general, I ran
these operations against actual ECS resources and cross-referenced the
documentation to figure out what actual exceptions should be thrown and
what the messages should be. Consequently, I didn't update any
exceptions that took more than trivial amount of time to reproduce with
real resources.
* Retrieve SAML Attribute by Name instead of relying on order which is too fragile
* Handle case when SAML Attribute SessionDuration is not provided, as it is not a required attribute from SAML response
When session duration not provided, AWS consider by default a duration of one hour as cited in the following documentation:
"If this attribute is not present, then the credential last for one hour (the default value of the DurationSeconds parameter of the AssumeRoleWithSAML API)."
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml_assertions.html#saml_role-session-duration
Traceback was:
[...]
File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/responses.py", line 79, in assume_role_with_saml
role = sts_backend.assume_role_with_saml(
File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/models.py", line 99, in assume_role_with_saml
role = AssumedRole(**kwargs)
TypeError: __init__() missing 1 required positional argument: 'duration'
* Process saml xml namespaces properly instead of relying on textual prefix that can vary between identity providers
* Handle when SAML response AttributeValue xml tag contains attributes that force xmltodict to build a dictionary as for complex types instead of directly returning string value
Leverage force_cdata option of xmltodict parser that always return a complex dictionary even if xml tag contains only text and no attributes.
* Improve existing test_assume_role_with_saml to be coherent with other assume_role_with_saml tests and remove dead code at the same time
By definition, `single-node` clusters can only consist of 1 node. Likewise,
`multi-node` clusters must have 2 or more nodes.
* Ensure `ClusterType` parameter is either `multi-node` or `single-node`.
* Ensure proper validation of `NumberOfNodes` parameter based on `ClusterType`
parameter.
* Fix existing test case that incorrectly allowed a `multi-node` cluster to
consist of 1 node.
* Add dedicated test for resizing a cluster from `single-node` to `multi-node`
and back again.
Behavior and error messages have been verified against a real AWS backend.
* Add format command to makefile
* Refactor executions to be a attribute of StateMachine
* Begin to add tests for execution history
* Add tests for failed and successful event histories, with implementations
* Add failure case to environment var check
* Skip test if in server mode and update implementation coverage
* Add conditional import for mock to cover python 2
* Refactor stop execution logic into StateMachine
* Refactor event history environment variable into settings.py
* Remove typing and os import
The `boto` library (long ago superseded by `boto3`) has not had an official
release in over two years or even a commit in the last 18 months. Importing
the package (or indirectly importing it by via `moto`) generates a deprecation
warning. Additionally, an ever-increasing number of `moto` users who have
left `boto` behind for `boto3` are still being forced to install `boto`.
This commit vendors a very small subset of the `boto` library--only the code
required by `moto` to run--into the /packages subdirectory. A README file
is included explaining the origin of the files and a recommendation for how
they can be removed entirely in a future release.
NOTE: Users of `boto` will still be able to use `moto` after this is merged.
closes#2978closes#3013closes#3170closes#3418
relates to #2950
A test added in #2401 copied the name of an existing test, preventing it
from being run. This commit renames the second test, allowing both to
be picked up by the test runner.
Original code was trying to raise a ClientError directly. Change to
appropriate Redshift exception class.
* Fix test assertion for `boto`.
* Add test coverage for `boto3`.
Previous code would raise `TypeError: 'dict_keys' object is not subscriptable`
when run under Python 3.
* Re-write code in Python 2/3 compatible way.
* Add clarifying comment.
* Add test coverage.
Supersedes #3227
The previous code was trying to raise a botocore ClientError directly, which
was actually generating a secondary AttributeError because the arguments passed
to ClientError() were incorrect.
This replaces the ClientError() call with a proper moto exception class for
Redshift and fixes the test assertions appropriately.
Supersedes #1957
* Fix:CloudWatch List metrics with dimensions
* Fix:CloudWatch List metrics with dimensions
* Fixed new cases and added more tests
Co-authored-by: usmankb <usman@krazybee.com>
* Extract Duplicate Code into Helper Method
DRY up the tests and replace the arbitrary `sleep()` calls with a more
explicit check before progressing.
* Improve Testing of batch:TerminateJob
The test now confirms that the job was terminated by sandwiching a `sleep`
command between two `echo` commands. In addition to the original checks
of the terminated job status/reason, the test now asserts that only the
first echo command succeeded, confirming that the job was indeed terminated
while in progress.
* Fix Race Condition in batch:SubmitJob
The `test_submit_job` in `test_batch.py` kicks off a job, calls `describe_jobs`
in a loop until the job status returned is SUCCEEDED, and then asserts against
the logged events.
The backend code that runs the submitted job does so in a separate thread. If
the job was successful, the job status was being set to SUCCEEDED *before* the
event logs had been written to the logging backend.
As a result, it was possible for the primary thread running the test to detect
that the job was successful immediately after the secondary thread had updated
the job status but before the secondary thread had written the logs to the
logging backend. Under the right conditions, this could cause the subsequent
logging assertions in the primary thread to fail.
Additionally, the code that collected the logs from the container was using
a "dodgy hack" of time.sleep() and a modulo-based conditional that was
ultimately non-deterministic and could result in log messages being dropped
or duplicated in certain scenarios.
In order to address these issues, this commit does the following:
* Carefully re-orders any code that sets a job status or timestamp
to avoid any obvious race conditions.
* Removes the "dodgy hack" in favor of a much more straightforward
(and less error-prone) method of collecting logs from the container.
* Removes arbitrary and unnecessary calls to time.sleep()
Before applying any changes, the flaky test was failing about 12% of the
time. Putting a sleep() call between setting the `job_status` to SUCCEEDED
and collecting the logs, resulted in a 100% failure rate. Simply moving
the code that sets the job status to SUCCEEDED to the end of the code block,
dropped the failure rate to ~2%. Finally, removing the log collection
hack allowed the test suite to run ~1000 times without a single failure.
Taken in aggregate, these changes make the batch backend more deterministic
and should put the nail in the coffin of this flaky test.
Closes#3475
* Added support for EMR Security Configurations and Kerberos Attributes.
* Revised exception-raising test to work with pytest api.
* Added htmlcov to .gitignore; upgrading botocore to 1.18.17, per commit d29475e.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Add support for empty strings in non-key attributes
https://github.com/spulec/moto/issues/3339
* Nose, not pytest
* Revert "Nose, not pytest"
This reverts commit 5a3cf6c887dd9fafa49096c82cfa3a3b7f91d224.
* PUT is default action
* Fixed issue 3448 for DynamoDB update_item
* Tidied up fix for issue 3448 for DynamoDB update_item
* Reformatted fix for issue 3448 for DynamoDB update_item
* removed use of f-strings in test case as it fails in Travis CI build due to Python 2.7 support of f strings
Replace the special-case code to handle Cloud Formation tags with a more
generic implementation that covers all instance tags.
Supersedes #2863Closes#2862
* Adding some basic endpoints for Amazon Forecast, including all dataset group related endpoints
* Adds better testing around exception handling in forecast endpoint, removes some unused code, and cleans up validation code
* Fix unused imports, optimize imports, code style fixes
Co-authored-by: Paul Miller <pwmiller@amazon.com>
`secretsmanager:DescribeSecret` returns `VersionIdsToStages`
`secretsmanager:ListSecrets` returns the same information in `SecretVersionsToStages`
* Verified fix against real AWS backend.
Fixes#3406
* Implement filtering for stepfunctions:ListExecutions
* Add pagination to Step Functions endpoints
Implements a generalized approach to pagination via a decorator method for the
Step Functions endpoints. Modeled on the real AWS backend behavior, `nextToken`
is a dictionary of pagination information encoded in an opaque string.
With just a bit of metadata hard-coded (`utils.PAGINATION_MODEL`), backend `list`
methods need only be decorated with `@paginate` and ensure that their returned
entities are sorted to get full pagination support without any duplicated code
polluting the model.
Closes#3137
* Add support for metric data query alarms (Metrics=[..])
* Fix trailing whitespace
* Allow for unordered metrics in Python 2.7
* Add describe_alarm assertions and support DatapointsToAlarm
* Notebook Lifecycle Config create, describe and delete
* PR3417 comment changes: raise on create with duplicate name, derive a ValidationException class and use it instead of RESTException, unit test for delete non-existing.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* ApplicationAutoscaling: support autoscaling policies, deregister_scalable_target.
* PR3350 comment changes: drop unnecessary pass statements, unit test three exception cases.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Add missing `Fn::GetAtt` attributes to S3 bucket mock
Addresses an issue reported here https://github.com/localstack/aws-cdk-local/issues/1
* Reformat touched files with `black`
* Reformat touched files with `black` on Python 3.7
The pagination for this endpoint has been modified to more closely
model the real AWS behavior:
* Log Groups are now sorted alphabetically by `logGroupName`.
* `nextToken` is now a string containing the last `logGroupName` in the
current response.
* Specifying an invalid `nextToken` does not generate an error, but does
return an empty group list.
* `nextToken` is not included in the response if there are no additional
items to return.
Fixes#3395
* Properly coerce `Encrypted` attribute to bool on request/response.
* Create and use a default AWS managed CMK for EBS when clients request
an encrypted volume without specifying a KmsKeyId.
NOTE: A client-provided KmsKeyId is simply stored as-is, and is not validated
against the KMS backend. This is in keeping with other moto backends (RDS, Redshift)
that currently also accept unvalidated customer master key (CMK) parameters, but could
be an area for future improvement.
Closes#3248
* Enable more unit tests for ACM
* put a smile on travis-ci face
Co-authored-by: nom3ad <nom3ad@gmail.com>
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* ACM: support `tags` parameter in import_certificate()
* ACM: support tags parameter in request_certificate()
* ACM: better tag operations with more unit tests
Co-authored-by: nom3ad <nom3ad@gmail.com>
The `tag_filter` method has been re-arranged to mimic the actual AWS behavior:
Return `True` if *any* tag matches a filter and *all* filters are matched.
Python's closures are late-binding, so we have to modify the lambdas accordingly!
Closes#2814
When comparing old and new values when doing a nested item update, the
`!=` implementation fails when the value being compared is `None`. This
results in an exception when trying to insert a new item into a nested
map. So just do a quick check that the original value is exists before
doing the comparison, as the `None` default is what is tripping this.
* SecretsManager - handle missing secrets versions
The get_secret_value method should raise ResourceNotFoundException
if a secret exists but the provided VersionId does not.
* Run black
* 2.x support
* black fix?
* secret is not a dict. Fix error msg output.
* Transcribe first cut: Medical: start, get and delete jobs.
* Added list_medical_transcription_job() support to Transcribe.
* Support for medical vocabularies.
* Added transcribe to list of backends to fix server mode error.
* PR3299 requested changes: don't offer deprecated decorator, regionalize download_uri, create/use service-specific exceptions.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Start working on flow logs
* Change test
* Constructing tests
* Changing exceptions and adding more tests
* Adding more tests
* Changing model and adding more tests
* Adding support for tags
* Mocking Access error with non-existing Log Group Name
* Adding FlowLogAlreadyExists support
* Changing style
* Reformatted code
* Reformatted tests
* Removing needless test
* Adding support for CloudFormation
* Reformatting slightly
* Removing arnparse and using split
* Rearranging tests
* Fixing FilterNotImplementedError test
* Moving imports to 'if' clauses and adding explicit test for 'cloud-watch-logs' type
* Setting names matching boto3 API and restoring 'not-implementd-filter' test
* Reformatting tests with black
* added organizations detach_policy response, model, and tests, issue #3239
Signed-off-by: Ben <ben.lipscomb@fmr.com>
* Created individual tests for detach_policy exceptions, updated regex statements for Root, OU, and Account Id
* Make ElasticAddress a tagged resource
To be able to filter on tags on ElasticAddresses, I need to have tags.
* remove unneeded commented lines
Was beginning of how to to it before further checking how it is done with other resources.
* do not ignore network-interface-owner-id filter
* add TODO about currently hardcoded region
* remove hardcoding region
* add testing for tags
creating and allocation, adding tags and querying for it
* separate test for tags into own method
* Linting
Co-authored-by: Bert Blommers <info@bertblommers.nl>
Support for cluster_id parameter substitution in autoscaling policy cloudwatch alarm dimensions.
New operations put_autoscaling_policy and remove_autoscaling_policy support
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* feature added: support for api PutUserPermissionsBoundary; DeleteRolePermissionsBoundary
* minor test fix
* lint fixed
* refractored test case
* Issue 3224 s3 copy glacier object (#3318)
* 3224 Enhancement - S3 Copy restored glacier objects
- adds setter for expiry date
- copy sets expiry date to none when source is glacier object
- throws error for copying glacier object only if not restored/still restoring
* 3224 Enhancement - S3 Copy restored glacier objects
- throws error for copying deep archive object only if not restored/still restoring
* Fix:s3 List Object response:delimiter (#3254)
* Fix:s3 List Object delimiter in response
* fixed tests
* fixed failed tests
Co-authored-by: usmankb <usman@krazybee.com>
* feature added: support for api PutUserPermissionsBoundary; DeleteRolePermissionsBoundary
* minor test fix
* lint fixed
* refractored test case
* added test case for put role exception
Co-authored-by: ruthbovell <63656505+ruthbovell@users.noreply.github.com>
Co-authored-by: usmangani1 <sgosman_chem@yahoo.com>
Co-authored-by: usmankb <usman@krazybee.com>
* Make IoT certificate id generation deterministic
Fixes#3321
As per https://stackoverflow.com/questions/55847788/how-does-aws-iot-generate-a-certificate-id,
the IoT certificate ID is the SHA256 fingerprint of the
certificate. Since moto doesn't generate full certificates we will
instead use the SHA256 hash of the passed certificate pem.
* Don't allow duplicate IoT certificates to be created
Fixes#3320
When using boto3, trying to register a certificate that already
exists will throw a ResourceAlreadyExistsException. Moto should
follow the same pattern to allow testing error handling code in
this area.
* 3224 Enhancement - S3 Copy restored glacier objects
- adds setter for expiry date
- copy sets expiry date to none when source is glacier object
- throws error for copying glacier object only if not restored/still restoring
* 3224 Enhancement - S3 Copy restored glacier objects
- throws error for copying deep archive object only if not restored/still restoring
* Change to test_s3 method test_presigned_url_restrict_parameters to tolerate change in exception messages, spurred by boto3 1.14.59 release.
* ApplicationAutoscaling: support for all the current various forms of resource_id.
* Factored logic for extracting application autoscaling resource_type from resource_id to separate function, per PR3304 comment.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* kinesisvideo create_stream
* add kinesis video stream description
* add kinesisvideo describe_stream
* add kinesisvideo list_streams
* add kinesisvideo delete_stream
* remove unused comment
* remove duplicated definition
* add kinesis video exceptions
* pass region_name to kinesisvideo client in test
* fix kinesisvideo url path
* resolve conflict of kinesisvideo url and kinesis url
* specify region name to kinesisvideobackend
* Add get-dataendpoint to kinesisvideo
* include stream name in ResourceInUseException of kinesisvideo
* use ACCOUNT_ID from moto.core in kinesisvideo
* add server test for kinesisvideo
* split up kinesisvideo test
* added cognito-idp initiate_auth and PASSWORD_VERIFIER challenge to respond_to_auth_challenge
* fixed for python2
* added mfa, REFRESH_TOKEN to initiate_auth, SOFTWARE_TOKEN_MFA to respond_to_auth_challenge
* added negative tests
* test
* Fix: Api-Gateway ApiKeyAlreadyExists headers change.
* Added test for non decorator
* Fixed cli errors
* Fix:fixed build errors
* Fix: assert only in case of non server mode
Co-authored-by: usmankb <usman@krazybee.com>
* IAM User Cloudformation Enhancements: update, delete, getatt.
* AWS::IAM::Policy Support
* Added unit tests for AWS:IAM:Policy for roles and groups. Fixed bug related to groups.
* AWS:IAM:AccessKey CloudFormation support.
* Refactor of CloudFormation parsing.py methods to simplify and standardize how they call to the models. Adjusted some models accordingly.
* Further model CloudFormation support changes to align with revised CloudFormation logic. Mostly avoidance of getting resoure name from properties.
* Support for Kinesis Stream RetentionPeriodHours param.
* Kinesis Stream Cloudformation Tag Support.
* Added omitted 'region' param to boto3.client() calls in new tests.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Add check for existing execution, fix issue with make init
* Remove f-string usage
* Remove fstring usage in test
* Pin black and run formatting on test_stepfunction
* Reverse changes made by black 20.8b1
* Add support for Description in egress rule response
* Update SecurityGroup default egress rule ip range
* Remove extra commas
* Remove extra commas
* Lower docker package in Travis
* Add more lambda vars per PR 3247
* Remove code added in 3247
* Add tests for egress rules with Descriptions
* Reformat based on black
Co-authored-by: spillin <jmbollard@me.com>
* lambda-responses: add method to dispatch concurrency calls
* lambda-resources: add route to handle concurrency requests
* lambda-model: implement put_function_concurrency and concurrency attribute
* put-concurrency-tests: add one simple test
* get_function: add concurrency entry - with test
* lambda-reserved-concurrency: cloudformation support
* lambda-concurrency: implement delete_reserved with tests
* lambda-concurrency: implement get_reserved with tests
* lint
* implementation-cov: mark delete_function_concurrency, put_function_concurrency and get_function_concurrency
* botocore doesn't display concurrency entry for lambdas without it
* lambda(refactor): improvements on response's handler
* Support for CloudFormation stack resource deletion via backend resource method
delete_from_cloudformation_json() via parse_and_delete_resource().
* Correction to the inappropriate inclusion of EndingSequenceNumber in open shards.
This attribute should only appear in closed shards. This regretfully prevents
confirmation of consistent record counts after split/merge in unit tests.
* Added parameters/decorator to CloudFormationModel method declarations to calm-down Pycharm.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* First cut of S3 Cloudformation Update support: encryption property.
* Update type support for S3. Abstract base class for CloudFormation-aware models, as designed by @bblommers, introduced to decentralize CloudFormation resource and name property values to model objects.
* Blackened...
* Un-renamed param in s3.models.update_from_cloudformation_json() and its call to stay compatible with other modules.
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* implement register_image
* format code
* add user_ids to snapshot model
* implement register_image
* format code
* add user_ids to snapshot model
* trying to un-deprecate tests
* Write tests and finalize implementation
* Add region parameter to boto3 resource call
* fixed test error
* Feature: Support --filters opton in secretsmanager:ListSecrets
* Implement some of the secret filters
* Check listSecrets filters combine with an implicit AND operator
* Test all filter and multi-value filter and multi-word filter
* Fix matcher behavior, restructure code
* Implement remaining listSecrets filter cases
* Linter fixes
* Use contains-in-any-order assertions for test_list_secrets
* Linter fix again
* Attempt Python 2 fix for assert_items_equal
* Remove docstrings from test_list_secrets tests as they make the test reports weird
* Test and handle listSecrets filter with no values
* Add test for describe_instance_types
It currently fails due to an invalid XML schema
* Add more detail to test
* Fix the XML schema for describe_instance_types
* fix sqs tag list from cloudformation resource creation
the method `create_from_cloudformation_json` of the Sqs resource
does not handle the difference of format of the Tags field in the
resource template and the format expected in Sqs resource class.
In cfn resource template Tags is specified as a list of dicts. But
the Sqs resource expects that the tags field be a single dict.
This behaviour causes a crash when a queue is created with tags
from `create_from_cloudformation_json` and later the list_queue_tags
is called because it tries to call `items` from `queue.tags` but
tags is actually a list of dicts.
* fix comment
* fix linter
* minor
Co-authored-by: Hudo Assenco <hudo.assenco@gmail.com>
* Fixed defect with CloudFormation provisioning of S3 buckets occuring when stack has a long name, resulting in the default S3 bucket name's length exceeding its 63 char limit.
* PR 3169 July 23, 2020 2:57a ET comment: added additional asserts to assure provisioned bucket's name complies. Fixed bug in my earlier change that could produce default bucket names with illegal upper-case characters in it.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Enforce parameter naming
Parameters are not allowed to start with `ssm` or `aws`. This commit adds error messages which
correspond exactly to the error messages returned by boto3.
* Fix for Python 2 compatibility
f-strings not supported in Python 2.7
* Fix : SQS - Added support for attribute labels for send_message function
* Add integration test on receive message function
* Add send message invalid datetype integration test and fix SQS MessageAttributesInvalid exceptions
* Implement Filter: Contains functionality for describe_params
This commit adds the Contains functionality. Tests were created
to mimic behavior in AWS/boto3, including that filters with values
in the form of `/name` will match parameters named `/name/match` but
not parameters named `match/with/other-name`. In the test example, a
Contains filter with the value `/tan` would match: `/tangent-3` and
`tangram-4` but not `standby-5`.
* Enforce parameter filter restrictions on get_parameters_by_path
According to the boto3 documentation [1], `Name`, `Path`, and `Tier` are not
allowed values for `Key` in a parameter filter for `get_parameters_by_path`.
This commit enforces this by calling `_validate_parameter_filters` from the
`get_parameters_by_path` method, and adding a check to `_validate_parameter_filters`.
I added 3 test cases to `test_get_parameters_by_path` which check for the correct
exception when calling with a parameter filter using any of these keys.
* Code formatted to match style
* Refactored logic
* FIX:IAM-Added support for pathPrefix in list_users_function
* removed changes for roles
* Added test for non decorator
* changed filter function
Co-authored-by: usmankb <usman@krazybee.com>
* Adding attchment.vpc-id, attachment.state, type, and vpn-gateway-id filters for VPN Gateways. fixes#3154
* Run formatting on tests
Co-authored-by: Alan Baldwin <alan.baldwin.jr@gmail.com>
* initial implementation of taskSets. Fixed a bug with ECS Service where task_definition was a required parameter.
* Added update_task_set and tests. DRYed up ClusterNotFoundException. General cleanup.
* Added support for filtering tags on include parameter to describe_task_sets. Added additional tests.
* Fix copy/pasta in ClusterNotFoundException
* styling updates
* Added TODO for delete_task_set force parameter
* Updated multiple function and constructor calls to use named variables. Updated tests to reference variables instead of hardcoded strings.
* Run black for formatting
* Updated create_service function call to use named variables
* First failing test, and enough framework to run it.
* Rudimentary passing test.
* Sagemaker Notebook Support, take-1: create, describe, start, stop, delete.
* Added list_tags.
* Merged in model support from https://github.com/porthunt/moto/tree/sagemaker-support.
* Re-org'd
* Fixed up describe_model exception when no matching model.
* Segregated tests by Sagemaker entity. Model arn check by regex..
* Python2 compabitility changes.
* Added sagemaker to list of known backends. Corrected urls.
* Added sagemaker special case to moto.server.infer_service_region_host due to irregular url format (use of 'api' subdomain) to support server mode.
* Changes for PR 3105 comments of July 10, 2020
* PR3105 July 10, 2020, 8:55 AM EDT comment: dropped unnecessary re-addition of arn when formulating model list response.
* PR 3105 July 15, 2020 9:10 AM EDT Comment: clean-up SageMakerModelBackend.describe_models logic for finding the model in the dict.
* Optimized imports
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Make batch.utils.lowercase_first_key() recursive
* Reformat using black
* Add test of recursive lowercase_first_key()
* Fix typo in ttest_batch/test_cloud_formation.py
* Placeholder to test Application Autoscaling.
* Wire everything together and create a first passing test without any real functionality.
* Get one test working properly.
* Add some TODO items.
* Reformat code with black
* Second passing test for describe_scalable_targets.
* New test for NextToken.
* Add some tests for ParamValidationError and ValidationException.
* black
* Ensure scalable targets are being captured in an OrderedDict() for deterministic return later.
* Add validation to describe_scalable_targets and register_scalable_target.
* Fix tests.
* Add creation_time, refactor, add ECS backend, and add failing test for checking that ecs service exists.
* Add parameter validation.
* Improved documentation for CONTRIBUTING.md
Adds some details to give people an idea what's involved in adding new features/services
* Integrate with ECS.
* black
* Refactor to allow implementation of SuspendedState.
* Complete support for SuspendedState.
* Bump up implementation coverage percentage.
* Tidy up code; add comments.
* Implement suggested changes from code review.
* Minor refactorings for elegance.
* README update
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
This change fixes:
* Replace call to non-existent exception yaml.ParserError
* Catches yaml scanner error for valid json with tabs
* Supply yaml loader to ensure yaml loading throws exception validly for json with tabs and doesn't try to load the json incorrectly
* Several updates to the contributor documentation with extra information.
* Fix failing test by providing a region.
* Create test for issue 2210.
* Check if CreateBucketConfiguration is supplied and empty; raise MalformedXML error if so.
* change line position for uuid and last_modified because they're not input parameters
* add event_source_arn validator and setter
* refactor batch_size as setter
* add helper function to parse arn and return source service
* fix for EventSource's create_from_cfn, there was no reference in the lambda object for the esm if created by cfn
* add esm deletion by cloudformation
* remove unused variable in test
* add cfn's update
* add complete implementation of delete_from_cfn
* blacked changed files
* fix test with invalid batchsize for sqs
* Dynamodb2 Table - Bugfix for localindex and implemented get_cfn_attributes
* Dynamodb2 eventsource - fix test to use StreamArn attribute
* Lambda Test - fix test_update_event_source_mapping
* Correct group inline policy rendering in iam:GetAccountAuthorizationDetails response
* Include user inline policy if exists
* Add tests for IAM inline policies
* Remove unnecessary print stmts
* Fix: SecretsManager - Added missing pop() override to get_secret_name_from_arn (#3056)
* Added test case for delete_secret_force_with_arn (#3057)
* Fixed lint for test_delete_secret_force_with_arn (#3057)
* Enhancement : API-Gateway Put Integration Response - Adding support for contentHandling.
* Added tests where the contentHandling is None also gets tested.
* Linting
Co-authored-by: usmankb <usman@krazybee.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* add EventBus to model's map
* add support for creation of EventBus through cloudformation's api
* add cloudformation's delete
* add cloudformation's update
* add cloudformation's attribute
* CloudWwatch delete Alarm status code handling on invalid alarm Name
* Handled cases where a mix of existent and non existent alarms are tried to delete
* Linting
Co-authored-by: usmankb <usman@krazybee.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* Implemented IAM delete_instance_profile
* PR adjustment: positively verifying instance profile deletion in test case.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* implemented s3 default encryption methods
* PR adjustments: moved logic for retrieving bucket's encrypted status to the backend.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Corrected bug in IAM delete_role() due to overloading of name 'role' in function
* PR-requested fixes: added region to tests boto client create, reformatted with black
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* ENH: Add unit test for propagation tags
* BUG: Add missing translation of boolean PropagateAtLaunch tag values to strings
* BUG: Should really be checking for "true" and not True
* CLN: Black formatting
* Keep order in request body to ensure auth signing works.
* Lint.
* More OrderedDict to ensure data parameter order.
* Lint.
* Improve CF test assertions.
* Fix syntax error.
* Cleanup CF test.
* ENH: Add unit test for cloudformation DependsOn
* ENH: Add implementation of retrieving list of resources that account for dependencies
* ENH: Update the name mappings so that they are consistent with the latest cloudformation names
* ENH: Add launch configuration to type names
* ENH: Create subnet for test and test creation with dependencies
* CLN: Code reformatting
* CLN: Remove print statements
* BUG: Fix error resulting in possible infinite loop
* CLN: Remove commented out fixture decorator
* BUG: Remove subnet creation
* CLN: Remove main and ec2 dependencies
* BUG: Add back in instance profile name type
* CLN: Remove print
* BUG: Fix broken unit test
* CLN: Code reformatting
* CLN: Remove main
* ENH: Add autoscaling group name to type names
* ENH: Add unit test for string only dependency and add assertions to unit tests
* ENH: Add unit test for chained depends_on in cloudformation stack
* BUG: Remove f strings for python 2.7 compatibility
* BUG: List needs to be sorted for python2.7
* CLN: Fix code formatting
* Added some member and proposal functions
* Added additional member and proposal functions
* Fixed admin password return and added update_member along with tests
* Added network removal and member removal proposal
* Fixed failing test
* Fixed Python 2.7 test
When using the ADD syntax to sum up different components
the path that is provided is allowed to be non-existent.
In such a case DynamoDB will initialize it depending on
the type of the value.
If it is a number it will be initialized with 0.
If it is a set it will be initialized with an empty set.
When an instance is running, OpsWorks reports its status as "online"
[1], while EC2 reports it as "running". Until now, moto copied the EC2
instance's status as is. This commit introduces the converts the running
status to online when returned by OpsWorks.
[1]: https://docs.aws.amazon.com/cli/latest/reference/opsworks/describe-instances.html
Part of structured approach for UpdateExpressions:
1) Expression gets parsed into a tokenlist (tokenized)
2) Tokenlist get transformed to expression tree (AST)
3) The AST gets validated (full semantic correctness)
4) AST gets processed to perform the update -> this commit
This commit uses the AST to execute the UpdateExpression.
All the existing tests pass. The only tests that have been
updated are in test_dynamodb_table_with_range_key.py because
they wrongly allow adding a set to a path that doesn't exist.
This has been alligend to correspond to the behavior of AWS
DynamoDB.
This commit will resolve https://github.com/spulec/moto/issues/2806
Multiple tests have been implemented that verify this.
This commit puts AST validation on the execution path. This means updates get
validated prior to being executed. There were quite a few tests that were not
working against Amazon DDB. These tests I considered broken and as such this
commit adapts them such that they pass against Amazon DDB.
test_update_item_on_map()
=> One of the SET actions would try to set a nested element by specifying the nesting on the path
rather than by putting a map as a value for a non-existent key. This got changed.
test_item_size_is_under_400KB
=> Used the keyword "item" which DDB doesn't like. Change to cont in order to keep the same sizings.
=> Secondly the size error messages differs a bit depending whether it is part of the update or part
of a put_item. For an update it should be:
Item size to update has exceeded the maximum allowed size
otherwise it is
Item size has exceeded the maximum allowed size'
test_remove_top_level_attribute
=> Used a keyword item. Use ExpressionAttributeNames
test_update_item_double_nested_remove
=> Used keywords name & first. Migrated to non-deprecated API and use ExpressionAttributeNames
test_update_item_set &
test_boto3_update_item_conditions_pass &
test_boto3_update_item_conditions_pass_because_expect_not_exists &
test_boto3_update_item_conditions_pass_because_expect_not_exists_by_compare_to_null &
test_boto3_update_item_conditions_pass_because_expect_exists_by_compare_to_not_null &
test_boto3_update_item_conditions_fail &
test_boto3_update_item_conditions_fail_because_expect_not_exists &
test_boto3_update_item_conditions_fail_because_expect_not_exists_by_compare_to_null
=> Were broken tests which had string literal instead of value placeholder
Part of structured approach for UpdateExpressions:
1) Expression gets parsed into a tokenlist (tokenized)
2) Tokenlist get transformed to expression tree (AST)
3) The AST gets validated (full semantic correctness) -> this commit
4) AST gets processed to perform the update
This commit uses the AST to perform validation. Validation makes sure the
nodes encounterd have valid values and they will also resolve values for
references that refer to item state or values passed into the expression.
Part of structured approach for UpdateExpressions:
1) Expression gets parsed into a tokenlist (tokenized)
2) Tokenlist get transformed to expression tree (AST) -> This commit
3) The AST gets validated (full semantic correctness)
4) AST gets processed to perform the update
This commit uses the tokenlist to build an expression tree. This tree is not
yet used. Still it allows to raise additional Validation Exceptions which
previously were missed silently therefore it allows tests to catch these type of
ValidationException. For that reason DDB UpdateExpressions will be parsed
already. It also makes sure we won't break existing tests.
One of the existing tests had to be changed in order to still pass:
- test_dynamodb_table_with_range_key.test_update_item_with_expression
This test passed in a numeric literal which is not supported by DynamoDB
and with the current tokenization it would get the same error as in AWS
DynamoDB.
Currently the mock for DynamoDB has adhoc code to implement
its updateExpression functionality. This series will
transform the logic such that Update Expressions are processed
as follows:
1) Expression gets parsed into a tokenlist (tokenized) -> This commit
2) Tokenlist get transformed to expression tree (AST)
3) The AST gets validated (full semantic correctness)
4) AST gets processed to perform the update
This alows for a more realistic mocking. It will throw exceptions much
more aggressively avoiding situations where a test passes against the
mock but fails with an exception when running against AWS.
Introduction of step 3 also allows to have the update expression as an
atomic unit of work. So updates at the start of the expression cannot
be performed if there is an error further down the expression.
This specific commit will tokenize expressions but the tokenlist is not
yet used. It is purely to keep clear boundaries. It does do a minor
refactoring of the exceptions to allow more re-use and to ease testing.
This series of changes is to aid providing a long-term solution for
https://github.com/spulec/moto/issues/2806.
The support in this patch is preliminary and may or may not be feature complete.
It provides the basic support for update_secret so that future work can build
on it as needed.
Currently, the delete_parameter function for the ssm client will respond with a dict containing a key of Invalid Parameter which has a value of a list containing the parameter name that was requested to be deleted when a parameter with said name doesn't exist which doesn't match the behavior of boto3.
To match the SWF documentation, an empty task is one where the
task-token is the empty string, rather than being a nonexistant key
Signed-off-by: Laurie O <laurie_opperman@hotmail.com>
Include 'closeStatus' and 'closeTimestamp' when describing SWF workflow
execution using 'describe_workflow_execution'
Signed-off-by: Laurie O <laurie_opperman@hotmail.com>
If an invitation is attempted to be reset to a pool we validate that the user is indeed already in the pool else we raise a UserNotFoundException to match AWS behaviour
According to the AWS documentation:
```
The HTTP status code will be in the 200 range for successful request.
For the RequestResponse invocation type this status code will be 200.
For the Event invocation type this status code will be 202.
For the DryRun invocation type the status code will be 204.
```
AWS defines this option as:
```
--family-prefix (string)
The full family name with which to filter the ListTaskDefinitions
results. Specifying a familyPrefix limits the listed task
defini-tions to task definition revisions that belong to that
family.
```
This option behaves differently than ecs:ListTaskDefinitionFamilies.
Instead of doing a comparison like `startswith`, it does a full string
comparison by matching the entire task definition family to the prefix.
For example, let's say there exists a task definition with the family
`super-cool-task-def`.
ListTaskDefinitionFamilies would look like this:
```
aws ecs list-task-definition-families --family-prefix super-cool
{
"families": [
"super-cool-task-def"
]
}
```
ListTaskDefinitions would look like this:
```
aws ecs list-task-definitions --family-prefix super-cool
{
"taskDefinitionArns": []
}
```
