* Properly coerce `privateDnsEnabled` to boolean value when parsing requests.
* Per AWS spec, default `privateDnsEnabled` request value to `True`.
* Properly serialize `privateDnsEnabled` as boolean value in responses.
* Add test coverage.
Ref: #3540
Applies the user credentials pattern from the ADMIN_NO_SRP_AUTH flow
to the ADMIN_USER_PASSWORD_AUTH auth flow for Cognito admin_initiate_auth
requests.
Co-authored-by: Robin Wilkins <r.wilkins@waracle.com>
* fix https://github.com/localstack/localstack/issues/3339
* fixe lint issues
* Fix review comments
- move deduplication time to constants
- make tests parameterized
- update tests as per review comments
* change variable name expectedCount => expected_count
* fix tests for python 2.7
increase deduplication mock config to account for delays
* ignore time mocking test in server mode
These tests were passing with TZ=UTC, but under a non-UTC timezone
they were failing:
E AssertionError: given
E X = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzutc()), …}, …]
E and
E Y = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzlocal()), …}, …]
E X[0]['timestamp'] != Y[0]['timestamp']
With this fix, they pass either way.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
* Pass the "default" cluster
* Mock ECS exceptions more accurately
Moto's mock ECS has drifted fairly far from the actual ECS API in terms
of which exceptions it throws. This change begins to bring mock ECS's
exceptions in line with actual ECS exceptions. Most notably:
- Several custom exceptions have been replaced with their real ECS
exception. For example, "{0} is not a cluster" has been replaced with
ClusterNotFoundException
- Tests have been added to verify (most of) these exceptions work
correctly. The test coverage was a little spotty to begin with.
- The new exceptions plus the change to pass the "default" cluster
exposed a lot of places where mock ECS was behaving incorrectly. For
example, the ListTasks action is always scoped to a single cluster in
ECS but it listed tasks for all clusters in the mock. I've minimally
updated the tests to make them pass, but there's lots of opportunity to
refactor both this method's test and its implementation.
This does not provide full coverage of exceptions. In general, I ran
these operations against actual ECS resources and cross-referenced the
documentation to figure out what actual exceptions should be thrown and
what the messages should be. Consequently, I didn't update any
exceptions that took more than trivial amount of time to reproduce with
real resources.
* Retrieve SAML Attribute by Name instead of relying on order which is too fragile
* Handle case when SAML Attribute SessionDuration is not provided, as it is not a required attribute from SAML response
When session duration not provided, AWS consider by default a duration of one hour as cited in the following documentation:
"If this attribute is not present, then the credential last for one hour (the default value of the DurationSeconds parameter of the AssumeRoleWithSAML API)."
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml_assertions.html#saml_role-session-duration
Traceback was:
[...]
File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/responses.py", line 79, in assume_role_with_saml
role = sts_backend.assume_role_with_saml(
File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/models.py", line 99, in assume_role_with_saml
role = AssumedRole(**kwargs)
TypeError: __init__() missing 1 required positional argument: 'duration'
* Process saml xml namespaces properly instead of relying on textual prefix that can vary between identity providers
* Handle when SAML response AttributeValue xml tag contains attributes that force xmltodict to build a dictionary as for complex types instead of directly returning string value
Leverage force_cdata option of xmltodict parser that always return a complex dictionary even if xml tag contains only text and no attributes.
* Improve existing test_assume_role_with_saml to be coherent with other assume_role_with_saml tests and remove dead code at the same time
By definition, `single-node` clusters can only consist of 1 node. Likewise,
`multi-node` clusters must have 2 or more nodes.
* Ensure `ClusterType` parameter is either `multi-node` or `single-node`.
* Ensure proper validation of `NumberOfNodes` parameter based on `ClusterType`
parameter.
* Fix existing test case that incorrectly allowed a `multi-node` cluster to
consist of 1 node.
* Add dedicated test for resizing a cluster from `single-node` to `multi-node`
and back again.
Behavior and error messages have been verified against a real AWS backend.
* Add format command to makefile
* Refactor executions to be a attribute of StateMachine
* Begin to add tests for execution history
* Add tests for failed and successful event histories, with implementations
* Add failure case to environment var check
* Skip test if in server mode and update implementation coverage
* Add conditional import for mock to cover python 2
* Refactor stop execution logic into StateMachine
* Refactor event history environment variable into settings.py
* Remove typing and os import
The `boto` library (long ago superseded by `boto3`) has not had an official
release in over two years or even a commit in the last 18 months. Importing
the package (or indirectly importing it by via `moto`) generates a deprecation
warning. Additionally, an ever-increasing number of `moto` users who have
left `boto` behind for `boto3` are still being forced to install `boto`.
This commit vendors a very small subset of the `boto` library--only the code
required by `moto` to run--into the /packages subdirectory. A README file
is included explaining the origin of the files and a recommendation for how
they can be removed entirely in a future release.
NOTE: Users of `boto` will still be able to use `moto` after this is merged.
closes#2978closes#3013closes#3170closes#3418
relates to #2950
A test added in #2401 copied the name of an existing test, preventing it
from being run. This commit renames the second test, allowing both to
be picked up by the test runner.
Original code was trying to raise a ClientError directly. Change to
appropriate Redshift exception class.
* Fix test assertion for `boto`.
* Add test coverage for `boto3`.
Previous code would raise `TypeError: 'dict_keys' object is not subscriptable`
when run under Python 3.
* Re-write code in Python 2/3 compatible way.
* Add clarifying comment.
* Add test coverage.
Supersedes #3227
The previous code was trying to raise a botocore ClientError directly, which
was actually generating a secondary AttributeError because the arguments passed
to ClientError() were incorrect.
This replaces the ClientError() call with a proper moto exception class for
Redshift and fixes the test assertions appropriately.
Supersedes #1957
* Fix:CloudWatch List metrics with dimensions
* Fix:CloudWatch List metrics with dimensions
* Fixed new cases and added more tests
Co-authored-by: usmankb <usman@krazybee.com>
* Extract Duplicate Code into Helper Method
DRY up the tests and replace the arbitrary `sleep()` calls with a more
explicit check before progressing.
* Improve Testing of batch:TerminateJob
The test now confirms that the job was terminated by sandwiching a `sleep`
command between two `echo` commands. In addition to the original checks
of the terminated job status/reason, the test now asserts that only the
first echo command succeeded, confirming that the job was indeed terminated
while in progress.
* Fix Race Condition in batch:SubmitJob
The `test_submit_job` in `test_batch.py` kicks off a job, calls `describe_jobs`
in a loop until the job status returned is SUCCEEDED, and then asserts against
the logged events.
The backend code that runs the submitted job does so in a separate thread. If
the job was successful, the job status was being set to SUCCEEDED *before* the
event logs had been written to the logging backend.
As a result, it was possible for the primary thread running the test to detect
that the job was successful immediately after the secondary thread had updated
the job status but before the secondary thread had written the logs to the
logging backend. Under the right conditions, this could cause the subsequent
logging assertions in the primary thread to fail.
Additionally, the code that collected the logs from the container was using
a "dodgy hack" of time.sleep() and a modulo-based conditional that was
ultimately non-deterministic and could result in log messages being dropped
or duplicated in certain scenarios.
In order to address these issues, this commit does the following:
* Carefully re-orders any code that sets a job status or timestamp
to avoid any obvious race conditions.
* Removes the "dodgy hack" in favor of a much more straightforward
(and less error-prone) method of collecting logs from the container.
* Removes arbitrary and unnecessary calls to time.sleep()
Before applying any changes, the flaky test was failing about 12% of the
time. Putting a sleep() call between setting the `job_status` to SUCCEEDED
and collecting the logs, resulted in a 100% failure rate. Simply moving
the code that sets the job status to SUCCEEDED to the end of the code block,
dropped the failure rate to ~2%. Finally, removing the log collection
hack allowed the test suite to run ~1000 times without a single failure.
Taken in aggregate, these changes make the batch backend more deterministic
and should put the nail in the coffin of this flaky test.
Closes#3475
* Added support for EMR Security Configurations and Kerberos Attributes.
* Revised exception-raising test to work with pytest api.
* Added htmlcov to .gitignore; upgrading botocore to 1.18.17, per commit d29475e.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Add support for empty strings in non-key attributes
https://github.com/spulec/moto/issues/3339
* Nose, not pytest
* Revert "Nose, not pytest"
This reverts commit 5a3cf6c887dd9fafa49096c82cfa3a3b7f91d224.
* PUT is default action
* Fixed issue 3448 for DynamoDB update_item
* Tidied up fix for issue 3448 for DynamoDB update_item
* Reformatted fix for issue 3448 for DynamoDB update_item
* removed use of f-strings in test case as it fails in Travis CI build due to Python 2.7 support of f strings
Replace the special-case code to handle Cloud Formation tags with a more
generic implementation that covers all instance tags.
Supersedes #2863Closes#2862
* Adding some basic endpoints for Amazon Forecast, including all dataset group related endpoints
* Adds better testing around exception handling in forecast endpoint, removes some unused code, and cleans up validation code
* Fix unused imports, optimize imports, code style fixes
Co-authored-by: Paul Miller <pwmiller@amazon.com>
`secretsmanager:DescribeSecret` returns `VersionIdsToStages`
`secretsmanager:ListSecrets` returns the same information in `SecretVersionsToStages`
* Verified fix against real AWS backend.
Fixes#3406
* Implement filtering for stepfunctions:ListExecutions
* Add pagination to Step Functions endpoints
Implements a generalized approach to pagination via a decorator method for the
Step Functions endpoints. Modeled on the real AWS backend behavior, `nextToken`
is a dictionary of pagination information encoded in an opaque string.
With just a bit of metadata hard-coded (`utils.PAGINATION_MODEL`), backend `list`
methods need only be decorated with `@paginate` and ensure that their returned
entities are sorted to get full pagination support without any duplicated code
polluting the model.
Closes#3137
* Add support for metric data query alarms (Metrics=[..])
* Fix trailing whitespace
* Allow for unordered metrics in Python 2.7
* Add describe_alarm assertions and support DatapointsToAlarm
* Notebook Lifecycle Config create, describe and delete
* PR3417 comment changes: raise on create with duplicate name, derive a ValidationException class and use it instead of RESTException, unit test for delete non-existing.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* ApplicationAutoscaling: support autoscaling policies, deregister_scalable_target.
* PR3350 comment changes: drop unnecessary pass statements, unit test three exception cases.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Add missing `Fn::GetAtt` attributes to S3 bucket mock
Addresses an issue reported here https://github.com/localstack/aws-cdk-local/issues/1
* Reformat touched files with `black`
* Reformat touched files with `black` on Python 3.7
The pagination for this endpoint has been modified to more closely
model the real AWS behavior:
* Log Groups are now sorted alphabetically by `logGroupName`.
* `nextToken` is now a string containing the last `logGroupName` in the
current response.
* Specifying an invalid `nextToken` does not generate an error, but does
return an empty group list.
* `nextToken` is not included in the response if there are no additional
items to return.
Fixes#3395
* Properly coerce `Encrypted` attribute to bool on request/response.
* Create and use a default AWS managed CMK for EBS when clients request
an encrypted volume without specifying a KmsKeyId.
NOTE: A client-provided KmsKeyId is simply stored as-is, and is not validated
against the KMS backend. This is in keeping with other moto backends (RDS, Redshift)
that currently also accept unvalidated customer master key (CMK) parameters, but could
be an area for future improvement.
Closes#3248
* Enable more unit tests for ACM
* put a smile on travis-ci face
Co-authored-by: nom3ad <nom3ad@gmail.com>
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* ACM: support `tags` parameter in import_certificate()
* ACM: support tags parameter in request_certificate()
* ACM: better tag operations with more unit tests
Co-authored-by: nom3ad <nom3ad@gmail.com>
The `tag_filter` method has been re-arranged to mimic the actual AWS behavior:
Return `True` if *any* tag matches a filter and *all* filters are matched.
Python's closures are late-binding, so we have to modify the lambdas accordingly!
Closes#2814
When comparing old and new values when doing a nested item update, the
`!=` implementation fails when the value being compared is `None`. This
results in an exception when trying to insert a new item into a nested
map. So just do a quick check that the original value is exists before
doing the comparison, as the `None` default is what is tripping this.
* SecretsManager - handle missing secrets versions
The get_secret_value method should raise ResourceNotFoundException
if a secret exists but the provided VersionId does not.
* Run black
* 2.x support
* black fix?
* secret is not a dict. Fix error msg output.
* Transcribe first cut: Medical: start, get and delete jobs.
* Added list_medical_transcription_job() support to Transcribe.
* Support for medical vocabularies.
* Added transcribe to list of backends to fix server mode error.
* PR3299 requested changes: don't offer deprecated decorator, regionalize download_uri, create/use service-specific exceptions.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Start working on flow logs
* Change test
* Constructing tests
* Changing exceptions and adding more tests
* Adding more tests
* Changing model and adding more tests
* Adding support for tags
* Mocking Access error with non-existing Log Group Name
* Adding FlowLogAlreadyExists support
* Changing style
* Reformatted code
* Reformatted tests
* Removing needless test
* Adding support for CloudFormation
* Reformatting slightly
* Removing arnparse and using split
* Rearranging tests
* Fixing FilterNotImplementedError test
* Moving imports to 'if' clauses and adding explicit test for 'cloud-watch-logs' type
* Setting names matching boto3 API and restoring 'not-implementd-filter' test
* Reformatting tests with black
* added organizations detach_policy response, model, and tests, issue #3239
Signed-off-by: Ben <ben.lipscomb@fmr.com>
* Created individual tests for detach_policy exceptions, updated regex statements for Root, OU, and Account Id
* Make ElasticAddress a tagged resource
To be able to filter on tags on ElasticAddresses, I need to have tags.
* remove unneeded commented lines
Was beginning of how to to it before further checking how it is done with other resources.
* do not ignore network-interface-owner-id filter
* add TODO about currently hardcoded region
* remove hardcoding region
* add testing for tags
creating and allocation, adding tags and querying for it
* separate test for tags into own method
* Linting
Co-authored-by: Bert Blommers <info@bertblommers.nl>
Support for cluster_id parameter substitution in autoscaling policy cloudwatch alarm dimensions.
New operations put_autoscaling_policy and remove_autoscaling_policy support
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* feature added: support for api PutUserPermissionsBoundary; DeleteRolePermissionsBoundary
* minor test fix
* lint fixed
* refractored test case
* Issue 3224 s3 copy glacier object (#3318)
* 3224 Enhancement - S3 Copy restored glacier objects
- adds setter for expiry date
- copy sets expiry date to none when source is glacier object
- throws error for copying glacier object only if not restored/still restoring
* 3224 Enhancement - S3 Copy restored glacier objects
- throws error for copying deep archive object only if not restored/still restoring
* Fix:s3 List Object response:delimiter (#3254)
* Fix:s3 List Object delimiter in response
* fixed tests
* fixed failed tests
Co-authored-by: usmankb <usman@krazybee.com>
* feature added: support for api PutUserPermissionsBoundary; DeleteRolePermissionsBoundary
* minor test fix
* lint fixed
* refractored test case
* added test case for put role exception
Co-authored-by: ruthbovell <63656505+ruthbovell@users.noreply.github.com>
Co-authored-by: usmangani1 <sgosman_chem@yahoo.com>
Co-authored-by: usmankb <usman@krazybee.com>
* Make IoT certificate id generation deterministic
Fixes#3321
As per https://stackoverflow.com/questions/55847788/how-does-aws-iot-generate-a-certificate-id,
the IoT certificate ID is the SHA256 fingerprint of the
certificate. Since moto doesn't generate full certificates we will
instead use the SHA256 hash of the passed certificate pem.
* Don't allow duplicate IoT certificates to be created
Fixes#3320
When using boto3, trying to register a certificate that already
exists will throw a ResourceAlreadyExistsException. Moto should
follow the same pattern to allow testing error handling code in
this area.
* 3224 Enhancement - S3 Copy restored glacier objects
- adds setter for expiry date
- copy sets expiry date to none when source is glacier object
- throws error for copying glacier object only if not restored/still restoring
* 3224 Enhancement - S3 Copy restored glacier objects
- throws error for copying deep archive object only if not restored/still restoring
* Change to test_s3 method test_presigned_url_restrict_parameters to tolerate change in exception messages, spurred by boto3 1.14.59 release.
* ApplicationAutoscaling: support for all the current various forms of resource_id.
* Factored logic for extracting application autoscaling resource_type from resource_id to separate function, per PR3304 comment.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
