Commit Graph

3116 Commits

Author SHA1 Message Date
Robin Wilkins
a31599d000
Support cognito admin user password auth flow (#3547)
Applies the user credentials pattern from the ADMIN_NO_SRP_AUTH flow
to the ADMIN_USER_PASSWORD_AUTH auth flow for Cognito admin_initiate_auth
requests.

Co-authored-by: Robin Wilkins <r.wilkins@waracle.com>
2020-12-15 07:48:52 +00:00
Pavel
c9c30b8286
Enabled SNS endpoint attr must be lowercased (#3543)
As per sample response:
https://docs.aws.amazon.com/sns/latest/api/API_GetEndpointAttributes.html
2020-12-14 15:08:33 +00:00
irahulranjan
54bd336457
Fix: FIFO DLQ locks message group id (#3537)
* fix https://github.com/localstack/localstack/issues/3319

* fix review comments
2020-12-14 09:19:22 +00:00
irahulranjan
7b97141184
fix Fifo queue delivers duplicate message(s) (#3538)
* fix https://github.com/localstack/localstack/issues/3339

* fixe lint issues

* Fix review comments
- move deduplication time to constants
- make tests parameterized
- update tests as per review comments

* change variable name expectedCount => expected_count

* fix tests for python 2.7
increase deduplication mock config to account for delays

* ignore time mocking test in server mode
2020-12-13 14:39:10 +00:00
irahulranjan
dffc0e449c
Fix: ListObjectsV2 behaving differently than AWS API (#3545)
* fix : localstack issue https://github.com/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22+language%3Ago

* add assertion for error message as well
2020-12-13 13:38:25 +00:00
Don Kuntz
effb075b62
Add glue.batch_update_partition (#3534)
* Add glue.batch_update_partition

* Fix error output for glue.batch_update_partition and add test case for non-existent partition in the batch update
2020-12-10 20:03:37 +00:00
Anders Kaseorg
4aff7147b5
test_stepfunctions: Fix failures under non-UTC timezone (#3528)
These tests were passing with TZ=UTC, but under a non-UTC timezone
they were failing:

E       AssertionError: given
E       X = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzutc()), …}, …]
E           and
E       Y = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzlocal()), …}, …]
E       X[0]['timestamp'] != Y[0]['timestamp']

With this fix, they pass either way.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-12-09 12:25:11 +00:00
Jordan Sanders
b4e961148f
Pass the default ECS cluster and raise accurate exceptions (#3522)
* Pass the "default" cluster

* Mock ECS exceptions more accurately

Moto's mock ECS has drifted fairly far from the actual ECS API in terms
of which exceptions it throws. This change begins to bring mock ECS's
exceptions in line with actual ECS exceptions. Most notably:

- Several custom exceptions have been replaced with their real ECS
exception. For example, "{0} is not a cluster" has been replaced with
ClusterNotFoundException
- Tests have been added to verify (most of) these exceptions work
correctly. The test coverage was a little spotty to begin with.
- The new exceptions plus the change to pass the "default" cluster
exposed a lot of places where mock ECS was behaving incorrectly. For
example, the ListTasks action is always scoped to a single cluster in
ECS but it listed tasks for all clusters in the mock. I've minimally
updated the tests to make them pass, but there's lots of opportunity to
refactor both this method's test and its implementation.

This does not provide full coverage of exceptions. In general, I ran
these operations against actual ECS resources and cross-referenced the
documentation to figure out what actual exceptions should be thrown and
what the messages should be. Consequently, I didn't update any
exceptions that took more than trivial amount of time to reproduce with
real resources.
2020-12-08 12:55:49 +00:00
Benjamin Brabant
7c7a1222d2
Fix saml-assertion parsing in assume-role-with-saml (#3523)
* Retrieve SAML Attribute by Name instead of relying on order which is too fragile

* Handle case when SAML Attribute SessionDuration is not provided, as it is not a required attribute from SAML response

When session duration not provided, AWS consider by default a duration of one hour as cited in the following documentation:
"If this attribute is not present, then the credential last for one hour (the default value of the DurationSeconds parameter of the AssumeRoleWithSAML API)."
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml_assertions.html#saml_role-session-duration

Traceback was:
[...]
  File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/responses.py", line 79, in assume_role_with_saml
    role = sts_backend.assume_role_with_saml(
  File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/models.py", line 99, in assume_role_with_saml
    role = AssumedRole(**kwargs)
TypeError: __init__() missing 1 required positional argument: 'duration'

* Process saml xml namespaces properly instead of relying on textual prefix that can vary between identity providers

* Handle when SAML response AttributeValue xml tag contains attributes that force xmltodict to build a dictionary as for complex types instead of directly returning string value

Leverage force_cdata option of xmltodict parser that always return a complex dictionary even if xml tag contains only text and no attributes.

* Improve existing test_assume_role_with_saml to be coherent with other assume_role_with_saml tests and remove dead code at the same time
2020-12-08 09:08:40 +00:00
usmangani1
af60306371
Fix:Ec2-VPC:Added functionality describe-vpc-endpoint (#3524)
* Fix:Ec2:Added fucntionality describe-vpc-endpoint

* Refactor

* Added test cases for errors
2020-12-07 21:39:57 +00:00
Bert Blommers
e1fc3a9596
#3506 - DynamoDB - Allow StringSets to be passed to update_item() (#3519) 2020-12-07 09:31:53 +00:00
usmangani1
d44beb0c9d
Fix:Cloudformation descibe stacks special chars (#3514)
* Fix:descibe stacks special chars

* removed debug statements

* Linting

Co-authored-by: usmanokc <usman@okcredit.in>
2020-12-06 08:11:52 +00:00
usmangani1
cfaaa70ea9
FIX:Ec2:Added:Tags for EIP (#3505)
* FIX:Ec2:Added:Tags for EIP

* removed files

* Fix tests
2020-12-06 00:04:46 +00:00
Brian Pandola
8f83f51897
Refactor tests marked xfail to use proper assertions (#3515) 2020-12-04 16:00:48 +00:00
Brian Pandola
dcd034045b
Fix: ClusterType, NumberOfNodes not validated when modifying Redshift cluster (#3513)
By definition, `single-node` clusters can only consist of 1 node. Likewise,
`multi-node` clusters must have 2 or more nodes.

* Ensure `ClusterType` parameter is either `multi-node` or `single-node`.
* Ensure proper validation of `NumberOfNodes` parameter based on `ClusterType`
  parameter.
* Fix existing test case that incorrectly allowed a `multi-node` cluster to
  consist of 1 node.
* Add dedicated test for resizing a cluster from `single-node` to `multi-node`
  and back again.

Behavior and error messages have been verified against a real AWS backend.
2020-12-04 13:22:19 +00:00
Brian Pandola
5e21e50424
Fix: elasticbeanstalk ApplicationArn does not contain ApplicationName (#3511)
Fixes #3510
2020-12-04 10:31:28 +00:00
Steve Pulec
b6369d6250
Merge pull request #3512 from xnegativx/fix-auth-params
Fix challenge parameters for cognito-idp/initiate_auth flow
2020-12-03 18:07:32 -06:00
Steve Pulec
51928f2410
Merge pull request #3508 from bblommers/feature/ec2-describe-instance-type-offerings
EC2: describe instance type offerings
2020-12-03 18:06:06 -06:00
Ciaran Evans
48df5bd5af
Add basic get_execution_history implementation for Step Functions (#3507)
* Add format command to makefile

* Refactor executions to be a attribute of StateMachine

* Begin to add tests for execution history

* Add tests for failed and successful event histories, with implementations

* Add failure case to environment var check

* Skip test if in server mode and update implementation coverage

* Add conditional import for mock to cover python 2

* Refactor stop execution logic into StateMachine

* Refactor event history environment variable into settings.py

* Remove typing and os import
2020-12-03 18:32:06 +00:00
Antonin
c5d0aa0c10 fix SALT and SRP_B params 2020-12-03 11:42:10 +01:00
usmangani1
ffa7f2e41a
Fix:SES:Get Template Html part (#3504)
* SES:Get Template Html part

* Linting

* Linting

Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-12-03 07:42:19 +00:00
Bert Blommers
936fb19539 DescribeInstanceTypeOfferings - script that hardcodes all offerings 2020-12-02 14:41:24 +00:00
Bert Blommers
5d208bfd04 Merge branch 'master' into add_instance_type_offerings 2020-12-02 10:57:07 +00:00
usmangani1
b2adcdf518
Fix:RDS:add DBParameterGroupArn in describe-db-parameter-groups & cre… (#3462)
* Fix:RDS:add DBParameterGroupArn in describe-db-parameter-groups & create-db-parameter-group

* Test change

* Fixed tests

* tests change acconutID

* linting

Co-authored-by: usmankb <usman@krazybee.com>
2020-12-01 19:53:01 +00:00
Brian Pandola
ae85c539fd Remove boto package dependency
The `boto` library (long ago superseded by `boto3`) has not had an official
release in over two years or even a commit in the last 18 months.  Importing
the package (or indirectly importing it by via `moto`) generates a deprecation
warning.  Additionally, an ever-increasing number of `moto` users who have
left `boto` behind for `boto3` are still being forced to install `boto`.

This commit vendors a very small subset of the `boto` library--only the code
required by `moto` to run--into the /packages subdirectory.  A README file
is included explaining the origin of the files and a recommendation for how
they can be removed entirely in a future release.

NOTE: Users of `boto` will still be able to use `moto` after this is merged.

closes #2978
closes #3013
closes #3170
closes #3418

relates to #2950
2020-11-26 23:59:15 -08:00
Garrett
9e3b23758a
[dynamodb2] Support include projection on indexes (#3498)
* [dynamodb2] Support include projection on indexes

* linter
2020-11-25 20:28:05 +00:00
Brian Pandola
d58d3e2c2e
Fix: yield tests ignored by pytest runner (#3500)
Closes #3499
2020-11-25 10:48:05 +00:00
Brian Pandola
5a2cbf1eca Fix: Duplicate test name causing loss of coverage
A test added in #2401 copied the name of an existing test, preventing it
from being run.  This commit renames the second test, allowing both to
be picked up by the test runner.
2020-11-24 03:29:52 -08:00
Brian Pandola
cf7869d0e2 Add check for UnknownSnapshotCopyRegionFault error 2020-11-24 03:29:52 -08:00
Brian Pandola
b4d7d183ab Add additional detail to ClientError assertions
We check the message now to ensure we've raised the *correct* ClientError
2020-11-24 03:29:52 -08:00
Brian Pandola
49c6e65603 Fix: DeleteCluster behavior with SkipFinalClusterSnapshot
Original code was trying to raise a ClientError directly.  Change to
appropriate Redshift exception class.

* Fix test assertion for `boto`.
* Add test coverage for `boto3`.
2020-11-24 03:29:52 -08:00
Brian Pandola
555be78f6e Fix: redshift:DescribeClusterSnapshots should not raise ClusterNotFoundError
Real AWS backend returns an empty array instead of raising an error.
2020-11-24 03:29:52 -08:00
Brian Pandola
161cb46886 Add coverage for ContentType=JSON server requests
The `boto` library explicitly requests JSON responses from Redshift endpoints
2020-11-24 03:29:52 -08:00
Nuwan Goonasekera
c658cfea0d
Run black and fix tests 2020-11-24 00:51:00 +05:30
Nuwan Goonasekera
114a8efac8
Improved describe instance types + tests 2020-11-23 15:30:52 +05:30
Nuwan Goonasekera
a69bad57ef
Added instance type offer filters 2020-11-23 15:30:52 +05:30
Brian Pandola
53a3e52c67
Fix: EMR ReleaseLabel validation does not respect semantic versioning (#3489)
Fixes #3474
2020-11-22 18:54:59 +00:00
Brian Pandola
93b393c679
Fix: Python 2/3 Incompatibility (#3488)
Previous code would raise `TypeError: 'dict_keys' object is not subscriptable`
when run under Python 3.

* Re-write code in Python 2/3 compatible way.
* Add clarifying comment.
* Add test coverage.

Supersedes #3227
2020-11-21 13:36:33 +00:00
Brian Pandola
4245497a97
Fix: redshift:EnableSnapshotCopy raises incorrect Exception (#3485)
The previous code was trying to raise a botocore ClientError directly, which
was actually generating a secondary AttributeError because the arguments passed
to ClientError() were incorrect.

This replaces the ClientError() call with a proper moto exception class for
Redshift and fixes the test assertions appropriately.

Supersedes #1957
2020-11-21 10:35:46 +00:00
Mykola Mykhalov
689cd8f285
Implement IAM instance profile associations (#3482)
* Add associate_iam_instance_profile describe_iam_instance_profile_associations, disassociate_iam_instance_profile, replace_iam_instance_profile_association

* More tests, removed type hints, filter fix

* Ec2 fix
2020-11-20 22:00:53 +00:00
usmangani1
54e296eb53
Fix:CloudWatch List metrics with dimensions (#3461)
* Fix:CloudWatch List metrics with dimensions

* Fix:CloudWatch List metrics with dimensions

* Fixed new cases and added more tests

Co-authored-by: usmankb <usman@krazybee.com>
2020-11-20 12:38:48 +00:00
Rafael de Elvira
8a95878a81
Route53: Implement dummy GetChange endpoint (#3486)
* Route53: Implement dummy GetChange endpoint

* fix typo in GET_CHANGE_RESPONSE
2020-11-20 07:21:05 +00:00
Brian Pandola
f7467164e4
Fix Race Condition in batch:SubmitJob (#3480)
* Extract Duplicate Code into Helper Method

DRY up the tests and replace the arbitrary `sleep()` calls with a more
explicit check before progressing.

* Improve Testing of batch:TerminateJob

The test now confirms that the job was terminated by sandwiching a `sleep`
command between two `echo` commands.  In addition to the original checks
of the terminated job status/reason, the test now asserts that only the
first echo command succeeded, confirming that the job was indeed terminated
while in progress.

* Fix Race Condition in batch:SubmitJob

The `test_submit_job` in `test_batch.py` kicks off a job, calls `describe_jobs`
in a loop until the job status returned is SUCCEEDED, and then asserts against
the logged events.

The backend code that runs the submitted job does so in a separate thread. If
the job was successful, the job status was being set to SUCCEEDED *before* the
event logs had been written to the logging backend.

As a result, it was possible for the primary thread running the test to detect
that the job was successful immediately after the secondary thread had updated
the job status but before the secondary thread had written the logs to the
logging backend.  Under the right conditions, this could cause the subsequent
logging assertions in the primary thread to fail.

Additionally, the code that collected the logs from the container was using
a "dodgy hack" of time.sleep() and a modulo-based conditional that was
ultimately non-deterministic and could result in log messages being dropped
or duplicated in certain scenarios.

In order to address these issues, this commit does the following:

* Carefully re-orders any code that sets a job status or timestamp
  to avoid any obvious race conditions.
* Removes the "dodgy hack" in favor of a much more straightforward
  (and less error-prone) method of collecting logs from the container.
* Removes arbitrary and unnecessary calls to time.sleep()

Before applying any changes, the flaky test was failing about 12% of the
time.  Putting a sleep() call between setting the `job_status` to SUCCEEDED
and collecting the logs, resulted in a 100% failure rate.  Simply moving
the code that sets the job status to SUCCEEDED to the end of the code block,
dropped the failure rate to ~2%.  Finally, removing the log collection
hack allowed the test suite to run ~1000 times without a single failure.

Taken in aggregate, these changes make the batch backend more deterministic
and should put the nail in the coffin of this flaky test.

Closes #3475
2020-11-18 10:49:25 +00:00
Oide Brett
83507fbc37
fixed issue in update_configuration for lambda when setting VPC config property (#3479) 2020-11-18 08:45:31 +00:00
Ayush Ghosh
7f73015f02
Fix XML encoding in Route53 JInja2 Templates #3469 (#3473)
* Use Jinja2 escape functionality to escape html attributes in value response

* Add tests

* fix formatting
2020-11-18 07:23:49 +00:00
Guillermo Arribas
62fd975da0
EventBridge: put_rule and list_rules should store and retrieve EventBusName property (#3472)
Co-authored-by: Guillermo Arribas <garribas@atlassian.com>
2020-11-17 15:36:17 +00:00
jweite
5fe921c2bc
Added support for EMR Security Configurations and Kerberos Attributes. (#3456)
* Added support for EMR Security Configurations and Kerberos Attributes.

* Revised exception-raising test to work with pytest api.

* Added htmlcov to .gitignore; upgrading botocore to 1.18.17, per commit d29475e.

Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-11-17 10:54:34 +00:00
Rich Unger
f045af7e0a
Add support for empty strings in non-key dynamo attributes (#3467)
* Add support for empty strings in non-key attributes

https://github.com/spulec/moto/issues/3339

* Nose, not pytest

* Revert "Nose, not pytest"

This reverts commit 5a3cf6c887dd9fafa49096c82cfa3a3b7f91d224.

* PUT is default action
2020-11-17 09:12:39 +00:00
Oide Brett
62d382ff70
Fixed issue 3448 for DynamoDB update_item (#3463)
* Fixed issue 3448 for DynamoDB update_item

* Tidied up fix for issue 3448 for DynamoDB update_item

* Reformatted fix for issue 3448 for DynamoDB update_item

* removed use of f-strings in test case as it fails in Travis CI build due to Python 2.7 support of f strings
2020-11-17 07:41:54 +00:00
Brian Pandola
d29475ed19
Fix: TagList missing in rds:DescribeDBInstance response (#3459)
Previously, tags were only available via rds:ListTagsForResource, but are now
included in the Create/DescribeDBInstance responses as of Botocore 1.18.17[1]

[1]: f29d23c53e (diff-d10722c0e11ded323c8d240066d7ed31e93a1e6423d54e091b7d54b86e6bd4e0)

Fixes #3458
2020-11-16 09:30:53 +00:00