Commit Graph

3293 Commits

Author SHA1 Message Date
Laurie O
ddd3c0edc4
Assume synchronous Lambda invocation by default (#3615)
* Assume synchronous Lambda invocation by default

* Support Python 2 with dict-unpacking
2021-01-26 13:28:01 +00:00
Bert Blommers
8591eda9d6
Introduce Github Actions to replace TravisCI (#3610) 2021-01-26 12:37:03 +00:00
Andrea Amorosi
5a41866f71
Added redshift.get_cluster_credentials (#3611)
* Added redshift.get_cluster_credentials

* Marked endpoint in list

* Removed f string from tests

* Python 2.7 compat changes

* Fixed parameter retrieval

* Formatting

* Removed try/catch in favor of if

* Changed to existing random_string util

Co-authored-by: Andrea Amorosi <aamorosi@amazon.es>
2021-01-25 13:19:50 +00:00
Brian Pandola
38124ab1c9
Fix ec2 filter by empty tag value (#3605)
* Fix ec2 filter by empty tag value

Return `None` instead of an empty string when the tag key does not exist
and replace the falsy check with a more explicit `is None`, which allows
empty string values to correctly pass through the filter comparator.

Behavior confirmed against a real AWS backend.

Closes #3603

* Make test case more explicit

Test case now pulled directly from the issue report (#3603).

Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
2021-01-24 12:00:25 +00:00
Brian Pandola
cb03223c9b
Add TagSpecification support to ec2:CreateImage (#3604)
* Bump `botocore` to minimum version that supports TagSpecifications for this action.
* Add test coverage.

Closes #3602
2021-01-23 12:57:34 +00:00
Jordan Dimov
f11e3183bb
Implementation of core AWS Media Live endpoins (#3428)
* Initial attempt to mock AWS Media Live create_channel endpoint. Test fails.

* Completes basic implementation of Media Live create_channel endpoint

* Completes basic implementation of Media Live list_channels endpoint

* Adds skaffolds for describe_channel and delete_channel

* Adds unit test for delete_channel

* Adds unit test for describe_channel

* Reduces repetitive code by introducing a Channel model

* Implements MediaLive start_channel and stop_channel endpoints

* Fixes lack of support for the dash character in resource ARNs

* Implements MediaLive update_channel endpoint.

* Implements MediaLive create_input endpoint (and Input model).

* Implements MediaLive describe_input endpoint.

* Implements MediaLive list_inputs endpoint.

* Implements MediaLive update_input endpoint.

* Addse server tests for MediaLive

* Adds further url patterns for medialive

* Fixes url patterns

* Fixes url patterns
2021-01-19 16:11:39 +00:00
Macwan Nevil
02ac5ca111
added sse support for s3 (#3592)
* added sse support for s3

* lint fixed for py37
2021-01-18 18:17:13 +00:00
Erez Freiberger
9784e1b487
adding layers support for lambda backend (#3563)
* adding layers support for lambda backend

* improving lambda layer tests

* adding lambda list_layers and fixing tests

* make format
2021-01-17 15:28:49 +00:00
Modhaffer Rahmani
8fe5a680a5
Added explicit exception raise when no stack found. (#3559)
* Added explicit exception raise when no stack found.

Currently, any operation that uses 'get_stack' method from 'CloudFormationBackend' class
will fail with AttributeError or jinja2 exception if ran against non-existing stack(created/deleted)
To fix the issue I explicitly raised a 'ValidationError' exception.
Added tests for boto and boto3 responses.

* Moved non-existing stack tests to 'test_stack_events'

When using 'update_stack' to test raising an exception when the stack doesn't exist
test coverage dropped by 0.5%. I am using stack_events instead.

* Removed some unreachable paths

After adding the exception couple of paths in the code are unreachable as 'get_stack' doesn't return 'None' anymore.
This is the reason why coverall was reporting decreased coverage.

* Removed an unreachable path I missed

* Added couple of tests in cloudformation/models

* Added more assertions around raised exception

* Formatted document using black to fix issue with travis.
2021-01-16 14:42:51 +00:00
Rahul Ranjan
628c026a07
Sort s3 object versions by last_modified_on in descending order (#3588)
* Sort s3 object versions by last_modified_on in descending order

- addresses https://github.com/localstack/localstack/issues/3433

* fix for python 2

* fix lint
2021-01-15 18:28:28 +00:00
Brian Pandola
7ff60683e0
Expose contextual data when Lambda test fails (#3590)
This test is flaky, but when it fails we don't get any indication as to why.
This commit ensures that the reason for failure will be part of the assertion
message.

Once we have information about why this test fails, we can troubleshoot further
and hopefully come up with a permanent fix.
2021-01-15 14:42:35 +00:00
Anders Kaseorg
8befcb6a48
Leave the global responses configuration alone (#3529)
Instead of modifying responses._default_mock, create our own
responses.RequestsMock object that we can modify as needed without
interfering with other users of the responses library.

Fixes #3264.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2021-01-15 09:12:41 +00:00
Szymon Zmilczak
c8151e1bb4
IoT: Added Rules implementation (#3552)
* feat(iot): Added IoT Rules implementation

* fix(iot): Split IoT Rules tests into multiple test cases

* fix(iot): Updated rule creation to work with python 2

Co-authored-by: Szymon Zmilczak <szymon.zmilczak@machiq.com>
2021-01-14 14:52:23 +00:00
Bert Blommers
d1696a37e5
#2710 - S3 - Raise specific error when GetObject is called with unknown VersionId (#3532) 2021-01-13 16:06:09 +00:00
Bert Blommers
0f32f3c50c
#3542 - CloudFormation - Fix SecurityGroup ingress rules (#3553) 2021-01-13 13:36:56 +00:00
Bert Blommers
df8bd43a45
#2965 - S3 - GetObject - Treat empty Range-argument as a regular GET (#3531) 2021-01-13 10:00:18 +00:00
Bert Blommers
77dd35e3b8
Use known ImageID in internal tests (#3555) 2021-01-13 09:02:11 +00:00
aaronfowles-emis
a2246df1a4
Implement full Database object for Glue get_database() - fix for #3571. (#3572)
* implement potential fix for #3571.

* freeze_time decorator not used in TEST_SERVER_MODE
2021-01-11 13:10:18 +00:00
Anton Grübel
17d94f9e09
Add validation to put_events (#3579)
* Add validation to put_events

* Add extra test for put_events

* Fix Python 2.7 error
2021-01-11 09:41:31 +00:00
Erez Freiberger
f749f583ee
Add get_function_configuration support for Lambda (#3562)
* Add get_function_configuration support for Lambda

* remove unnesecary code from test and use _lambda_region when asserting

* rename function and skip coping configuration

* run black formatting
2021-01-10 15:24:04 +00:00
Brian Pandola
640df04840
Implement Add to List for dynamodb:UpdateItem (#3585)
This handles the add-to-list case using the legacy `AttributeUpdates` parameter.

* Added test coverage.
* Verified against real AWS backend.

Closes #3561
2021-01-10 14:20:41 +00:00
Miltos
13252cc4e3
Fix ResourceNotFoundError exception message (#3582)
* Fix ResourceNotFoundError message

* Fix and update unit test for invalid `describe_identity_pool`.
2021-01-10 13:26:40 +00:00
Jordan Sanders
6dfd64ff3c
More accurately mock ECS RegisterTaskDefinition (#3584)
The mocked response for ECS RegisterTaskDefinition has drifted from what
actually returns when run against a real ECS endpoint. I created a
minimal task definition for both EC2:

```
>>> ecs.register_task_definition(
      family="moto",
      containerDefinitions=[
          {
              "name": "hello_world",
              "image": "hello-world:latest",
              "memory": 400
           }
       ]
  )["taskDefinition"]

{'taskDefinitionArn': 'arn:aws:ecs:us-east-1:************:task-definition/moto:1',
 'containerDefinitions': [{'name': 'hello_world',
   'image': 'hello-world:latest',
   'cpu': 0,
   'memory': 400,
   'portMappings': [],
   'essential': True,
   'environment': [],
   'mountPoints': [],
   'volumesFrom': []}],
 'family': 'moto',
 'revision': 1,
 'volumes': [],
 'status': 'ACTIVE',
 'placementConstraints': [],
 'compatibilities': ['EC2']}
```

and FARGATE:
```
>>> ecs.register_task_definition(
      family="moto",
      containerDefinitions=[
          {
              "name": "hello_world",
              "image": "hello-world:latest",
              "memory": 400
           }
       ],
       requiresCompatibilities=["FARGATE"],
       networkMode="awsvpc",
       cpu="256",
       memory="512"
  )["taskDefinition"]

{'taskDefinitionArn': 'arn:aws:ecs:us-east-1:************:task-definition/moto:2',
 'containerDefinitions': [{'name': 'hello_world',
   'image': 'hello-world:latest',
   'cpu': 0,
   'memory': 400,
   'portMappings': [],
   'essential': True,
   'environment': [],
   'mountPoints': [],
   'volumesFrom': []}],
 'family': 'moto',
 'networkMode': 'awsvpc',
 'revision': 2,
 'volumes': [],
 'status': 'ACTIVE',
 'requiresAttributes': [{'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.18'},
  {'name': 'ecs.capability.task-eni'}],
 'placementConstraints': [],
 'compatibilities': ['EC2', 'FARGATE'],
 'requiresCompatibilities': ['FARGATE'],
 'cpu': '256',
 'memory': '512'}
```

This change adds several default keys to the task based on those two
real responses and the AWS documentation:
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RegisterTaskDefinition.html

The mock still doesn't match the real response exactly but it's much
closer than it was before.
2021-01-09 14:07:35 +00:00
Anton Grübel
f4418185d1
Add launchType parameter to ECS (#3578) 2021-01-08 17:14:40 +00:00
Brian Pandola
d712a98ce1
Enable AWSLambda and STS mocking for AWS China regions/endpoints (#3574)
Fixes #3570
2021-01-08 14:22:12 +00:00
Brian Pandola
1a98c4f14a
Fix: handling of boolean value privateDnsEnabled in ec2:VpcEndpoint (#3566)
* Properly coerce `privateDnsEnabled` to boolean value when parsing requests.
* Per AWS spec, default `privateDnsEnabled` request value to `True`.
* Properly serialize `privateDnsEnabled` as boolean value in responses.
* Add test coverage.

Ref: #3540
2021-01-08 08:05:44 +00:00
usmangani1
bce682a867
Fix:Added nat-gateway tags (#3560)
* Fix:Added nat-gateway tags

* Removed changes
2021-01-07 17:59:46 +00:00
Robin Wilkins
a31599d000
Support cognito admin user password auth flow (#3547)
Applies the user credentials pattern from the ADMIN_NO_SRP_AUTH flow
to the ADMIN_USER_PASSWORD_AUTH auth flow for Cognito admin_initiate_auth
requests.

Co-authored-by: Robin Wilkins <r.wilkins@waracle.com>
2020-12-15 07:48:52 +00:00
Pavel
c9c30b8286
Enabled SNS endpoint attr must be lowercased (#3543)
As per sample response:
https://docs.aws.amazon.com/sns/latest/api/API_GetEndpointAttributes.html
2020-12-14 15:08:33 +00:00
irahulranjan
54bd336457
Fix: FIFO DLQ locks message group id (#3537)
* fix https://github.com/localstack/localstack/issues/3319

* fix review comments
2020-12-14 09:19:22 +00:00
irahulranjan
7b97141184
fix Fifo queue delivers duplicate message(s) (#3538)
* fix https://github.com/localstack/localstack/issues/3339

* fixe lint issues

* Fix review comments
- move deduplication time to constants
- make tests parameterized
- update tests as per review comments

* change variable name expectedCount => expected_count

* fix tests for python 2.7
increase deduplication mock config to account for delays

* ignore time mocking test in server mode
2020-12-13 14:39:10 +00:00
irahulranjan
dffc0e449c
Fix: ListObjectsV2 behaving differently than AWS API (#3545)
* fix : localstack issue https://github.com/issues?q=is%3Aopen+is%3Aissue+label%3A%22good+first+issue%22+language%3Ago

* add assertion for error message as well
2020-12-13 13:38:25 +00:00
Don Kuntz
effb075b62
Add glue.batch_update_partition (#3534)
* Add glue.batch_update_partition

* Fix error output for glue.batch_update_partition and add test case for non-existent partition in the batch update
2020-12-10 20:03:37 +00:00
Anders Kaseorg
4aff7147b5
test_stepfunctions: Fix failures under non-UTC timezone (#3528)
These tests were passing with TZ=UTC, but under a non-UTC timezone
they were failing:

E       AssertionError: given
E       X = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzutc()), …}, …]
E           and
E       Y = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzlocal()), …}, …]
E       X[0]['timestamp'] != Y[0]['timestamp']

With this fix, they pass either way.

Signed-off-by: Anders Kaseorg <andersk@mit.edu>
2020-12-09 12:25:11 +00:00
Jordan Sanders
b4e961148f
Pass the default ECS cluster and raise accurate exceptions (#3522)
* Pass the "default" cluster

* Mock ECS exceptions more accurately

Moto's mock ECS has drifted fairly far from the actual ECS API in terms
of which exceptions it throws. This change begins to bring mock ECS's
exceptions in line with actual ECS exceptions. Most notably:

- Several custom exceptions have been replaced with their real ECS
exception. For example, "{0} is not a cluster" has been replaced with
ClusterNotFoundException
- Tests have been added to verify (most of) these exceptions work
correctly. The test coverage was a little spotty to begin with.
- The new exceptions plus the change to pass the "default" cluster
exposed a lot of places where mock ECS was behaving incorrectly. For
example, the ListTasks action is always scoped to a single cluster in
ECS but it listed tasks for all clusters in the mock. I've minimally
updated the tests to make them pass, but there's lots of opportunity to
refactor both this method's test and its implementation.

This does not provide full coverage of exceptions. In general, I ran
these operations against actual ECS resources and cross-referenced the
documentation to figure out what actual exceptions should be thrown and
what the messages should be. Consequently, I didn't update any
exceptions that took more than trivial amount of time to reproduce with
real resources.
2020-12-08 12:55:49 +00:00
Benjamin Brabant
7c7a1222d2
Fix saml-assertion parsing in assume-role-with-saml (#3523)
* Retrieve SAML Attribute by Name instead of relying on order which is too fragile

* Handle case when SAML Attribute SessionDuration is not provided, as it is not a required attribute from SAML response

When session duration not provided, AWS consider by default a duration of one hour as cited in the following documentation:
"If this attribute is not present, then the credential last for one hour (the default value of the DurationSeconds parameter of the AssumeRoleWithSAML API)."
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml_assertions.html#saml_role-session-duration

Traceback was:
[...]
  File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/responses.py", line 79, in assume_role_with_saml
    role = sts_backend.assume_role_with_saml(
  File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/models.py", line 99, in assume_role_with_saml
    role = AssumedRole(**kwargs)
TypeError: __init__() missing 1 required positional argument: 'duration'

* Process saml xml namespaces properly instead of relying on textual prefix that can vary between identity providers

* Handle when SAML response AttributeValue xml tag contains attributes that force xmltodict to build a dictionary as for complex types instead of directly returning string value

Leverage force_cdata option of xmltodict parser that always return a complex dictionary even if xml tag contains only text and no attributes.

* Improve existing test_assume_role_with_saml to be coherent with other assume_role_with_saml tests and remove dead code at the same time
2020-12-08 09:08:40 +00:00
usmangani1
af60306371
Fix:Ec2-VPC:Added functionality describe-vpc-endpoint (#3524)
* Fix:Ec2:Added fucntionality describe-vpc-endpoint

* Refactor

* Added test cases for errors
2020-12-07 21:39:57 +00:00
Bert Blommers
e1fc3a9596
#3506 - DynamoDB - Allow StringSets to be passed to update_item() (#3519) 2020-12-07 09:31:53 +00:00
usmangani1
d44beb0c9d
Fix:Cloudformation descibe stacks special chars (#3514)
* Fix:descibe stacks special chars

* removed debug statements

* Linting

Co-authored-by: usmanokc <usman@okcredit.in>
2020-12-06 08:11:52 +00:00
usmangani1
cfaaa70ea9
FIX:Ec2:Added:Tags for EIP (#3505)
* FIX:Ec2:Added:Tags for EIP

* removed files

* Fix tests
2020-12-06 00:04:46 +00:00
Brian Pandola
8f83f51897
Refactor tests marked xfail to use proper assertions (#3515) 2020-12-04 16:00:48 +00:00
Brian Pandola
dcd034045b
Fix: ClusterType, NumberOfNodes not validated when modifying Redshift cluster (#3513)
By definition, `single-node` clusters can only consist of 1 node. Likewise,
`multi-node` clusters must have 2 or more nodes.

* Ensure `ClusterType` parameter is either `multi-node` or `single-node`.
* Ensure proper validation of `NumberOfNodes` parameter based on `ClusterType`
  parameter.
* Fix existing test case that incorrectly allowed a `multi-node` cluster to
  consist of 1 node.
* Add dedicated test for resizing a cluster from `single-node` to `multi-node`
  and back again.

Behavior and error messages have been verified against a real AWS backend.
2020-12-04 13:22:19 +00:00
Brian Pandola
5e21e50424
Fix: elasticbeanstalk ApplicationArn does not contain ApplicationName (#3511)
Fixes #3510
2020-12-04 10:31:28 +00:00
Steve Pulec
b6369d6250
Merge pull request #3512 from xnegativx/fix-auth-params
Fix challenge parameters for cognito-idp/initiate_auth flow
2020-12-03 18:07:32 -06:00
Steve Pulec
51928f2410
Merge pull request #3508 from bblommers/feature/ec2-describe-instance-type-offerings
EC2: describe instance type offerings
2020-12-03 18:06:06 -06:00
Ciaran Evans
48df5bd5af
Add basic get_execution_history implementation for Step Functions (#3507)
* Add format command to makefile

* Refactor executions to be a attribute of StateMachine

* Begin to add tests for execution history

* Add tests for failed and successful event histories, with implementations

* Add failure case to environment var check

* Skip test if in server mode and update implementation coverage

* Add conditional import for mock to cover python 2

* Refactor stop execution logic into StateMachine

* Refactor event history environment variable into settings.py

* Remove typing and os import
2020-12-03 18:32:06 +00:00
Antonin
c5d0aa0c10 fix SALT and SRP_B params 2020-12-03 11:42:10 +01:00
usmangani1
ffa7f2e41a
Fix:SES:Get Template Html part (#3504)
* SES:Get Template Html part

* Linting

* Linting

Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-12-03 07:42:19 +00:00
Bert Blommers
936fb19539 DescribeInstanceTypeOfferings - script that hardcodes all offerings 2020-12-02 14:41:24 +00:00
Bert Blommers
5d208bfd04 Merge branch 'master' into add_instance_type_offerings 2020-12-02 10:57:07 +00:00
usmangani1
b2adcdf518
Fix:RDS:add DBParameterGroupArn in describe-db-parameter-groups & cre… (#3462)
* Fix:RDS:add DBParameterGroupArn in describe-db-parameter-groups & create-db-parameter-group

* Test change

* Fixed tests

* tests change acconutID

* linting

Co-authored-by: usmankb <usman@krazybee.com>
2020-12-01 19:53:01 +00:00
Brian Pandola
ae85c539fd Remove boto package dependency
The `boto` library (long ago superseded by `boto3`) has not had an official
release in over two years or even a commit in the last 18 months.  Importing
the package (or indirectly importing it by via `moto`) generates a deprecation
warning.  Additionally, an ever-increasing number of `moto` users who have
left `boto` behind for `boto3` are still being forced to install `boto`.

This commit vendors a very small subset of the `boto` library--only the code
required by `moto` to run--into the /packages subdirectory.  A README file
is included explaining the origin of the files and a recommendation for how
they can be removed entirely in a future release.

NOTE: Users of `boto` will still be able to use `moto` after this is merged.

closes #2978
closes #3013
closes #3170
closes #3418

relates to #2950
2020-11-26 23:59:15 -08:00
Garrett
9e3b23758a
[dynamodb2] Support include projection on indexes (#3498)
* [dynamodb2] Support include projection on indexes

* linter
2020-11-25 20:28:05 +00:00
Brian Pandola
d58d3e2c2e
Fix: yield tests ignored by pytest runner (#3500)
Closes #3499
2020-11-25 10:48:05 +00:00
Brian Pandola
5a2cbf1eca Fix: Duplicate test name causing loss of coverage
A test added in #2401 copied the name of an existing test, preventing it
from being run.  This commit renames the second test, allowing both to
be picked up by the test runner.
2020-11-24 03:29:52 -08:00
Brian Pandola
cf7869d0e2 Add check for UnknownSnapshotCopyRegionFault error 2020-11-24 03:29:52 -08:00
Brian Pandola
b4d7d183ab Add additional detail to ClientError assertions
We check the message now to ensure we've raised the *correct* ClientError
2020-11-24 03:29:52 -08:00
Brian Pandola
49c6e65603 Fix: DeleteCluster behavior with SkipFinalClusterSnapshot
Original code was trying to raise a ClientError directly.  Change to
appropriate Redshift exception class.

* Fix test assertion for `boto`.
* Add test coverage for `boto3`.
2020-11-24 03:29:52 -08:00
Brian Pandola
555be78f6e Fix: redshift:DescribeClusterSnapshots should not raise ClusterNotFoundError
Real AWS backend returns an empty array instead of raising an error.
2020-11-24 03:29:52 -08:00
Brian Pandola
161cb46886 Add coverage for ContentType=JSON server requests
The `boto` library explicitly requests JSON responses from Redshift endpoints
2020-11-24 03:29:52 -08:00
Nuwan Goonasekera
c658cfea0d
Run black and fix tests 2020-11-24 00:51:00 +05:30
Nuwan Goonasekera
114a8efac8
Improved describe instance types + tests 2020-11-23 15:30:52 +05:30
Nuwan Goonasekera
a69bad57ef
Added instance type offer filters 2020-11-23 15:30:52 +05:30
Brian Pandola
53a3e52c67
Fix: EMR ReleaseLabel validation does not respect semantic versioning (#3489)
Fixes #3474
2020-11-22 18:54:59 +00:00
Brian Pandola
93b393c679
Fix: Python 2/3 Incompatibility (#3488)
Previous code would raise `TypeError: 'dict_keys' object is not subscriptable`
when run under Python 3.

* Re-write code in Python 2/3 compatible way.
* Add clarifying comment.
* Add test coverage.

Supersedes #3227
2020-11-21 13:36:33 +00:00
Brian Pandola
4245497a97
Fix: redshift:EnableSnapshotCopy raises incorrect Exception (#3485)
The previous code was trying to raise a botocore ClientError directly, which
was actually generating a secondary AttributeError because the arguments passed
to ClientError() were incorrect.

This replaces the ClientError() call with a proper moto exception class for
Redshift and fixes the test assertions appropriately.

Supersedes #1957
2020-11-21 10:35:46 +00:00
Mykola Mykhalov
689cd8f285
Implement IAM instance profile associations (#3482)
* Add associate_iam_instance_profile describe_iam_instance_profile_associations, disassociate_iam_instance_profile, replace_iam_instance_profile_association

* More tests, removed type hints, filter fix

* Ec2 fix
2020-11-20 22:00:53 +00:00
usmangani1
54e296eb53
Fix:CloudWatch List metrics with dimensions (#3461)
* Fix:CloudWatch List metrics with dimensions

* Fix:CloudWatch List metrics with dimensions

* Fixed new cases and added more tests

Co-authored-by: usmankb <usman@krazybee.com>
2020-11-20 12:38:48 +00:00
Rafael de Elvira
8a95878a81
Route53: Implement dummy GetChange endpoint (#3486)
* Route53: Implement dummy GetChange endpoint

* fix typo in GET_CHANGE_RESPONSE
2020-11-20 07:21:05 +00:00
Brian Pandola
f7467164e4
Fix Race Condition in batch:SubmitJob (#3480)
* Extract Duplicate Code into Helper Method

DRY up the tests and replace the arbitrary `sleep()` calls with a more
explicit check before progressing.

* Improve Testing of batch:TerminateJob

The test now confirms that the job was terminated by sandwiching a `sleep`
command between two `echo` commands.  In addition to the original checks
of the terminated job status/reason, the test now asserts that only the
first echo command succeeded, confirming that the job was indeed terminated
while in progress.

* Fix Race Condition in batch:SubmitJob

The `test_submit_job` in `test_batch.py` kicks off a job, calls `describe_jobs`
in a loop until the job status returned is SUCCEEDED, and then asserts against
the logged events.

The backend code that runs the submitted job does so in a separate thread. If
the job was successful, the job status was being set to SUCCEEDED *before* the
event logs had been written to the logging backend.

As a result, it was possible for the primary thread running the test to detect
that the job was successful immediately after the secondary thread had updated
the job status but before the secondary thread had written the logs to the
logging backend.  Under the right conditions, this could cause the subsequent
logging assertions in the primary thread to fail.

Additionally, the code that collected the logs from the container was using
a "dodgy hack" of time.sleep() and a modulo-based conditional that was
ultimately non-deterministic and could result in log messages being dropped
or duplicated in certain scenarios.

In order to address these issues, this commit does the following:

* Carefully re-orders any code that sets a job status or timestamp
  to avoid any obvious race conditions.
* Removes the "dodgy hack" in favor of a much more straightforward
  (and less error-prone) method of collecting logs from the container.
* Removes arbitrary and unnecessary calls to time.sleep()

Before applying any changes, the flaky test was failing about 12% of the
time.  Putting a sleep() call between setting the `job_status` to SUCCEEDED
and collecting the logs, resulted in a 100% failure rate.  Simply moving
the code that sets the job status to SUCCEEDED to the end of the code block,
dropped the failure rate to ~2%.  Finally, removing the log collection
hack allowed the test suite to run ~1000 times without a single failure.

Taken in aggregate, these changes make the batch backend more deterministic
and should put the nail in the coffin of this flaky test.

Closes #3475
2020-11-18 10:49:25 +00:00
Oide Brett
83507fbc37
fixed issue in update_configuration for lambda when setting VPC config property (#3479) 2020-11-18 08:45:31 +00:00
Ayush Ghosh
7f73015f02
Fix XML encoding in Route53 JInja2 Templates #3469 (#3473)
* Use Jinja2 escape functionality to escape html attributes in value response

* Add tests

* fix formatting
2020-11-18 07:23:49 +00:00
Guillermo Arribas
62fd975da0
EventBridge: put_rule and list_rules should store and retrieve EventBusName property (#3472)
Co-authored-by: Guillermo Arribas <garribas@atlassian.com>
2020-11-17 15:36:17 +00:00
jweite
5fe921c2bc
Added support for EMR Security Configurations and Kerberos Attributes. (#3456)
* Added support for EMR Security Configurations and Kerberos Attributes.

* Revised exception-raising test to work with pytest api.

* Added htmlcov to .gitignore; upgrading botocore to 1.18.17, per commit d29475e.

Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-11-17 10:54:34 +00:00
Rich Unger
f045af7e0a
Add support for empty strings in non-key dynamo attributes (#3467)
* Add support for empty strings in non-key attributes

https://github.com/spulec/moto/issues/3339

* Nose, not pytest

* Revert "Nose, not pytest"

This reverts commit 5a3cf6c887dd9fafa49096c82cfa3a3b7f91d224.

* PUT is default action
2020-11-17 09:12:39 +00:00
Oide Brett
62d382ff70
Fixed issue 3448 for DynamoDB update_item (#3463)
* Fixed issue 3448 for DynamoDB update_item

* Tidied up fix for issue 3448 for DynamoDB update_item

* Reformatted fix for issue 3448 for DynamoDB update_item

* removed use of f-strings in test case as it fails in Travis CI build due to Python 2.7 support of f strings
2020-11-17 07:41:54 +00:00
Brian Pandola
d29475ed19
Fix: TagList missing in rds:DescribeDBInstance response (#3459)
Previously, tags were only available via rds:ListTagsForResource, but are now
included in the Create/DescribeDBInstance responses as of Botocore 1.18.17[1]

[1]: f29d23c53e (diff-d10722c0e11ded323c8d240066d7ed31e93a1e6423d54e091b7d54b86e6bd4e0)

Fixes #3458
2020-11-16 09:30:53 +00:00
Brian Pandola
93453eba05
Improve ec2:DescribeSubnets filtering (#3457)
* Add response/model/test coverage for filtering by `state`
* Add explicit test case for filtering by `vpc-id`

Closes #801
2020-11-16 08:17:36 +00:00
Steve Pulec
6107561717
Merge pull request #3451 from bpandola/fix-3450
Fix: Return `Tags` in iam:CreateUserResponse
2020-11-15 11:25:10 -06:00
gsamaras
d068653dea
dynamodb2 support for default Action ('Put') in update_item (#3454)
Co-authored-by: Georgios Samaras <gsamaras@amazon.com>
2020-11-14 11:10:38 +00:00
Bert Blommers
273ca63d59 Linting 2020-11-11 15:55:37 +00:00
Bert Blommers
cb6731f340 Convert fixtures/exceptions to Pytest 2020-11-11 15:54:01 +00:00
Matěj Cepl
6d364dc7aa Pytest model of exceptions uses 'value' attribute instead of 'exception'. 2020-11-11 12:21:04 +01:00
Matěj Cepl
2705698d83 Mark functions as requiring network 2020-11-10 14:12:38 +01:00
Matěj Cepl
5697ff87a8 Back to Black 2020-11-10 14:12:38 +01:00
Matěj Cepl
ea489bce6c Finish porting from nose to pytest. 2020-11-10 08:25:05 +01:00
Matěj Cepl
77dc60ea97 Port test suite from nose to pytest.
This just eliminates all errors on the tests collection. Elimination of
failures is left to the next commit.
2020-11-10 08:23:44 +01:00
Steve Pulec
e5adfd6f12
Merge pull request #3443 from bpandola/fix-2862
Add ssm:SendCommand support for instance tag Targets
2020-11-09 19:57:25 -06:00
Brian Pandola
b8e08539e3 Fix: Return Tags in iam:CreateUserResponse
Fixes #3450
2020-11-09 14:59:06 -08:00
Brian Pandola
2f70373c2e
Add CloudFormation Update support for AWS::StepFunctions::StateMachine (#3440)
Closes #3402
2020-11-08 12:44:23 +00:00
Brian Pandola
3b9635b3c7 Add ssm:SendCommand support for instance tag Targets
Replace the special-case code to handle Cloud Formation tags with a more
generic implementation that covers all instance tags.

Supersedes #2863
Closes #2862
2020-11-08 00:06:35 -08:00
pwrmiller
725ad7571d
Adds some basic endpoints for Amazon Forecast (#3434)
* Adding some basic endpoints for Amazon Forecast, including all dataset group related endpoints

* Adds better testing around exception handling in forecast endpoint, removes some unused code, and cleans up validation code

* Fix unused imports, optimize imports, code style fixes

Co-authored-by: Paul Miller <pwmiller@amazon.com>
2020-11-06 08:23:47 +00:00
Peter Lithammer
b7cf2d4478
ecr: Fix "imageDigest" value in ecr.list_images() response (#3436) 2020-11-05 14:10:23 +00:00
Brian Pandola
574f46e212
Implement additional Step Functions endpoints (#3437)
* Implement tagging/untagging for State Machine resources

* Implement `stepfunctions:UpdateStateMachine` endpoint
2020-11-05 10:16:48 +00:00
Brian Pandola
76265576ac
Fix: describe/list attribute discrepancy in Secrets Manager (#3432)
`secretsmanager:DescribeSecret` returns `VersionIdsToStages`
`secretsmanager:ListSecrets` returns the same information in `SecretVersionsToStages`

* Verified fix against real AWS backend.

Fixes #3406
2020-11-03 14:18:56 +00:00
Brian Pandola
f584e16ab9
Fix: eventName for a deleted record should be REMOVE instead of DELETE (#3431)
Verified API documentation[1] against the real AWS backend.

[1]: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_streams_Record.html#DDB-Type-streams_Record-eventName

Fixes #3400
2020-11-02 17:21:09 +00:00
usmangani1
f57a77451c
Fix:Added Tags for Network-ACL,RouteTable,InternetGateway (#3430)
* Fix:Added Tags for Network-ACL,RouteTable,InternetGateway

* Modified internet-gateway tags

* Lint

Co-authored-by: usmankb <usman@krazybee.com>
2020-11-02 13:30:02 +00:00
Brian Pandola
171130fe7b
Add support for CloudFormation resource AWS::StepFunctions::StateMachine (#3429)
Closes #3402

Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
2020-11-02 11:53:03 +00:00
usmangani1
53c3eb6240
Fix:SecretsManager :Error on Invalid secretID (#3413)
* Fix:SecretsManager :Error on Invalid secretID

* Fixed tests

Co-authored-by: usmankb <usman@krazybee.com>
2020-11-02 10:15:40 +00:00
usmangani1
9970be2309
Fix: Adding alarm arn to describe alarms response (#3409)
* Fix: adding alarm arn to describe alarms response

* Fix:Delete subscriptions on delete topic

* modified tests

Co-authored-by: usmankb <usman@krazybee.com>
2020-11-02 08:56:18 +00:00
Brian Pandola
68e3d394ab
Stepfunctions improvements (#3427)
* Implement filtering for stepfunctions:ListExecutions

* Add pagination to Step Functions endpoints

Implements a generalized approach to pagination via a decorator method for the
Step Functions endpoints.  Modeled on the real AWS backend behavior, `nextToken`
is a dictionary of pagination information encoded in an opaque string.

With just a bit of metadata hard-coded (`utils.PAGINATION_MODEL`), backend `list`
methods need only be decorated with `@paginate` and ensure that their returned
entities are sorted to get full pagination support without any duplicated code
polluting the model.

Closes #3137
2020-11-01 10:16:41 +00:00
Eoin Shanaghy
a3880c4c35
Metric data query alarms (#3419)
* Add support for metric data query alarms (Metrics=[..])

* Fix trailing whitespace

* Allow for unordered metrics in Python 2.7

* Add describe_alarm assertions and support DatapointsToAlarm
2020-10-31 15:56:24 +00:00
jweite
f8d2ce2e6a
Notebook Lifecycle Config create, describe and delete (#3417)
* Notebook Lifecycle Config create, describe and delete

* PR3417 comment changes: raise on create with duplicate name, derive a ValidationException class and use it instead of RESTException, unit test for delete non-existing.

Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-10-30 21:05:06 +00:00
jweite
cbd4efb42d
ApplicationAutoscaling: support autoscaling policies, deregister_scalable_target (#3350)
* ApplicationAutoscaling: support autoscaling policies, deregister_scalable_target.

* PR3350 comment changes: drop unnecessary pass statements, unit test three exception cases.

Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-10-30 15:21:34 +00:00
Erinna Chen
d499d4d179
Fix update for application autoscaling register target (#3423) 2020-10-29 14:18:38 +00:00
Brian Pandola
2f23f6b26b
Fix dynamodb:TransactGetItems response for items that do not exist (#3420)
If the requested item has no projected attributes, the corresponding ItemResponse
object is an empty Map.[1]  Verified against real AWS.

Fix existing general test case and add an explicit test case to cover this scenario.

[1]: https://docs.aws.amazon.com/amazondynamodb/latest/APIReference/API_TransactGetItems.html#API_TransactGetItems_ResponseElements

Fixes #3404
2020-10-29 11:50:45 +00:00
usmangani1
19fc76f466
Fix: SNS Delete subscriptions on topic deletion (#3410)
* Fix:Delete subscriptions on delete topic

* Changed tests

Co-authored-by: usmankb <usman@krazybee.com>
2020-10-29 08:52:02 +00:00
Bert Blommers
cc27f1ef0c S3 - Add more detail to error responses 2020-10-28 14:22:18 +00:00
Neal Granger
a5fc14b5bc
Add missing Fn::GetAtt attributes to S3 bucket mock (#3396)
* Add missing `Fn::GetAtt` attributes to S3 bucket mock

Addresses an issue reported here https://github.com/localstack/aws-cdk-local/issues/1

* Reformat touched files with `black`

* Reformat touched files with `black` on Python 3.7
2020-10-27 16:04:32 +00:00
Artem
53cc3dd67a
Fix SQS md5 attribute hashing. (#3403)
* Fix sqs md5 attribute hashing.

* Fix test name.

* Fix format.
2020-10-27 12:13:47 +00:00
Brian Pandola
c0a25bbd9a
Fix: VpnGatewayId parameter casing in ec2:CreateVpnConnection request (#3401)
Fixes #3397
2020-10-27 07:41:01 +00:00
usmangani1
14980371d7
FIX:Add secrets Manager Tag resource Funtionality (#3392)
* FIX:Add secrets Manager Tag resoruce Funtionality

* Fixed review comments

Co-authored-by: usmankb <usman@krazybee.com>
2020-10-22 11:14:32 +01:00
Brian Pandola
9eb58eea41
Fix: nextToken value in logs:DescribeLogGroups response (#3398)
The pagination for this endpoint has been modified to more closely
model the real AWS behavior:
* Log Groups are now sorted alphabetically by `logGroupName`.
* `nextToken` is now a string containing the last `logGroupName` in the
  current response.
* Specifying an invalid `nextToken` does not generate an error, but does
  return an empty group list.
* `nextToken` is not included in the response if there are no additional
  items to return.

Fixes #3395
2020-10-21 09:47:09 +01:00
usmangani1
fcc85a9645
Including Message attributes when ALL is passed (#3393)
* Including Message attributes when ALL is passes

* Added tests

Co-authored-by: usmankb <usman@krazybee.com>
2020-10-19 09:04:38 +01:00
davidaah
2fe3aee359
Allow creation of subnets from secondary VPC IPv4 CIDR blocks (#3391)
* allow subnets to be created from secondary vpc cidr block

* add additional test case for invalid cidr
2020-10-16 16:02:01 +01:00
Brian Pandola
28c1690fc2
Add Support for SNS Topic KmsMasterKeyId Attribute (#3389)
We do not do any validation of the `KmsMasterKeyId` attribute, and simply
store it as-as.  This mimics the behavior in AWS, where the key is not
validated until it is actually used (when publishing[1]).

[1]: https://docs.aws.amazon.com/sns/latest/dg/sns-server-side-encryption.html

Closes #3216
2020-10-16 12:30:07 +01:00
Brian Pandola
6505c893b8
Fix: S3 Bucket does not support attribute type Arn in Fn::GetAtt (#3388)
Fixes #3387
2020-10-16 11:29:26 +01:00
Brian Pandola
99556620a9
Fix: Empty sets not removed from item after UpdateExpression (#3386)
DynamoDB does not support empty sets. If the last item in a set is
deleted as part of an UpdateExpression, the attribute must be removed.

Ref: https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.NamingRulesDataTypes.html

Fixes #3296
2020-10-14 16:32:42 +01:00
Brian Pandola
ccda76898a
Add KMS Support to EBS Encrypted Volumes (#3383)
* Properly coerce `Encrypted` attribute to bool on request/response.
* Create and use a default AWS managed CMK for EBS when clients request
  an encrypted volume without specifying a KmsKeyId.

NOTE: A client-provided KmsKeyId is simply stored as-is, and is not validated
against the KMS backend. This is in keeping with other moto backends (RDS, Redshift)
that currently also accept unvalidated customer master key (CMK) parameters, but could
be an area for future improvement.

Closes #3248
2020-10-14 15:18:50 +01:00
Brian Pandola
ea19466c38
Fix missing properties when ecs:TaskDefinition created via CloudFormation (#3378)
There's a larger problem here that needs a more generalized solution,
but this solves the immediate issue with a minimum amount of code.

Closes #3171
2020-10-12 20:53:30 +01:00
nom3ad
fe361f861d
Enable more unit tests for ACM (#3372)
* Enable more unit tests for ACM

* put a smile on travis-ci face

Co-authored-by: nom3ad <nom3ad@gmail.com>
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
2020-10-12 14:43:36 +01:00
waynemetcalfe
0a938f7bb4
issue-3379 iam list_roles: implement PathPrefix, MaxItems and Marker (#3380)
* issue-3379 iam list_roles: implement PathPrefix, MaxItems and Marker

* issue-3379 fix cloudformation test
2020-10-12 12:13:20 +01:00
nom3ad
ea0ba91f63
support Tags parameter in ACM import_certificate() and request_certificate() methods (#3373)
* ACM: support `tags` parameter in import_certificate()

* ACM: support tags parameter in  request_certificate()

* ACM: better tag operations with more unit tests

Co-authored-by: nom3ad <nom3ad@gmail.com>
2020-10-12 07:55:14 +01:00
nom3ad
502818be4c
Fixes acm:describe_certificate for imported certificates having no SAN extension (#3370)
* Fix(test_acm): describe_certificate testcase does't execute

* handle acm certificates lacking extensions

Co-authored-by: nom3ad <nom3ad@gmail.com>
2020-10-10 22:32:42 +01:00
Brian Pandola
c54f182ca1
Implement additional filters for ecs.ListTasks (#3376)
Closes #1785
2020-10-10 20:02:08 +01:00
Brian Pandola
4a336b8b04
Add ec2.vpc resource support to Tagging API (#3375)
Closes #1849
2020-10-10 19:05:21 +01:00
Brian Pandola
a2bd4515eb
Add better support for SQS MaximumMessageSize attribute (#3374)
Closes #3205
2020-10-10 17:54:36 +01:00
Brian Pandola
db7842653f
iot:DeleteThingGroup should return success even for non-existent groups (#3367)
Closes #3026
2020-10-09 15:57:00 +01:00
Brian Pandola
c1b2c78db2
Fix TagFilter implementation in tag:GetResources (#3366)
The `tag_filter` method has been re-arranged to mimic the actual AWS behavior:
Return `True` if *any* tag matches a filter and *all* filters are matched.

Python's closures are late-binding, so we have to modify the lambdas accordingly!

Closes #2814
2020-10-09 13:55:48 +01:00
Brian Pandola
d00cefa25c
Add tagging to ec2.CopySnapshot (#3365)
The `@freeze_time` decorator was removed because it is not necessary (and was
causing the test to be skipped).

Closes #2940
2020-10-09 12:33:07 +01:00
Anton Grübel
349b9a990d
Add registeredAt to ecs container instance (#3358) 2020-10-05 15:39:59 +01:00
Sarang Joshi
08da2c92d9
fix(dynamodb2): Fix update_item nested insert (#3355)
When comparing old and new values when doing a nested item update, the
`!=` implementation fails when the value being compared is `None`. This
results in an exception when trying to insert a new item into a nested
map. So just do a quick check that the original value is exists before
doing the comparison, as the `None` default is what is tripping this.
2020-10-05 13:40:33 +01:00
Jon Michaelchuck
2391a4ab97
[SecretsManager] Handle missing secrets versions (#3349)
* SecretsManager - handle missing secrets versions

The get_secret_value method should raise ResourceNotFoundException
if a secret exists but the provided VersionId does not.

* Run black

* 2.x support

* black fix?

* secret is not a dict. Fix error msg output.
2020-10-05 12:22:54 +01:00
Roman Dmytrenko
c1a7f29c62
Fix issue with wrong parameter signed to secret_binary during secret rotation in Secrets Manager (#3348)
* fix issue with wrong parameter signed to secret_binary

* reformat test
2020-10-05 11:10:32 +01:00
usmangani1
a65c0f004c
Fix:SQS Receive Message (MessageAttributes) in response (#3303)
* Fix:SQS Receive Message (MessageAttributes) in response

* Fixed tests

Co-authored-by: usmankb <usman@krazybee.com>
2020-10-05 10:10:24 +01:00
jweite
9bc6bded6e
EMR: Support for StepConcurrencyLevel. (#3351)
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-10-02 14:07:13 +01:00
ljakimczuk
a668349a70
Add set_default_policy_version to the IAM backend (#3347)
* Adding set_default_policy_version

* Adding tests and reformatting

* Reformatting tests
2020-10-01 10:24:03 +01:00
Steve Pulec
7cead73aa1
Merge pull request #3341 from bblommers/bugfix/3302
SecretsManager: Make Secret ARN persistent
2020-09-30 16:02:32 -05:00
usmangani1
f9ce99f0d1
Fix:SQS md5 calculation for custom string data type. (#3346)
* Fix:SQS md5 calculation for custom string data type.

* Linting

Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-09-30 18:40:00 +01:00
usmangani1
1c3ba83fc2
Fix : SQS Create Queue with attributes (#3345)
* Considering only new attributes while queue creation

* Linting

Co-authored-by: usmankb <usman@krazybee.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-09-30 16:34:15 +01:00
jweite
1dd5cf08a8
Transcribe Medical Support (#3299)
* Transcribe first cut: Medical: start, get and delete jobs.

* Added list_medical_transcription_job() support to Transcribe.

* Support for medical vocabularies.

* Added transcribe to list of backends to fix server mode error.

* PR3299 requested changes: don't offer deprecated decorator, regionalize download_uri, create/use service-specific exceptions.

Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-09-30 13:18:26 +01:00
Giovanni Torres
195ba81c56
feat: update events remove_targets API response (#3340)
* feat: update events remove_targets API response

* test: add missing test for expected exception
2020-09-29 11:51:17 +01:00
Bert Blommers
369f6bbfc9 3302 - Make Secret ARN persistent 2020-09-28 14:51:30 +01:00
ljakimczuk
3bc18455a2
WIP: Introducing VPC Flow Logs (#3337)
* Start working on flow logs

* Change test

* Constructing tests

* Changing exceptions and adding more tests

* Adding more tests

* Changing model and adding more tests

* Adding support for tags

* Mocking Access error with non-existing Log Group Name

* Adding FlowLogAlreadyExists support

* Changing style

* Reformatted code

* Reformatted tests

* Removing needless test

* Adding support for CloudFormation

* Reformatting slightly

* Removing arnparse and using split

* Rearranging tests

* Fixing FilterNotImplementedError test

* Moving imports to 'if' clauses and adding explicit test for 'cloud-watch-logs' type

* Setting names matching boto3 API and restoring 'not-implementd-filter' test

* Reformatting tests with black
2020-09-28 07:16:06 +01:00
usmangani1
55e7caccfe
Fix:EC2 Tags in create vpc and create subnet (#3338)
Co-authored-by: usmankb <usman@krazybee.com>
2020-09-27 09:24:17 +01:00
Benjamin
82dbaadfc4
added organizations detach_policy response, model, and tests, issue #… (#3278)
* added organizations detach_policy response, model, and tests, issue #3239

Signed-off-by: Ben <ben.lipscomb@fmr.com>

* Created individual tests for detach_policy exceptions, updated regex statements for Root, OU, and Account Id
2020-09-25 16:55:29 +01:00
Wolfgang Bauer
a4701dbbe6
Add tags to Elastic IP Addresses (#3310)
* Make ElasticAddress a tagged resource

To be able to filter on tags on ElasticAddresses, I need to have tags.

* remove unneeded commented lines

Was beginning of how to to it before further checking how it is done with other resources.

* do not ignore network-interface-owner-id filter

* add TODO about currently hardcoded region

* remove hardcoding region

* add testing for tags

creating and allocation, adding tags and querying for it

* separate test for tags into own method

* Linting

Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-09-25 15:25:30 +01:00
Steve Pulec
ce60e9e3b8
Merge pull request #3314 from bblommers/dependency-test
List dependencies for services - add integration test
2020-09-23 07:07:31 -05:00
jweite
cd20668e9c
Support for autoscaling policies in run_jobflow, add_instance_group and list_instance_groups. (#3288)
Support for cluster_id parameter substitution in autoscaling policy cloudwatch alarm dimensions.
New operations put_autoscaling_policy and remove_autoscaling_policy support

Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-09-23 11:21:45 +01:00
Macwan Nevil
427a222aa0
feature added: support for api RolePermissionsBoundary (#3329)
* feature added: support for api PutUserPermissionsBoundary; DeleteRolePermissionsBoundary

* minor test fix

* lint fixed

* refractored test case

* Issue 3224 s3 copy glacier object (#3318)

* 3224 Enhancement - S3 Copy restored glacier objects

- adds setter for expiry date
- copy sets expiry date to none when source is glacier object
- throws error for copying glacier object only if not restored/still restoring

* 3224 Enhancement - S3 Copy restored glacier objects

- throws error for copying deep archive object only if not restored/still restoring

* Fix:s3 List Object response:delimiter  (#3254)

* Fix:s3 List Object delimiter in response

* fixed tests

* fixed failed tests

Co-authored-by: usmankb <usman@krazybee.com>

* feature added: support for api PutUserPermissionsBoundary; DeleteRolePermissionsBoundary

* minor test fix

* lint fixed

* refractored test case

* added test case for put role exception

Co-authored-by: ruthbovell <63656505+ruthbovell@users.noreply.github.com>
Co-authored-by: usmangani1 <sgosman_chem@yahoo.com>
Co-authored-by: usmankb <usman@krazybee.com>
2020-09-22 12:43:59 +01:00
Ben Dennerley
958e95cf5c
Make IoT certificate ID generation deterministic and prevent duplicate certificates from being created (#3331)
* Make IoT certificate id generation deterministic

Fixes #3321

As per https://stackoverflow.com/questions/55847788/how-does-aws-iot-generate-a-certificate-id,
the IoT certificate ID is the SHA256 fingerprint of the
certificate. Since moto doesn't generate full certificates we will
instead use the SHA256 hash of the passed certificate pem.

* Don't allow duplicate IoT certificates to be created

Fixes #3320

When using boto3, trying to register a certificate that already
exists will throw a ResourceAlreadyExistsException. Moto should
follow the same pattern to allow testing error handling code in
this area.
2020-09-22 10:28:12 +01:00
Nick Stocchero
fd69c93a09 use botocore regions and refactor sorting 2020-09-21 17:42:44 -06:00
Nick Stocchero
8d5c70a924 different aggregation strategy 2020-09-21 17:42:44 -06:00
Nick Stocchero
d8cea0213d straighten out filter logic 2020-09-21 17:42:44 -06:00
Nick Stocchero
8dd90db83c add missing test for policies 2020-09-21 17:42:44 -06:00
Nick Stocchero
ff84b63484 address PR comments 2020-09-21 17:42:44 -06:00
Nick Stocchero
7bc5b5c08f Add IAM Role and Policy to Config 2020-09-21 17:42:44 -06:00
ayushbhawsar
2e0e542efe
added cognito idp function admin_set_user_password to the code (#3328)
* added cognito idp function  to the code

* fixed linting issues
2020-09-21 18:40:07 +01:00
usmangani1
0b9903a3df
Fix:s3 List Object response:delimiter (#3254)
* Fix:s3 List Object delimiter in response

* fixed tests

* fixed failed tests

Co-authored-by: usmankb <usman@krazybee.com>
2020-09-21 16:21:18 +01:00
ruthbovell
1022aa0968
Issue 3224 s3 copy glacier object (#3318)
* 3224 Enhancement - S3 Copy restored glacier objects

- adds setter for expiry date
- copy sets expiry date to none when source is glacier object
- throws error for copying glacier object only if not restored/still restoring

* 3224 Enhancement - S3 Copy restored glacier objects

- throws error for copying deep archive object only if not restored/still restoring
2020-09-21 07:37:50 +01:00
jweite
da4de072a9
ApplicationAutoscaling: support for all the current various forms of resource_id (#3305)
* Change to test_s3 method test_presigned_url_restrict_parameters to tolerate change in exception messages, spurred by boto3 1.14.59 release.

* ApplicationAutoscaling: support for all the current various forms of resource_id.

* Factored logic for extracting application autoscaling resource_type from resource_id to separate function, per PR3304 comment.

Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-09-19 20:13:44 +01:00
Guy Moses
7ce1e87477
[fix] cognito-idp list_users Filter arg now support spaces (#3317) 2020-09-19 11:26:01 +01:00
Leo Sutic
cc0bd5213f
Enable CORS from everywhere using flask-cors. (#3316)
Co-authored-by: Leo Sutic <leo.sutic@matterport.com>
2020-09-19 10:07:17 +01:00
Leo Sutic
94543f6e48
Include response headers when deleting objects. (#3313)
* Return delete meta.

* Add tests.

* Lint fixes.

Co-authored-by: Leo Sutic <leo.sutic@matterport.com>
2020-09-15 13:29:09 +01:00
Bert Blommers
0ab21f62a8 Linting 2020-09-13 19:42:38 +01:00
Bert Blommers
db1d7123f6 List dependencies for services - add integration test to verify 2020-09-13 16:08:23 +01:00
jweite
9f0f230d13
Change to test_s3 method test_presigned_url_restrict_parameters to tolerate change in exception messages, spurred by boto3 1.14.59 release. (#3308)
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-09-11 11:17:36 +01:00
Arcadiy Ivanov
c2d1ce2c14
Add If-Match, If-None-Match and If-Unmodified-Since to S3 GET/HEAD (#3021)
fixes #2705
2020-09-11 10:17:39 +01:00
Karthikeyan Singaravelan
7054143701
Fix deprecation warnings due to invalid escape sequences. (#3273)
* Fix deprecation warnings due to invalid escape sequences.

* Fix linter error.
2020-09-10 09:20:26 +01:00
Daniel Okey-Okoro
c321ad46b0
fix: deleting non-existent topic shouldn't raise KeyError (#3285) 2020-09-10 08:32:41 +01:00
Steve Pulec
c3b8b2343b
Merge pull request #3286 from ciaranevans/add-input-validation
Add Step Function Execution Input validation
2020-09-04 19:48:05 -05:00
Toshiya Kawasaki
c66812edba
Add kinesisvideo archived media (#3280)
* add get_hls_streaming_session_url

* add get_dash_streaming_session_url

* add get_clip

* add test_server for kinesisvideo archived media

* fix for lint

* fix for lint

* avoid testing kinesisvideoarchivedmedia with TEST_SERVER_MODE=true
2020-09-04 12:14:48 +01:00
Ciaran Evans
6a467ec50f Add validation on execution input 2020-09-04 10:22:21 +01:00
Ciaran Evans
ca64d8fc7a
Implement Execution inputs for Step Functions (#3284)
* Add input attribute to Execution and test with describe_execution

* Switch back method name
2020-09-04 09:58:16 +01:00
zhil3
8854fd06e8
Add describe_endpoint and register_certificate_without_ca in iot_mock module with unittest (#3279)
Co-authored-by: Zhi Li <zhi_li@trendmicro.com>
2020-09-04 09:11:17 +01:00
usmangani1
d2e16ecc2e
Fix:s3 Presign Post with object acl (#3246)
* Fix:s3 Presign Post with object acl

* Added import in tests

Co-authored-by: usmankb <usman@krazybee.com>
2020-09-02 18:35:53 +01:00
Karthikeyan Singaravelan
3fb7cf75d4
Fix deprecation warning due to base64.decodestring in Python 3. (#3272) 2020-09-02 11:10:29 +01:00
Toshiya Kawasaki
25161c0c18
Add kinesisvideo (#3271)
* kinesisvideo create_stream

* add kinesis video stream description

* add kinesisvideo describe_stream

* add kinesisvideo list_streams

* add kinesisvideo delete_stream

* remove unused comment

* remove duplicated definition

* add kinesis video exceptions

* pass region_name to kinesisvideo client in test

* fix kinesisvideo url path

* resolve conflict of kinesisvideo url and kinesis url

* specify region name to kinesisvideobackend

* Add get-dataendpoint to kinesisvideo

* include stream name in ResourceInUseException of kinesisvideo

* use ACCOUNT_ID from moto.core in kinesisvideo

* add server test for kinesisvideo

* split up kinesisvideo test
2020-09-02 08:51:51 +01:00
usmangani1
00a5641cb9
Fix:s3 Presign Put Request with File upload (#3235)
* Fix:s3 Presign Put Request with File upload

* Added imports in test

Co-authored-by: usmankb <usman@krazybee.com>
2020-09-02 07:10:56 +01:00
usmangani1
3ea46617d9
Fix:sqs get-queue-attributes response template (#3255)
* Fix:sqs get-queue-attributes response template

* Fix:removed debug statements

* Modified the template

* "fixed build issues"

* Linting

Co-authored-by: usmankb <usman@krazybee.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-09-01 18:05:25 +01:00
Peter Baumgartner
94c676b9cf
include=["TAGS"] for describe_task_definition (#3265)
* include=["TAGS"] for describe_task_definition

* Different approach

* describe_services tags and tests
2020-09-01 10:24:08 +01:00
xsphrx
236ab59afe
added cognito-idp initiate_auth and PASSWORD_VERIFIER challenge to respond_to_auth_challenge (#3260)
* added cognito-idp initiate_auth and PASSWORD_VERIFIER challenge to respond_to_auth_challenge

* fixed for python2

* added mfa, REFRESH_TOKEN to initiate_auth, SOFTWARE_TOKEN_MFA to respond_to_auth_challenge

* added negative tests

* test
2020-09-01 09:20:31 +01:00
usmangani1
bcf61d0b09
Fix: Api-Gateway ApiKeyAlreadyExists headers change. (#3162)
* Fix: Api-Gateway ApiKeyAlreadyExists headers change.

* Added test for non decorator

* Fixed cli errors

* Fix:fixed build errors

* Fix: assert only in case of non server mode

Co-authored-by: usmankb <usman@krazybee.com>
2020-09-01 08:25:59 +01:00
usmangani1
0a89f9d1df
Fix:SQS:Added Non existent queue name in ERROR RESPONSE (#3261)
* Fix:SQS:Added Non existent queue name in ERROR RESPONSE

* Linting

Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-08-27 17:31:20 +01:00
usmangani1
1c939a5f06
Fix:EC2-Create-Subnet availability Zone Id support (#3198)
* Fix:EC2-Create-Subnet availability Zone Id support

* Linting

* Fix:fixed build errors

* linting

Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
Co-authored-by: usmankb <usman@krazybee.com>
2020-08-27 16:31:39 +01:00
jweite
49d92861c0
Iam cloudformation update, singificant cloudformation refactoring (#3218)
* IAM User Cloudformation Enhancements: update, delete, getatt.

* AWS::IAM::Policy Support

* Added unit tests for AWS:IAM:Policy for roles and groups.  Fixed bug related to groups.

* AWS:IAM:AccessKey CloudFormation support.

* Refactor of CloudFormation parsing.py methods to simplify and standardize how they call to the models.  Adjusted some models accordingly.

* Further model CloudFormation support changes to align with revised CloudFormation logic.  Mostly avoidance of getting resoure name from properties.

* Support for Kinesis Stream RetentionPeriodHours param.

* Kinesis Stream Cloudformation Tag Support.

* Added omitted 'region' param to boto3.client() calls in new tests.

Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-08-27 10:11:47 +01:00
Ciaran Evans
3b06ce689e
Address SFN.Client.exceptions.ExecutionAlreadyExists Not implemented (#3263)
* Add check for existing execution, fix issue with make init

* Remove f-string usage

* Remove fstring usage in test

* Pin black and run formatting on test_stepfunction

* Reverse changes made by black 20.8b1
2020-08-27 08:22:44 +01:00
Kevin Frommelt
55b02c6ee9
Add support for Launch Templates in Auto Scaling Groups (#3236)
* Add support for Launch Templates in Auto Scaling Groups

* Use named parameters, simplify parameter validation
2020-08-26 15:15:07 +01:00
jmbollard
2a27e457bf
Security group egress ip permissions fix (#3250)
* Add support for Description in egress rule response

* Update SecurityGroup default egress rule ip range

* Remove extra commas

* Remove extra commas

* Lower docker package in Travis

* Add more lambda vars per PR 3247

* Remove code added in 3247

* Add tests for egress rules with Descriptions

* Reformat based on black

Co-authored-by: spillin <jmbollard@me.com>
2020-08-26 14:27:45 +01:00
Guilherme Martins Crocetti
f744356da7
Lambda reserved concurrency (#3215)
* lambda-responses: add method to dispatch concurrency calls

* lambda-resources: add route to handle concurrency requests

* lambda-model: implement put_function_concurrency and concurrency attribute

* put-concurrency-tests: add one simple test

* get_function: add concurrency entry - with test

* lambda-reserved-concurrency: cloudformation support

* lambda-concurrency: implement delete_reserved with tests

* lambda-concurrency: implement get_reserved with tests

* lint

* implementation-cov: mark delete_function_concurrency, put_function_concurrency and get_function_concurrency

* botocore doesn't display concurrency entry for lambdas without it

* lambda(refactor): improvements on response's handler
2020-08-26 11:06:53 +01:00
Kamil Mańkowski
47a227921d
SES: Fix sending email when use verify_email_address (#3242) 2020-08-25 13:51:58 +01:00
Kamil Mańkowski
8a551a9754
[SNS] Mock sending directly SMS (#3253)
* [SNS] Mock sending directly SMS

Proper behaviour when publishing to PhoneNumber is sending
message directly to this number, without any topic or
previous confirmation.

https://docs.aws.amazon.com/sns/latest/dg/sns-mobile-phone-number-as-subscriber.html

* Fix arguments order

* Omit checking local backend when tests in server mode
2020-08-25 13:05:49 +01:00
usmangani1
9894e1785a
Enhancement : Ec2 - Add describe-vpc-endpoint-services method support. (#3108)
* Enhancement : Ec2 - Add describe-vpc-endpoint-services method support.

* Fix:EC2-describe_vpc_endPoint_services changed the template

* Fixed comments

* Linting

Co-authored-by: usmankb <usman@krazybee.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-08-06 06:26:44 +01:00
usmangani1
a7ddcd7da3
Fix:EC2-authorize_security_group_ingress- add description to IP-Ranges (#3196)
* Fix:EC2-authorize_security_group_ingress- add description to IP-Ranges

* Fix:EC2-authorize_security_group_ingress- add test when description is not present.

* part commit

* Fix:fixed build errors

* Linting

* Allow for Python2 string/unicodes

Co-authored-by: usmankb <usman@krazybee.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-08-04 06:50:57 +01:00
jweite
da07adae52
* Support for CloudFormation update and delete of Kinesis Streams (#3212)
* Support for CloudFormation stack resource deletion via backend resource method
  delete_from_cloudformation_json() via parse_and_delete_resource().
* Correction to the inappropriate inclusion of EndingSequenceNumber in open shards.
  This attribute should only appear in closed shards.  This regretfully prevents
  confirmation of consistent record counts after split/merge in unit tests.
* Added parameters/decorator to CloudFormationModel method declarations to calm-down Pycharm.

Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
2020-08-03 16:04:05 +01:00
Yuuki Takahashi
99736c3101
fix clear pending messages when call purge_queue (#3208) 2020-08-03 15:09:25 +01:00
Ninh Khong
061c609a8f
Fix secretsmanager random password wrong length (#3213)
* Enhance function get_parameter by parameter name, version or labels

* Fix random password with exclude characters return wrong length
2020-08-03 13:42:42 +01:00
Anton Grübel
252d679b27
Organizations - implement Policy Type functionality (#3207)
* Add organizations.enable_policy_type

* Add organizations.disable_policy_type

* Add support for AISERVICES_OPT_OUT_POLICY
2020-08-02 10:56:19 +01:00
jweite
3342d49a43
S3 cloudformation update (#3199)
* First cut of S3 Cloudformation Update support: encryption property.

* Update type support for S3.  Abstract base class for CloudFormation-aware models, as designed by @bblommers, introduced to decentralize CloudFormation resource and name property values to model objects.

* Blackened...

* Un-renamed param in s3.models.update_from_cloudformation_json() and its call to stay compatible with other modules.

Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
2020-08-01 22:43:03 +01:00
Larry Aiello
06ed67a8e5
Implement UserIds for Snapshot Attributes (#3192)
* implement register_image

* format code

* add user_ids to snapshot model

* implement register_image

* format code

* add user_ids to snapshot model

* trying to un-deprecate tests

* Write tests and finalize implementation

* Add region parameter to boto3 resource call

* fixed test error
2020-08-01 17:03:54 +01:00