* refactor(ssm): Add Documents class to avoid dictionary handling
This also solves the datetime format issue in TestAccAWSCloudWatchEventTarget_ssmDocument
* **Fix bug.** If a cloudformation stack is updated with a new
parameter, that parameter should be honored. Several unit tests
had bugs where they were not providing parameters required by the template.
* **Fix bug.** Do not update stack parameters until after deleting removed
resources, so that any references to removed parameters can be resolved.
* **Fix bug.** Per the API, creation of a change set should not modify
a stack. The `diff` method, called in the creation of a
FakeChangeSet, was mutating the resource map which was problematic
* Update cloudwatch.put_metric_alarm to accept TreatMissingData and Tags parameter
* Add parameter ExtendedStatistic and EvaluateLowSampleCountPercentile to cloudwatch.put_metric_alarm
* Add parameter ThresholdMetricId to cloudwatch.put_metric_alarm
This fixes#4141, and pave ways for future changes around changeset.
We had subclassed FakeChangeSet from FakeStack, not from BaseModel. This made
us easier to send the response for describe_change_set calls, but when we are
handling the details of change set, the old approach won't work at all.
For example, when we were creating a changeset, we were actually creating a
stack without registering it (self.stacks), and future update onto this stack
is not really possible.
Signed-off-by: Kai Xia <kaix+github@fastmail.com>
* fix OPTIONS requests on non-existing API GW integrations
* add cloudformation models for API Gateway deployments
* bump version
* add backdoor to return CloudWatch metrics
* Updating implementation coverage
* Updating implementation coverage
* add cloudformation models for API Gateway deployments
* Updating implementation coverage
* Updating implementation coverage
* Implemented get-caller-identity returning real data depending on the access key used.
* bump version
* minor fixes
* fix Number data_type for SQS message attribute
* fix handling of encoding errors
* bump version
* make CF stack queryable before starting to initialize its resources
* bump version
* fix integration_method for API GW method integrations
* fix undefined status in CF FakeStack
* Fix apigateway issues with terraform v0.12.21
* resource_methods -> add handle for "DELETE" method
* integrations -> fix issue that "httpMethod" wasn't included in body request (this value was set as the value from refer method resource)
* bump version
* Fix setting http method for API gateway integrations (#6)
* bump version
* remove duplicate methods
* add storage class to S3 Key when completing multipart upload (#7)
* fix SQS performance issues; bump version
* add pagination to SecretsManager list-secrets (#9)
* fix default parameter groups in RDS
* fix adding S3 metadata headers with names containing dots (#13)
* Updating implementation coverage
* Updating implementation coverage
* add cloudformation models for API Gateway deployments
* Updating implementation coverage
* Updating implementation coverage
* Implemented get-caller-identity returning real data depending on the access key used.
* make CF stack queryable before starting to initialize its resources
* bump version
* remove duplicate methods
* fix adding S3 metadata headers with names containing dots (#13)
* Update amis.json to support EKS AMI mocks (#15)
* fix PascalCase for boolean value in ListMultipartUploads response (#17); fix _get_multi_param to parse nested list/dict query params
* determine non-zero container exit code in Batch API
* support filtering by dimensions in CW get_metric_statistics
* fix storing attributes for ELBv2 Route entities; API GW refactorings for TF tests
* add missing fields for API GW resources
* fix error messages for Route53 (TF-compat)
* various fixes for IAM resources (tf-compat)
* minor fixes for API GW models (tf-compat)
* minor fixes for API GW responses (tf-compat)
* add s3 exception for bucket notification filter rule validation
* change the way RESTErrors generate the response body and content-type header
* fix lint errors and disable "black" syntax enforcement
* remove return type hint in RESTError.get_body
* add RESTError XML template for IAM exceptions
* add support for API GW minimumCompressionSize
* fix casing getting PrivateDnsEnabled API GW attribute
* minor fixes for error responses
* fix escaping special chars for IAM role descriptions (tf-compat)
* minor fixes and tagging support for API GW and ELB v2 (tf-compat)
* Merge branch 'master' into localstack
* add "AlarmRule" attribute to enable support for composite CloudWatch metrics
* fix recursive parsing of complex/nested query params
* bump version
* add API to delete S3 website configurations (#18)
* use dict copy to allow parallelism and avoid concurrent modification exceptions in S3
* fix precondition check for etags in S3 (#19)
* minor fix for user filtering in Cognito
* fix API Gateway error response; avoid returning empty response templates (tf-compat)
* support tags and tracingEnabled attribute for API GW stages
* fix boolean value in S3 encryption response (#20)
* fix connection arn structure
* fix api destination arn structure
* black format
* release 2.0.3.37
* fix s3 exception tests
see botocore/parsers.py:1002 where RequestId is removed from parsed
* remove python 2 from build action
* add test failure annotations in build action
* fix events test arn comparisons
* fix s3 encryption response test
* return default value "0" if EC2 availableIpAddressCount is empty
* fix extracting SecurityGroupIds for EC2 VPC endpoints
* support deleting/updating API Gateway DomainNames
* fix(events): Return empty string instead of null when no pattern is specified in EventPattern (tf-compat) (#22)
* fix logic and revert CF changes to get tests running again (#21)
* add support for EC2 customer gateway API (#25)
* add support for EC2 Transit Gateway APIs (#24)
* feat(logs): add `kmsKeyId` into `LogGroup` entity (#23)
* minor change in ELBv2 logic to fix tests
* feat(events): add APIs to describe and delete CloudWatch Events connections (#26)
* add support for EC2 transit gateway route tables (#27)
* pass transit gateway route table ID in Describe API, minor refactoring (#29)
* add support for EC2 Transit Gateway Routes (#28)
* fix region on ACM certificate import (#31)
* add support for EC2 transit gateway attachments (#30)
* add support for EC2 Transit Gateway VPN attachments (#32)
* fix account ID for logs API
* add support for DeleteOrganization API
* feat(events): store raw filter representation for CloudWatch events patterns (tf-compat) (#36)
* feat(events): add support to describe/update/delete CloudWatch API destinations (#35)
* add Cognito UpdateIdentityPool, CW Logs PutResourcePolicy
* feat(events): add support for tags in EventBus API (#38)
* fix parameter validation for Batch compute environments (tf-compat)
* revert merge conflicts in IMPLEMENTATION_COVERAGE.md
* format code using black
* restore original README; re-enable and fix CloudFormation tests
* restore tests and old logic for CF stack parameters from SSM
* parameterize RequestId/RequestID in response messages and revert related test changes
* undo LocalStack-specific adaptations
* minor fix
* Update CodeCov config to reflect removal of Py2
* undo change related to CW metric filtering; add additional test for CW metric statistics with dimensions
* Terraform - Extend whitelist of running tests
Co-authored-by: acsbendi <acsbendi28@gmail.com>
Co-authored-by: Phan Duong <duongpv@outlook.com>
Co-authored-by: Thomas Rausch <thomas@thrau.at>
Co-authored-by: Macwan Nevil <macnev2013@gmail.com>
Co-authored-by: Dominik Schubert <dominik.schubert91@gmail.com>
Co-authored-by: Gonzalo Saad <saad.gonzalo.ale@gmail.com>
Co-authored-by: Mohit Alonja <monty16597@users.noreply.github.com>
Co-authored-by: Miguel Gagliardo <migag9@gmail.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* ELBv2 - ListenerRule condition validation
- ListenerRule condition model now uses upper case field names that
match input params for boto and CloudFormation.
- BaseResponse._get_params() introduced to make it easier to deal
with the querystring input params.
* - introduce environment variable for DEFAULT_KEY_BUFFER_SIZE
* - prefix env variable with MOTO_S3 to avoid env variable conflicts
* - reduce the DEFAULT_KEY_BUFFER_SIZE to be less than the S3_UPLOAD_PART_MIN_SIZE to prevent in memory caching of multi part uploads
* - black formatting
* - fix formatting
* - fix missing import
* If token and attributes are the same, return endpoint
* fix black
* moto sns platform_endpoint.attributes includes only token,enabled
* add tests when calling sns#create_platform_endpoint with same attrs for #4056
* Add delete container and list tags endpoints to MediaStore
* Black reformat
* Fixed Lint problems
* Check if dictionary was deleted effectively
* lint fix
* MediaPackageClientError
* Lint Fix
* Test unknown channel describe
* Concatenation Fix
* MediaPackage - fix error message
* MediaPackage ClientError part2
* Mediastoredata not working
Base url
tests and renaming
typo
List Items not returning proper JSON and wrongly hitting get_object response
MediaStore2
Tests
* More implementation
* Fix tests and format
* Comments fix
* Comments 2
* MediastoreData - alternative logic to figure out appropriate host
Co-authored-by: av <arcovoltaico@gmail.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* Add outputs and vpc interfaces to a mediaconnect flow
* Add negative tests for add_flow_outputs and add_flow_vpc_interfaces
* fix: fstring to format
* MediaConnect - add appropriate URLs for ServerMode tests
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* Update implementation coverage
* EC2 - Update instance type offerings
* IAM - update list of managed policies
* Changelog for release 2.0.9
* Instance Type Offerings - fix number of available offerings
* Add ssm parsing support for cloudformation stacks
* Start adding unit tests for ssm parameter parsing
* Add tests for code update
* Add tests to parse ssm parameters code
* Fix black lint errors
* Fix bug.
* Need to specify region_name
* region needs to be same
* Use ssm_backends[region] instead of ssm_backend
* StringList -> string
* Linting
* check if servermode tests are on
* Typo
* Added support for ListenerRule. Will remove cruft
* Pushing latest
* Something works
* Put back ripped out code
* Save point. Incase I need more validations
* Revert "Save point. Incase I need more validations"
This reverts commit dac4953335dd9335eddb7a91a63667bc3c17104c.
* Fixed validations and some refactor
* Fix formatting
* Linting
* Cannot refactor if I have to fix all tests
* Remove exceptions for now. Will do in another PR
* Remove validations. Will add in next PR
* Fix broken tests. Almost.:
* Fix all tests. Some sneaky for now.
* Python2 making me write bad code
* OrderedDict.move_to_end() does not work in python2
* Linting
* Add more checks to field in conditions later.
* Unwnated change in FakeListener
* Revert "Unwnated change in FakeListener"
This reverts commit 962c2fdfd76fce999de9feccf1dd1c3ec48c459f.
* Add back default listener rule
* Linting fix
* Fix priority sorting
* Add cloudformation test for edge case
* Add validation for ForwardConfig in Action of ListernRule CF
* use not in
* set the priority template correctly
* Check for boolean in condition
* One more check
* Implement update_from_cloudformation_json for Listener and ListenerRule
* Unwanted spaces
* Linting issues
* Add tests for code coverage
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* Add support for DynamoDB Backup/Restore
Basic support for the following endpoints has been implemented with full test coverage:
- create_backup
- delete_backup
- describe_backup
- list_backups
- restore_table_from_backup
Behavior and error messages verified against a real AWS backend.
* Refactor test based on PR feedback
* Add ssm parsing support for cloudformation stacks
* Start adding unit tests for ssm parameter parsing
* Add tests for code update
* Add tests to parse ssm parameters code
* Fix black lint errors
* Fix bug.
* Need to specify region_name
* region needs to be same
* Use ssm_backends[region] instead of ssm_backend
* StringList -> string
* Linting
* check if servermode tests are on
* Typo
* Added support for ListenerRule. Will remove cruft
* Pushing latest
* Something works
* Put back ripped out code
* Save point. Incase I need more validations
* Revert "Save point. Incase I need more validations"
This reverts commit dac4953335dd9335eddb7a91a63667bc3c17104c.
* Fixed validations and some refactor
* Fix formatting
* Linting
* Cannot refactor if I have to fix all tests
* Remove exceptions for now. Will do in another PR
* Remove validations. Will add in next PR
* Fix broken tests. Almost.:
* Fix all tests. Some sneaky for now.
* Python2 making me write bad code
* OrderedDict.move_to_end() does not work in python2
* Linting
* Add more checks to field in conditions later.
* Unwnated change in FakeListener
* Revert "Unwnated change in FakeListener"
This reverts commit 962c2fdfd76fce999de9feccf1dd1c3ec48c459f.
* Add back default listener rule
* Linting fix
* Fix priority sorting
* Add cloudformation test for edge case
* Add validation for ForwardConfig in Action of ListernRule CF
* use not in
* set the priority template correctly
* Check for boolean in condition
* One more check
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* - Adding checking for resource type in tag functions
- Adding TargetNotFoundException when no resource found
- Adding support for tags for root OU, OU and Policies
- Adding tests covering the new code
- Adding test for deletion of a tag
* fixed linting issue
* - renamed helper function to a more logical name
- added tests for helper function
- fixed bugs in tests for tag functions
Co-authored-by: Sjoerd Tromp <stromp@schubergphilis.com>
* Add ssm parsing support for cloudformation stacks
* Start adding unit tests for ssm parameter parsing
* Add tests for code update
* Add tests to parse ssm parameters code
* Fix black lint errors
* Fix bug.
* Need to specify region_name
* region needs to be same
* Use ssm_backends[region] instead of ssm_backend
* StringList -> string
* Linting
* check if servermode tests are on
* Typo
* Added support for ListenerRule. Will remove cruft
* Pushing latest
* Something works
* Put back ripped out code
* Save point. Incase I need more validations
* Revert "Save point. Incase I need more validations"
This reverts commit dac4953335dd9335eddb7a91a63667bc3c17104c.
* Fixed validations and some refactor
* Fix formatting
* Linting
* Cannot refactor if I have to fix all tests
* Remove exceptions for now. Will do in another PR
* Remove validations. Will add in next PR
* Fix broken tests. Almost.:
* Fix all tests. Some sneaky for now.
* Python2 making me write bad code
* OrderedDict.move_to_end() does not work in python2
* Linting
* Add more checks to field in conditions later.
* Unwnated change in FakeListener
* Revert "Unwnated change in FakeListener"
This reverts commit 962c2fdfd76fce999de9feccf1dd1c3ec48c459f.
* Add back default listener rule
* Linting fix
* Fix priority sorting
* Add cloudformation test for edge case
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* Including labels and versions in SSM Get Parameters
* implementing NextToken and MaxResults into the SSM Get Parameter History functionality
* Implementing unit tests and some lint refactoring for NextToken implementation in get_parameter_history
* Implement correct error when requesting specific version of a parameter which exists but does not have this version
* removing trailing whitespace causing lint failure
* Adding unit tests and fixing linting for new error handling
* Fixing small bug in response message
* Revert change in get_parameters as versioning is not currently implemented in this method. Will fix as a separate PR
* Check exit status of container
* Added support for job dependencies
* batch container overrides
* add AWS_BATCH_JOB_ID to container env variables
* lint with black
* refactor batch dependency test
* refactor batch dependency test
* fix index
Co-authored-by: jterry64 <justin.terry@wri.org>
Co-authored-by: Daniel Mannarino <daniel.mannarino@gmail.com>
* basic implementation of update rest api
* basic implementation of update rest api
* basic implementation of update rest api
* review comments from bblommers
Co-authored-by: rajinder saini <rajinder.saini@c02vt5k2htd6.corp.climate.com>
* Add delete container and list tags endpoints to MediaStore
* Black reformat
* Fixed Lint problems
* Check if dictionary was deleted effectively
* lint fix
Co-authored-by: av <arcovoltaico@gmail.com>
* Add ssm parsing support for cloudformation stacks
* Start adding unit tests for ssm parameter parsing
* Add tests for code update
* Add tests to parse ssm parameters code
* Fix black lint errors
* Fix bug.
* Need to specify region_name
* region needs to be same
* Use ssm_backends[region] instead of ssm_backend
* StringList -> string
* Linting
* check if servermode tests are on
* Typo
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* Fix:SES-Added params check in template
* Added more tests and handled message
* linting
* fixed tests
* fix attribute name in message
* fix logic for exception
* Fix sqs message retention logic
* Apply lint to moto/sqs/models.py
* Fix failed tests because of freezing time
* Fix freezing time in test_publish_to_sqs_in_different_region
* Dont fail if CodeCov fails - for now
* CI - Force cache rebuild
* Bump werkzeug to latest version
* CI - Enforce cache flush
* ManagedBlockchain - fix error format
* ManagedBlockchain - Fix tests to use pytest.raises paradigm
* Revert "Lock Flask (#3925)"
This reverts commit 8bb0feb956.
* CI - Enforce cache rebuild
* Support rotating secrets using Lambda
The Secrets manager rotation process uses an AWS Lambda function
to perform the rotation of a secret. [1]
In fact, it's not possible to trigger rotation of a Secret
without specifying a Lambda function at some point in the life
of the secret:
```
$ aws secretsmanager rotate-secret --secret-id /rotationTest
An error occurred (InvalidRequestException) when calling the RotateSecret operation: No Lambda rotation function ARN is associated with this secret.
```
`moto` can be a little more lenient in this regard and allow
`rotate_secret` to be called without a Lambda function being
present, if only to allow simulation of the `AWSCURRENT` and
`AWSPREVIOUS` labels moving across versions.
However, if a lambda function _has_ been specified when calling
`rotate_secret`, it should be invoked therefore providing the
developer with the full multi-stage process [3] which can be
used to test the Lambda function itself and ensuring that full
end-to-end testing is performed. Without this there's no easy
way to configure the Secret in the state needed to provide the
Lambda function with the data in the format it needs to be in
at each step of the invocation process.
[1]: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-lambda-function-overview.html
[2]: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.rotate_secret
[3]: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-lambda-function-overview.html#rotation-explanation-of-steps
* Run `black` over `secretsmanager/models.py`
* Make `lambda_backends` import local to the condition
* Implement `update_secret_version_stage`
Allow a staging label to be moved across versions.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.update_secret_version_stage
* Add an integration test for Secrets Manager & Lambda
* Support passing `ClientRequestToken` to `put_secret_value`
By passing `ClientRequestToken` to `put_secret_value` within
the lambda function invoked by calling `rotate_secret`, one
can update the value associated with the existing (pending)
version, without causing a new secret version to be created.
* Add application logic for `AWSPENDING`
The rotation function must end with the versions of the secret
in one of two states:
- The `AWSPENDING` and `AWSCURRENT` staging labels are
attached to the same version of the secret, or
- The `AWSPENDING` staging label is not attached to any
version of the secret.
If the `AWSPENDING` staging label is present but not attached
to the same version as `AWSCURRENT` then any later invocation
of RotateSecret assumes that a previous rotation request is
still in progress and returns an error.
* Update `default_version_id` after Lambda rotation concludes
Call `set_default_version_id` directly, rather than going
through `reset_default_version` as the Lambda function is
responsible for moving the version labels around, not `rotate_secret`.
* Run `black` over changed files
* Fix Python 2.7 compatibility
* Add additional test coverage for Secrets Manager
* Fix bug found by tests
AWSPENDING + AWSCURRENT check wasn't using `version_stages`.
Also tidy up the AWSCURRENT moving in `update_secret_version_stage`
to remove AWSPREVIOUS it from the new stage.
* Run `black` over changed files
* Add additional `rotate_secret` tests
* Skip `test_rotate_secret_lambda_invocations` in test server mode
* Add test for invalid Lambda ARN
* implement user pool mfa actions
* Add messages to errors
Add messages to errors
Fix error message
* Change exception type
* fix validation & add more tests
Co-authored-by: George Lewis <glewis@evertz.com>
* add test that fails with FilterNotImplementedError
* describe_network_acls: add support for owner-id filter
Co-authored-by: Kevin Neal <Kevin_Neal@intuit.com>
* `nextToken` value in `logs:describeLogStreams` response
Modified the pagination for FilterLogEvents to more closely follow
the real AWS behaviour.
* Make assertions work in py2 and py3.
* Fix : Remove Invalid request error
* Fix test and review commanets
* Remove duplicate put_integration_template
* API Gateway - Body can be None or empty
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* Fix: `nextToken` value in `logs:FilterLogEvents` response
Plagiarizing freely from @bpandola and his PR #3398, I have
modified the pagination for FilterLogEvents to more closely follow
the real AWS behaviour.
Fixes#3882
* Black reformatted my code.
* Remove timezone for python2.7 compatibility.
* Hopefully fix python2.7 compatibility for real.
* Additional test for a non-matching log group name in the nextToken.
* Implemented list instances in EMR
* removed import from tests
* make format
* fix W291 trailing whitespace
* removed to work for py2.7
* Storing only ec2_id and instance group in Fake instance
Co-authored-by: J <jdeepe@147dda1b0833.ant.amazon.com>
* Create SageMaker EndpointConfig with CloudFormation
Implement attributes for SM Endpoint Configs with CloudFormation
Delete SM Endpoint Configs with CloudFormation
Update SM Endpoint Configs with CloudFormation
* Fix typos in SM CF Model update test and refactor helper function for CF stack outputs
* Fixup weird commas in SM CF Test Configs from using black
* Create SageMaker Endpoints with CloudFormation
* Fix typos in SM CF update tests
* Create a formal interface for SM Cloudformation test configurations
* Create SageMaker Models with CloudFormation
* Utilize six for adding metaclass to TestConfig
* Update SM backend to return Model objects instead of response objects
* Create SageMaker Notebook Instance Lifecycle Configs with CloudFormation
Implement attributes for SM Notebook Instance Lifecycle Config in CloudFormation
Delete SM Notebook Instance Lifecycle Configs with CloudFormation
Update SM Notebook Instance Lifecycle Configs with CloudFormation
Also fixed error in create_from method where the properties where not
being referenced when setting OnCreate and OnStart.
Factor out template for SM Notebook Lifecycle Config CF tests
* Refactor SM CloudFormation create tests to use pytest.mark.parametrize
* Refactor SM CloudFormation get_attr tests to use pytest.mark.parametrize
Also update the NotebookInstance template function to use Name and Arn
for the output IDs so that the parametrization is easier.
* Refactor SM CloudFormation delete tests to use pytest.mark.parametrize
* Move event pattern validation into EventPattern class and apply enhanced pattern logic to all Rules
* Fix exists filtering logic to only match leaf nodes in event
* Apply black formatting
* Replace JSONDecodeError with ValueError for Python2 compatibility
* Update unit test names
* Move event pattern tests into test_event_pattern.py
* Apply black formatting
Co-authored-by: TSNoble <tom.noble@bjss.com>
* Make it possible to customize the ACM cert validation wait time.
Signed-off-by: Kai Xia <kaix+github@fastmail.com>
* address PR comments & change requests.
Signed-off-by: Kai Xia <kaix+github@fastmail.com>
* make tests work.
Signed-off-by: Kai Xia <kaix+github@fastmail.com>
* Create SageMaker Notebook Instances with CloudFormation
* Implement attributes for SageMaker notebook instance in Cloudformation
* Delete SageMaker Notebook Instances with CloudFormation
* Update SageMaker Notebook Instances with CloudFormation
* Factor out template into function for SageMaker notebook instance tests
* Py3: use unittest.mock instead of mock
* noqa
* oops
* just pull in patch()
* ignore RuntimeError when stopping patch
* ignore RuntimeError from default_session_mock.stop()
* Add prefix and numeric filtering logic for Archive EventPattern
* Pull EventPattern logic out into class and test logic more directly
* Apply black formatting
Co-authored-by: Tom Noble <tom.noble@bjss.com>
* Add exists filtering functionality to Archive. Add test case and refactor existing Archive EventPattern test cases
* Apply black formatting
* Change NotImplementedError to warning
* Simplify unimplemented warning for filters
* Change str check to six.string_types check for python2.7
Co-authored-by: Tom Noble <tom.noble@bjss.com>
* Add _does_event_match_pattern() to EventsBackend and use when determining whether to archive an event
* Add comment to _does_event_item_match_pattern_item()
* Expand test case for Archive EventFilter
* Apply black formatting
Co-authored-by: Tom Noble <tom.noble@bjss.com>
* Make security rules consistent between direct (backend) and indirect (api) boundaries
Security rules added directly via the backend were unable to be revoked via the API
because the port values were being stored as strings but were always coerced back
to integers by the botocore model. `"0" != 0`, so the rules would never match,
raising an `InvalidPermissionNotFoundError`.
This change ensures that the port values for a security group rule are always of type
`Union[int, None]`.
No tests needed to be modified as a result of this change. A new test was added that
explicitly covers the behavior that had been failing.
* Skip test in server mode
* Modify put_parameter() to raise ValidationError when value is empty string
* Simplify empty string check
Co-authored-by: Tom Noble <tom.noble@bjss.com>
The `InvalidPermission.Duplicate` error was already implemented for inbound rules,
but AWS also returns this error for duplicate outbound rules.
Very minor changes were needed on existing tests that were adding duplicate
outbound rules (when testing the RulesPerSecurityGroupLimitExceeded error).
* between clause is not case-sensitive anymore
* begins_with will raise an exception unless lower-case is used
Co-authored-by: David Pedrosa <d.pedrosa@indizen.com>
* adding physical_resource_id in SubnetRouteTableAssociation, Route and NatGW classes
* adding tests
* passing litern at test
* passsing black==19.10b0 as lintern
* passing test to python 2.7
* feature/support create_case, resolve_case, describe_cases and associated
tests
* review/support - addressed PR comments
- Have created a SupportCase object to persist case
- Associated testing to address PR comments
* Support - simplify tests
* Support - Simplify tests even more
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* fix route table association by internet gateway per https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_AssociateRouteTable.html
* Route53
- Add test for route table association by internet gateway
- Minor test tweak for Main route table values
TODO: explicitly set the route table main route association
* Route53
- forgot subnet id association test
Co-authored-by: Tony Greising-Murschel <tony@platform.sh>
The Record class was already capturing a unix timestamp, but it was incorrectly
converting it to ISO format when sending back to the client.
Updating the model to return the correct timestamp necessitated a minor change
to one of the tests because `botocore` converts non-timezone aware timestamps
to local time.
* Add IAM Role Description field to list_roles responses
The IAM ListRoles IAM API call will return the Description key/value
for each role if it exists. If it does not exist the Description
key is not included.
* fix handling in create_role resp
* blackg
* Combine two tests using pytest.mark.parametrize
* consistency
Also I found extra tests for describe_task_definition and deregister_task_definition that were not being run,
so I changed their names so they are found by pytest and made them pass. I also added checks to them for the new
status field.
* Support - added refresh_trusted_advisor_check
- Returns a random check status
- Returns the check id in the response
- Testing for these two functionalities
* test_support - addressed PR comments, to cycle through a faked number of
check status responses
* pool domain should always return a domain
Refs https://github.com/spulec/moto/issues/3706
* set character encoding
* test CloudFrontDomain exists on pool domain
* describe pool domain does not return cloudfront domain
* Support Podman for mocking Lambda
Podman supports all Docker APIs used in moto since version 3.0. Note
that Podman requires pulling the image before creating a container
using a fully-qualified image name (e.g., "docker.io/library/busybox"
instead of "busybox").
Test plan:
$ podman system service -t 0
$ DOCKER_HOST="unix://$XDG_RUNTIME_DIR/podman/podman.sock" pytest
Fixes https://github.com/spulec/moto/issues/3276
* Run black
* Python 2 compatibility
* Address review comments and improve parse_image_ref
* Fix: DynamoDB:GetItem throws wrong error when table doesn't exist
* Use unique exception for table not found, per PR feedback
* Just fix the reported issue, without touching anything else...
* events: fix archive event pattern match check
There is a missing `return True` for the positive match case in
matches_pattern, causing all valid patterns to fail.
* events: add test for valid, non-empty pattern match
* support - intial commit to kick off trusted advisor checks
* edit - expanded testing to include checking for expected check ids and
check names.
Added server testing
added support resource json to manifest file and simplified
support response return from reviewed comments
* Streamline loading of resource files
* edit - ensured regions are assigned in models
Co-authored-by: Bert Blommers <info@bertblommers.nl>
AWS requires certain parameters to be mutually inclusive.
Moto wasn't doing anything with the InstanceId parameter, which is now made
clear with a TODO.
* Fix `DBInstanceNotFound` error message
Changed from `Database` to `DBInstance`, which is actually what comes back from AWS.
* Remove duplicate test
The removed test actually fails if run in isolation because `rds2` is not a
valid boto3 client service. The reason this test never caused CI to fail is
because it is redefined later in the test suite, effectively making it dead
code that will never run.
Duplicate test has been removed and the remaining test has been improved
with more explicit asserts.
* fix heartbeatTimeout of NONE resulting in ValueError and polling returning empty string taskToken when it shouldn't be returned
* fix expected taskToken in impacted tests
Co-authored-by: Clint Parham <cparham@aligntech.com>
* correct exceptions when mising parameters
* test_render_template function
* update ses template function
* fix import
* except fixed
* tests and py2 fix
* Add support for RDS resource filtering
* Extensive testing was performed against real AWS endpoints in order to
nail down the filter behavior under various scenarios, ensuring that
`moto` returns the proper response or error.
* Full test coverage of all utility functions as well as several
filter/parameter combinations.
* Split up filter tests, per PR feedback
* Remove unused import
* Fix pytest teardown failure on Python 2.7
* Update the s3 post functionality to better support success_action_redirect
- Add the bucket/key values to the redirect url like s3 does, which
supports code that relies on the key value being there on the
redirect.
- Add support for replacing ${filename} in the key value with the actual
filename from the form upload.
See Issue #3667
* Update s3 tests for changed success_action_redirect behavior
- Adds a new test called test_s3_server_post_to_bucket_redirect that
tests both the ${filename} replacement and the key/value addition to the
redirect query args
- Updated the test_creating_presigned_post checks to handle the
key/value additions to the redirect url.
* Fix test updates to work with python2.7
- remove f-string usage
- fix urllib.parse imports to use six
Co-authored-by: Wynn Wilkes <wynn@leading2lean.com>
* Add KmsKeyId to Redshift Cluster
Add the KmsKeyId property when creating a cluster so that it is also
returned when querying the describe_clusters endpoint.
* Run black on updated files
* Add unit test for Redshift KmsKeyId
* Re-run black with correct version
The response returned for sns.get_endpoint_attributes was not in
sync with the actual response from boto.
Co-authored-by: Antillon, Alejandro <alejandro.antillon@f-secure.com>
These tests, when run, do not execute any `moto` code. They fail the
parameter validation check in `botocore`, which raises an exception
before ever sending a request. These tests do not cover or verify
any `moto` behavior and have been removed.
The `botocore` response parsers are forgiving when it comes to timestamps,
but a real AWS backend does return time zone details for this attribute.
Verified failure/fix using the Go repo included in the issue report.
Fixes#3516
* Address `boto` deprecation warnings
This commit eliminates the following warning:
../boto/ec2/connection.py:582:
PendingDeprecationWarning: The current get_all_instances implementation will be replaced with get_all_reservations.
`boto` isn't likely to ever make good on this warning, but doing the replacement will
declutter the `moto` test output.
* Remove `invoke_lambda` tracebacks from unit test logging
If an exception is encountered, the details are returned in the response payload.
Printing the traceback was just adding noise to the pytest output.
* Use known AMIs in unit tests
This commit eliminates the following warning in the pytest output:
`PendingDeprecationWarning: Could not find AMI with image-id:ami-123456, in the near future this will cause an error.`
Known, pre-loaded AMI image ids are used instead of random ids that don't actually
exist in the moto backend. The integrity of the tests is unaffected by this change.
A test has been added to provide explicit coverage of the PendingDeprecationWarning
raised when an invalid AMI image id is passed to moto.
The latest release of `botocore` (1.19.62) makes changes to the parameter
validation code, which for some reason was also covered by a couple of
`moto` tests.
These tests, when run, do not execute any `moto` code. They fail the
parameter validation check in `botocore`, which raises an exception
before ever sending a request. These tests do not cover or verify
any `moto` behavior and have been removed.
Ref: ff8ae76eccCloses#3627
Moto's implementation of autoscaling:CreateLaunchConfiguration is a little too loose,
allowing the ImageId parameter to be omitted and defaulting it to None, which results
in dozens of deprecation warnings:
`PendingDeprecationWarning: Could not find AMI with image-id:ami-123456, in the near future this will cause an error.`
This commit refactors calls to CreateLaunchConfiguration to explicitly pass in a known
AMI ImageId in order to avoid these warnings.
The integrity of the tests is unaffected by this change.
This commit eliminates the following warning (of which there are currently dozens):
../boto/ec2/connection.py:582:
PendingDeprecationWarning: The current get_all_instances implementation will be replaced with get_all_reservations.
`boto` isn't likely to ever make good on this warning, but doing the replacement will
declutter the `moto` test output.
* Added redshift.get_cluster_credentials
* Marked endpoint in list
* Removed f string from tests
* Python 2.7 compat changes
* Fixed parameter retrieval
* Formatting
* Removed try/catch in favor of if
* Changed to existing random_string util
Co-authored-by: Andrea Amorosi <aamorosi@amazon.es>
* Fix ec2 filter by empty tag value
Return `None` instead of an empty string when the tag key does not exist
and replace the falsy check with a more explicit `is None`, which allows
empty string values to correctly pass through the filter comparator.
Behavior confirmed against a real AWS backend.
Closes#3603
* Make test case more explicit
Test case now pulled directly from the issue report (#3603).
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* Initial attempt to mock AWS Media Live create_channel endpoint. Test fails.
* Completes basic implementation of Media Live create_channel endpoint
* Completes basic implementation of Media Live list_channels endpoint
* Adds skaffolds for describe_channel and delete_channel
* Adds unit test for delete_channel
* Adds unit test for describe_channel
* Reduces repetitive code by introducing a Channel model
* Implements MediaLive start_channel and stop_channel endpoints
* Fixes lack of support for the dash character in resource ARNs
* Implements MediaLive update_channel endpoint.
* Implements MediaLive create_input endpoint (and Input model).
* Implements MediaLive describe_input endpoint.
* Implements MediaLive list_inputs endpoint.
* Implements MediaLive update_input endpoint.
* Addse server tests for MediaLive
* Adds further url patterns for medialive
* Fixes url patterns
* Fixes url patterns
* Added explicit exception raise when no stack found.
Currently, any operation that uses 'get_stack' method from 'CloudFormationBackend' class
will fail with AttributeError or jinja2 exception if ran against non-existing stack(created/deleted)
To fix the issue I explicitly raised a 'ValidationError' exception.
Added tests for boto and boto3 responses.
* Moved non-existing stack tests to 'test_stack_events'
When using 'update_stack' to test raising an exception when the stack doesn't exist
test coverage dropped by 0.5%. I am using stack_events instead.
* Removed some unreachable paths
After adding the exception couple of paths in the code are unreachable as 'get_stack' doesn't return 'None' anymore.
This is the reason why coverall was reporting decreased coverage.
* Removed an unreachable path I missed
* Added couple of tests in cloudformation/models
* Added more assertions around raised exception
* Formatted document using black to fix issue with travis.
This test is flaky, but when it fails we don't get any indication as to why.
This commit ensures that the reason for failure will be part of the assertion
message.
Once we have information about why this test fails, we can troubleshoot further
and hopefully come up with a permanent fix.
Instead of modifying responses._default_mock, create our own
responses.RequestsMock object that we can modify as needed without
interfering with other users of the responses library.
Fixes#3264.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
* Add get_function_configuration support for Lambda
* remove unnesecary code from test and use _lambda_region when asserting
* rename function and skip coping configuration
* run black formatting
This handles the add-to-list case using the legacy `AttributeUpdates` parameter.
* Added test coverage.
* Verified against real AWS backend.
Closes#3561
The mocked response for ECS RegisterTaskDefinition has drifted from what
actually returns when run against a real ECS endpoint. I created a
minimal task definition for both EC2:
```
>>> ecs.register_task_definition(
family="moto",
containerDefinitions=[
{
"name": "hello_world",
"image": "hello-world:latest",
"memory": 400
}
]
)["taskDefinition"]
{'taskDefinitionArn': 'arn:aws:ecs:us-east-1:************:task-definition/moto:1',
'containerDefinitions': [{'name': 'hello_world',
'image': 'hello-world:latest',
'cpu': 0,
'memory': 400,
'portMappings': [],
'essential': True,
'environment': [],
'mountPoints': [],
'volumesFrom': []}],
'family': 'moto',
'revision': 1,
'volumes': [],
'status': 'ACTIVE',
'placementConstraints': [],
'compatibilities': ['EC2']}
```
and FARGATE:
```
>>> ecs.register_task_definition(
family="moto",
containerDefinitions=[
{
"name": "hello_world",
"image": "hello-world:latest",
"memory": 400
}
],
requiresCompatibilities=["FARGATE"],
networkMode="awsvpc",
cpu="256",
memory="512"
)["taskDefinition"]
{'taskDefinitionArn': 'arn:aws:ecs:us-east-1:************:task-definition/moto:2',
'containerDefinitions': [{'name': 'hello_world',
'image': 'hello-world:latest',
'cpu': 0,
'memory': 400,
'portMappings': [],
'essential': True,
'environment': [],
'mountPoints': [],
'volumesFrom': []}],
'family': 'moto',
'networkMode': 'awsvpc',
'revision': 2,
'volumes': [],
'status': 'ACTIVE',
'requiresAttributes': [{'name': 'com.amazonaws.ecs.capability.docker-remote-api.1.18'},
{'name': 'ecs.capability.task-eni'}],
'placementConstraints': [],
'compatibilities': ['EC2', 'FARGATE'],
'requiresCompatibilities': ['FARGATE'],
'cpu': '256',
'memory': '512'}
```
This change adds several default keys to the task based on those two
real responses and the AWS documentation:
https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RegisterTaskDefinition.html
The mock still doesn't match the real response exactly but it's much
closer than it was before.
* Properly coerce `privateDnsEnabled` to boolean value when parsing requests.
* Per AWS spec, default `privateDnsEnabled` request value to `True`.
* Properly serialize `privateDnsEnabled` as boolean value in responses.
* Add test coverage.
Ref: #3540
Applies the user credentials pattern from the ADMIN_NO_SRP_AUTH flow
to the ADMIN_USER_PASSWORD_AUTH auth flow for Cognito admin_initiate_auth
requests.
Co-authored-by: Robin Wilkins <r.wilkins@waracle.com>
* fix https://github.com/localstack/localstack/issues/3339
* fixe lint issues
* Fix review comments
- move deduplication time to constants
- make tests parameterized
- update tests as per review comments
* change variable name expectedCount => expected_count
* fix tests for python 2.7
increase deduplication mock config to account for delays
* ignore time mocking test in server mode
These tests were passing with TZ=UTC, but under a non-UTC timezone
they were failing:
E AssertionError: given
E X = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzutc()), …}, …]
E and
E Y = [{'timestamp': datetime.datetime(2020, 1, 1, 0, 0, tzinfo=tzlocal()), …}, …]
E X[0]['timestamp'] != Y[0]['timestamp']
With this fix, they pass either way.
Signed-off-by: Anders Kaseorg <andersk@mit.edu>
* Pass the "default" cluster
* Mock ECS exceptions more accurately
Moto's mock ECS has drifted fairly far from the actual ECS API in terms
of which exceptions it throws. This change begins to bring mock ECS's
exceptions in line with actual ECS exceptions. Most notably:
- Several custom exceptions have been replaced with their real ECS
exception. For example, "{0} is not a cluster" has been replaced with
ClusterNotFoundException
- Tests have been added to verify (most of) these exceptions work
correctly. The test coverage was a little spotty to begin with.
- The new exceptions plus the change to pass the "default" cluster
exposed a lot of places where mock ECS was behaving incorrectly. For
example, the ListTasks action is always scoped to a single cluster in
ECS but it listed tasks for all clusters in the mock. I've minimally
updated the tests to make them pass, but there's lots of opportunity to
refactor both this method's test and its implementation.
This does not provide full coverage of exceptions. In general, I ran
these operations against actual ECS resources and cross-referenced the
documentation to figure out what actual exceptions should be thrown and
what the messages should be. Consequently, I didn't update any
exceptions that took more than trivial amount of time to reproduce with
real resources.
* Retrieve SAML Attribute by Name instead of relying on order which is too fragile
* Handle case when SAML Attribute SessionDuration is not provided, as it is not a required attribute from SAML response
When session duration not provided, AWS consider by default a duration of one hour as cited in the following documentation:
"If this attribute is not present, then the credential last for one hour (the default value of the DurationSeconds parameter of the AssumeRoleWithSAML API)."
https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_providers_create_saml_assertions.html#saml_role-session-duration
Traceback was:
[...]
File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/responses.py", line 79, in assume_role_with_saml
role = sts_backend.assume_role_with_saml(
File "/Users/benjamin.brabant/Projects/PERSO/moto/moto/sts/models.py", line 99, in assume_role_with_saml
role = AssumedRole(**kwargs)
TypeError: __init__() missing 1 required positional argument: 'duration'
* Process saml xml namespaces properly instead of relying on textual prefix that can vary between identity providers
* Handle when SAML response AttributeValue xml tag contains attributes that force xmltodict to build a dictionary as for complex types instead of directly returning string value
Leverage force_cdata option of xmltodict parser that always return a complex dictionary even if xml tag contains only text and no attributes.
* Improve existing test_assume_role_with_saml to be coherent with other assume_role_with_saml tests and remove dead code at the same time
By definition, `single-node` clusters can only consist of 1 node. Likewise,
`multi-node` clusters must have 2 or more nodes.
* Ensure `ClusterType` parameter is either `multi-node` or `single-node`.
* Ensure proper validation of `NumberOfNodes` parameter based on `ClusterType`
parameter.
* Fix existing test case that incorrectly allowed a `multi-node` cluster to
consist of 1 node.
* Add dedicated test for resizing a cluster from `single-node` to `multi-node`
and back again.
Behavior and error messages have been verified against a real AWS backend.
* Add format command to makefile
* Refactor executions to be a attribute of StateMachine
* Begin to add tests for execution history
* Add tests for failed and successful event histories, with implementations
* Add failure case to environment var check
* Skip test if in server mode and update implementation coverage
* Add conditional import for mock to cover python 2
* Refactor stop execution logic into StateMachine
* Refactor event history environment variable into settings.py
* Remove typing and os import
The `boto` library (long ago superseded by `boto3`) has not had an official
release in over two years or even a commit in the last 18 months. Importing
the package (or indirectly importing it by via `moto`) generates a deprecation
warning. Additionally, an ever-increasing number of `moto` users who have
left `boto` behind for `boto3` are still being forced to install `boto`.
This commit vendors a very small subset of the `boto` library--only the code
required by `moto` to run--into the /packages subdirectory. A README file
is included explaining the origin of the files and a recommendation for how
they can be removed entirely in a future release.
NOTE: Users of `boto` will still be able to use `moto` after this is merged.
closes#2978closes#3013closes#3170closes#3418
relates to #2950
A test added in #2401 copied the name of an existing test, preventing it
from being run. This commit renames the second test, allowing both to
be picked up by the test runner.
Original code was trying to raise a ClientError directly. Change to
appropriate Redshift exception class.
* Fix test assertion for `boto`.
* Add test coverage for `boto3`.
Previous code would raise `TypeError: 'dict_keys' object is not subscriptable`
when run under Python 3.
* Re-write code in Python 2/3 compatible way.
* Add clarifying comment.
* Add test coverage.
Supersedes #3227
The previous code was trying to raise a botocore ClientError directly, which
was actually generating a secondary AttributeError because the arguments passed
to ClientError() were incorrect.
This replaces the ClientError() call with a proper moto exception class for
Redshift and fixes the test assertions appropriately.
Supersedes #1957
* Fix:CloudWatch List metrics with dimensions
* Fix:CloudWatch List metrics with dimensions
* Fixed new cases and added more tests
Co-authored-by: usmankb <usman@krazybee.com>
* Extract Duplicate Code into Helper Method
DRY up the tests and replace the arbitrary `sleep()` calls with a more
explicit check before progressing.
* Improve Testing of batch:TerminateJob
The test now confirms that the job was terminated by sandwiching a `sleep`
command between two `echo` commands. In addition to the original checks
of the terminated job status/reason, the test now asserts that only the
first echo command succeeded, confirming that the job was indeed terminated
while in progress.
* Fix Race Condition in batch:SubmitJob
The `test_submit_job` in `test_batch.py` kicks off a job, calls `describe_jobs`
in a loop until the job status returned is SUCCEEDED, and then asserts against
the logged events.
The backend code that runs the submitted job does so in a separate thread. If
the job was successful, the job status was being set to SUCCEEDED *before* the
event logs had been written to the logging backend.
As a result, it was possible for the primary thread running the test to detect
that the job was successful immediately after the secondary thread had updated
the job status but before the secondary thread had written the logs to the
logging backend. Under the right conditions, this could cause the subsequent
logging assertions in the primary thread to fail.
Additionally, the code that collected the logs from the container was using
a "dodgy hack" of time.sleep() and a modulo-based conditional that was
ultimately non-deterministic and could result in log messages being dropped
or duplicated in certain scenarios.
In order to address these issues, this commit does the following:
* Carefully re-orders any code that sets a job status or timestamp
to avoid any obvious race conditions.
* Removes the "dodgy hack" in favor of a much more straightforward
(and less error-prone) method of collecting logs from the container.
* Removes arbitrary and unnecessary calls to time.sleep()
Before applying any changes, the flaky test was failing about 12% of the
time. Putting a sleep() call between setting the `job_status` to SUCCEEDED
and collecting the logs, resulted in a 100% failure rate. Simply moving
the code that sets the job status to SUCCEEDED to the end of the code block,
dropped the failure rate to ~2%. Finally, removing the log collection
hack allowed the test suite to run ~1000 times without a single failure.
Taken in aggregate, these changes make the batch backend more deterministic
and should put the nail in the coffin of this flaky test.
Closes#3475
* Added support for EMR Security Configurations and Kerberos Attributes.
* Revised exception-raising test to work with pytest api.
* Added htmlcov to .gitignore; upgrading botocore to 1.18.17, per commit d29475e.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* Add support for empty strings in non-key attributes
https://github.com/spulec/moto/issues/3339
* Nose, not pytest
* Revert "Nose, not pytest"
This reverts commit 5a3cf6c887dd9fafa49096c82cfa3a3b7f91d224.
* PUT is default action
* Fixed issue 3448 for DynamoDB update_item
* Tidied up fix for issue 3448 for DynamoDB update_item
* Reformatted fix for issue 3448 for DynamoDB update_item
* removed use of f-strings in test case as it fails in Travis CI build due to Python 2.7 support of f strings
