* refactor(ssm): Add Documents class to avoid dictionary handling
This also solves the datetime format issue in TestAccAWSCloudWatchEventTarget_ssmDocument
* **Fix bug.** If a cloudformation stack is updated with a new
parameter, that parameter should be honored. Several unit tests
had bugs where they were not providing parameters required by the template.
* **Fix bug.** Do not update stack parameters until after deleting removed
resources, so that any references to removed parameters can be resolved.
* **Fix bug.** Per the API, creation of a change set should not modify
a stack. The `diff` method, called in the creation of a
FakeChangeSet, was mutating the resource map which was problematic
* Update cloudwatch.put_metric_alarm to accept TreatMissingData and Tags parameter
* Add parameter ExtendedStatistic and EvaluateLowSampleCountPercentile to cloudwatch.put_metric_alarm
* Add parameter ThresholdMetricId to cloudwatch.put_metric_alarm
This fixes#4141, and pave ways for future changes around changeset.
We had subclassed FakeChangeSet from FakeStack, not from BaseModel. This made
us easier to send the response for describe_change_set calls, but when we are
handling the details of change set, the old approach won't work at all.
For example, when we were creating a changeset, we were actually creating a
stack without registering it (self.stacks), and future update onto this stack
is not really possible.
Signed-off-by: Kai Xia <kaix+github@fastmail.com>
* fix OPTIONS requests on non-existing API GW integrations
* add cloudformation models for API Gateway deployments
* bump version
* add backdoor to return CloudWatch metrics
* Updating implementation coverage
* Updating implementation coverage
* add cloudformation models for API Gateway deployments
* Updating implementation coverage
* Updating implementation coverage
* Implemented get-caller-identity returning real data depending on the access key used.
* bump version
* minor fixes
* fix Number data_type for SQS message attribute
* fix handling of encoding errors
* bump version
* make CF stack queryable before starting to initialize its resources
* bump version
* fix integration_method for API GW method integrations
* fix undefined status in CF FakeStack
* Fix apigateway issues with terraform v0.12.21
* resource_methods -> add handle for "DELETE" method
* integrations -> fix issue that "httpMethod" wasn't included in body request (this value was set as the value from refer method resource)
* bump version
* Fix setting http method for API gateway integrations (#6)
* bump version
* remove duplicate methods
* add storage class to S3 Key when completing multipart upload (#7)
* fix SQS performance issues; bump version
* add pagination to SecretsManager list-secrets (#9)
* fix default parameter groups in RDS
* fix adding S3 metadata headers with names containing dots (#13)
* Updating implementation coverage
* Updating implementation coverage
* add cloudformation models for API Gateway deployments
* Updating implementation coverage
* Updating implementation coverage
* Implemented get-caller-identity returning real data depending on the access key used.
* make CF stack queryable before starting to initialize its resources
* bump version
* remove duplicate methods
* fix adding S3 metadata headers with names containing dots (#13)
* Update amis.json to support EKS AMI mocks (#15)
* fix PascalCase for boolean value in ListMultipartUploads response (#17); fix _get_multi_param to parse nested list/dict query params
* determine non-zero container exit code in Batch API
* support filtering by dimensions in CW get_metric_statistics
* fix storing attributes for ELBv2 Route entities; API GW refactorings for TF tests
* add missing fields for API GW resources
* fix error messages for Route53 (TF-compat)
* various fixes for IAM resources (tf-compat)
* minor fixes for API GW models (tf-compat)
* minor fixes for API GW responses (tf-compat)
* add s3 exception for bucket notification filter rule validation
* change the way RESTErrors generate the response body and content-type header
* fix lint errors and disable "black" syntax enforcement
* remove return type hint in RESTError.get_body
* add RESTError XML template for IAM exceptions
* add support for API GW minimumCompressionSize
* fix casing getting PrivateDnsEnabled API GW attribute
* minor fixes for error responses
* fix escaping special chars for IAM role descriptions (tf-compat)
* minor fixes and tagging support for API GW and ELB v2 (tf-compat)
* Merge branch 'master' into localstack
* add "AlarmRule" attribute to enable support for composite CloudWatch metrics
* fix recursive parsing of complex/nested query params
* bump version
* add API to delete S3 website configurations (#18)
* use dict copy to allow parallelism and avoid concurrent modification exceptions in S3
* fix precondition check for etags in S3 (#19)
* minor fix for user filtering in Cognito
* fix API Gateway error response; avoid returning empty response templates (tf-compat)
* support tags and tracingEnabled attribute for API GW stages
* fix boolean value in S3 encryption response (#20)
* fix connection arn structure
* fix api destination arn structure
* black format
* release 2.0.3.37
* fix s3 exception tests
see botocore/parsers.py:1002 where RequestId is removed from parsed
* remove python 2 from build action
* add test failure annotations in build action
* fix events test arn comparisons
* fix s3 encryption response test
* return default value "0" if EC2 availableIpAddressCount is empty
* fix extracting SecurityGroupIds for EC2 VPC endpoints
* support deleting/updating API Gateway DomainNames
* fix(events): Return empty string instead of null when no pattern is specified in EventPattern (tf-compat) (#22)
* fix logic and revert CF changes to get tests running again (#21)
* add support for EC2 customer gateway API (#25)
* add support for EC2 Transit Gateway APIs (#24)
* feat(logs): add `kmsKeyId` into `LogGroup` entity (#23)
* minor change in ELBv2 logic to fix tests
* feat(events): add APIs to describe and delete CloudWatch Events connections (#26)
* add support for EC2 transit gateway route tables (#27)
* pass transit gateway route table ID in Describe API, minor refactoring (#29)
* add support for EC2 Transit Gateway Routes (#28)
* fix region on ACM certificate import (#31)
* add support for EC2 transit gateway attachments (#30)
* add support for EC2 Transit Gateway VPN attachments (#32)
* fix account ID for logs API
* add support for DeleteOrganization API
* feat(events): store raw filter representation for CloudWatch events patterns (tf-compat) (#36)
* feat(events): add support to describe/update/delete CloudWatch API destinations (#35)
* add Cognito UpdateIdentityPool, CW Logs PutResourcePolicy
* feat(events): add support for tags in EventBus API (#38)
* fix parameter validation for Batch compute environments (tf-compat)
* revert merge conflicts in IMPLEMENTATION_COVERAGE.md
* format code using black
* restore original README; re-enable and fix CloudFormation tests
* restore tests and old logic for CF stack parameters from SSM
* parameterize RequestId/RequestID in response messages and revert related test changes
* undo LocalStack-specific adaptations
* minor fix
* Update CodeCov config to reflect removal of Py2
* undo change related to CW metric filtering; add additional test for CW metric statistics with dimensions
* Terraform - Extend whitelist of running tests
Co-authored-by: acsbendi <acsbendi28@gmail.com>
Co-authored-by: Phan Duong <duongpv@outlook.com>
Co-authored-by: Thomas Rausch <thomas@thrau.at>
Co-authored-by: Macwan Nevil <macnev2013@gmail.com>
Co-authored-by: Dominik Schubert <dominik.schubert91@gmail.com>
Co-authored-by: Gonzalo Saad <saad.gonzalo.ale@gmail.com>
Co-authored-by: Mohit Alonja <monty16597@users.noreply.github.com>
Co-authored-by: Miguel Gagliardo <migag9@gmail.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* ELBv2 - ListenerRule condition validation
- ListenerRule condition model now uses upper case field names that
match input params for boto and CloudFormation.
- BaseResponse._get_params() introduced to make it easier to deal
with the querystring input params.
* - introduce environment variable for DEFAULT_KEY_BUFFER_SIZE
* - prefix env variable with MOTO_S3 to avoid env variable conflicts
* - reduce the DEFAULT_KEY_BUFFER_SIZE to be less than the S3_UPLOAD_PART_MIN_SIZE to prevent in memory caching of multi part uploads
* - black formatting
* - fix formatting
* - fix missing import
* If token and attributes are the same, return endpoint
* fix black
* moto sns platform_endpoint.attributes includes only token,enabled
* add tests when calling sns#create_platform_endpoint with same attrs for #4056
* Add delete container and list tags endpoints to MediaStore
* Black reformat
* Fixed Lint problems
* Check if dictionary was deleted effectively
* lint fix
* MediaPackageClientError
* Lint Fix
* Test unknown channel describe
* Concatenation Fix
* MediaPackage - fix error message
* MediaPackage ClientError part2
* Mediastoredata not working
Base url
tests and renaming
typo
List Items not returning proper JSON and wrongly hitting get_object response
MediaStore2
Tests
* More implementation
* Fix tests and format
* Comments fix
* Comments 2
* MediastoreData - alternative logic to figure out appropriate host
Co-authored-by: av <arcovoltaico@gmail.com>
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* Add outputs and vpc interfaces to a mediaconnect flow
* Add negative tests for add_flow_outputs and add_flow_vpc_interfaces
* fix: fstring to format
* MediaConnect - add appropriate URLs for ServerMode tests
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* Update implementation coverage
* EC2 - Update instance type offerings
* IAM - update list of managed policies
* Changelog for release 2.0.9
* Instance Type Offerings - fix number of available offerings
* Add ssm parsing support for cloudformation stacks
* Start adding unit tests for ssm parameter parsing
* Add tests for code update
* Add tests to parse ssm parameters code
* Fix black lint errors
* Fix bug.
* Need to specify region_name
* region needs to be same
* Use ssm_backends[region] instead of ssm_backend
* StringList -> string
* Linting
* check if servermode tests are on
* Typo
* Added support for ListenerRule. Will remove cruft
* Pushing latest
* Something works
* Put back ripped out code
* Save point. Incase I need more validations
* Revert "Save point. Incase I need more validations"
This reverts commit dac4953335dd9335eddb7a91a63667bc3c17104c.
* Fixed validations and some refactor
* Fix formatting
* Linting
* Cannot refactor if I have to fix all tests
* Remove exceptions for now. Will do in another PR
* Remove validations. Will add in next PR
* Fix broken tests. Almost.:
* Fix all tests. Some sneaky for now.
* Python2 making me write bad code
* OrderedDict.move_to_end() does not work in python2
* Linting
* Add more checks to field in conditions later.
* Unwnated change in FakeListener
* Revert "Unwnated change in FakeListener"
This reverts commit 962c2fdfd76fce999de9feccf1dd1c3ec48c459f.
* Add back default listener rule
* Linting fix
* Fix priority sorting
* Add cloudformation test for edge case
* Add validation for ForwardConfig in Action of ListernRule CF
* use not in
* set the priority template correctly
* Check for boolean in condition
* One more check
* Implement update_from_cloudformation_json for Listener and ListenerRule
* Unwanted spaces
* Linting issues
* Add tests for code coverage
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* Add support for DynamoDB Backup/Restore
Basic support for the following endpoints has been implemented with full test coverage:
- create_backup
- delete_backup
- describe_backup
- list_backups
- restore_table_from_backup
Behavior and error messages verified against a real AWS backend.
* Refactor test based on PR feedback
* Add ssm parsing support for cloudformation stacks
* Start adding unit tests for ssm parameter parsing
* Add tests for code update
* Add tests to parse ssm parameters code
* Fix black lint errors
* Fix bug.
* Need to specify region_name
* region needs to be same
* Use ssm_backends[region] instead of ssm_backend
* StringList -> string
* Linting
* check if servermode tests are on
* Typo
* Added support for ListenerRule. Will remove cruft
* Pushing latest
* Something works
* Put back ripped out code
* Save point. Incase I need more validations
* Revert "Save point. Incase I need more validations"
This reverts commit dac4953335dd9335eddb7a91a63667bc3c17104c.
* Fixed validations and some refactor
* Fix formatting
* Linting
* Cannot refactor if I have to fix all tests
* Remove exceptions for now. Will do in another PR
* Remove validations. Will add in next PR
* Fix broken tests. Almost.:
* Fix all tests. Some sneaky for now.
* Python2 making me write bad code
* OrderedDict.move_to_end() does not work in python2
* Linting
* Add more checks to field in conditions later.
* Unwnated change in FakeListener
* Revert "Unwnated change in FakeListener"
This reverts commit 962c2fdfd76fce999de9feccf1dd1c3ec48c459f.
* Add back default listener rule
* Linting fix
* Fix priority sorting
* Add cloudformation test for edge case
* Add validation for ForwardConfig in Action of ListernRule CF
* use not in
* set the priority template correctly
* Check for boolean in condition
* One more check
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* - Adding checking for resource type in tag functions
- Adding TargetNotFoundException when no resource found
- Adding support for tags for root OU, OU and Policies
- Adding tests covering the new code
- Adding test for deletion of a tag
* fixed linting issue
* - renamed helper function to a more logical name
- added tests for helper function
- fixed bugs in tests for tag functions
Co-authored-by: Sjoerd Tromp <stromp@schubergphilis.com>
* Add ssm parsing support for cloudformation stacks
* Start adding unit tests for ssm parameter parsing
* Add tests for code update
* Add tests to parse ssm parameters code
* Fix black lint errors
* Fix bug.
* Need to specify region_name
* region needs to be same
* Use ssm_backends[region] instead of ssm_backend
* StringList -> string
* Linting
* check if servermode tests are on
* Typo
* Added support for ListenerRule. Will remove cruft
* Pushing latest
* Something works
* Put back ripped out code
* Save point. Incase I need more validations
* Revert "Save point. Incase I need more validations"
This reverts commit dac4953335dd9335eddb7a91a63667bc3c17104c.
* Fixed validations and some refactor
* Fix formatting
* Linting
* Cannot refactor if I have to fix all tests
* Remove exceptions for now. Will do in another PR
* Remove validations. Will add in next PR
* Fix broken tests. Almost.:
* Fix all tests. Some sneaky for now.
* Python2 making me write bad code
* OrderedDict.move_to_end() does not work in python2
* Linting
* Add more checks to field in conditions later.
* Unwnated change in FakeListener
* Revert "Unwnated change in FakeListener"
This reverts commit 962c2fdfd76fce999de9feccf1dd1c3ec48c459f.
* Add back default listener rule
* Linting fix
* Fix priority sorting
* Add cloudformation test for edge case
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* Including labels and versions in SSM Get Parameters
* implementing NextToken and MaxResults into the SSM Get Parameter History functionality
* Implementing unit tests and some lint refactoring for NextToken implementation in get_parameter_history
* Implement correct error when requesting specific version of a parameter which exists but does not have this version
* removing trailing whitespace causing lint failure
* Adding unit tests and fixing linting for new error handling
* Fixing small bug in response message
* Revert change in get_parameters as versioning is not currently implemented in this method. Will fix as a separate PR
* Check exit status of container
* Added support for job dependencies
* batch container overrides
* add AWS_BATCH_JOB_ID to container env variables
* lint with black
* refactor batch dependency test
* refactor batch dependency test
* fix index
Co-authored-by: jterry64 <justin.terry@wri.org>
Co-authored-by: Daniel Mannarino <daniel.mannarino@gmail.com>
* basic implementation of update rest api
* basic implementation of update rest api
* basic implementation of update rest api
* review comments from bblommers
Co-authored-by: rajinder saini <rajinder.saini@c02vt5k2htd6.corp.climate.com>
* Add delete container and list tags endpoints to MediaStore
* Black reformat
* Fixed Lint problems
* Check if dictionary was deleted effectively
* lint fix
Co-authored-by: av <arcovoltaico@gmail.com>
* Add ssm parsing support for cloudformation stacks
* Start adding unit tests for ssm parameter parsing
* Add tests for code update
* Add tests to parse ssm parameters code
* Fix black lint errors
* Fix bug.
* Need to specify region_name
* region needs to be same
* Use ssm_backends[region] instead of ssm_backend
* StringList -> string
* Linting
* check if servermode tests are on
* Typo
Co-authored-by: Bert Blommers <bblommers@users.noreply.github.com>
* Fix:SES-Added params check in template
* Added more tests and handled message
* linting
* fixed tests
* fix attribute name in message
* fix logic for exception
* Fix sqs message retention logic
* Apply lint to moto/sqs/models.py
* Fix failed tests because of freezing time
* Fix freezing time in test_publish_to_sqs_in_different_region
* Dont fail if CodeCov fails - for now
* CI - Force cache rebuild
* Bump werkzeug to latest version
* CI - Enforce cache flush
* ManagedBlockchain - fix error format
* ManagedBlockchain - Fix tests to use pytest.raises paradigm
* Revert "Lock Flask (#3925)"
This reverts commit 8bb0feb956.
* CI - Enforce cache rebuild
* Support rotating secrets using Lambda
The Secrets manager rotation process uses an AWS Lambda function
to perform the rotation of a secret. [1]
In fact, it's not possible to trigger rotation of a Secret
without specifying a Lambda function at some point in the life
of the secret:
```
$ aws secretsmanager rotate-secret --secret-id /rotationTest
An error occurred (InvalidRequestException) when calling the RotateSecret operation: No Lambda rotation function ARN is associated with this secret.
```
`moto` can be a little more lenient in this regard and allow
`rotate_secret` to be called without a Lambda function being
present, if only to allow simulation of the `AWSCURRENT` and
`AWSPREVIOUS` labels moving across versions.
However, if a lambda function _has_ been specified when calling
`rotate_secret`, it should be invoked therefore providing the
developer with the full multi-stage process [3] which can be
used to test the Lambda function itself and ensuring that full
end-to-end testing is performed. Without this there's no easy
way to configure the Secret in the state needed to provide the
Lambda function with the data in the format it needs to be in
at each step of the invocation process.
[1]: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-lambda-function-overview.html
[2]: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.rotate_secret
[3]: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-lambda-function-overview.html#rotation-explanation-of-steps
* Run `black` over `secretsmanager/models.py`
* Make `lambda_backends` import local to the condition
* Implement `update_secret_version_stage`
Allow a staging label to be moved across versions.
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/secretsmanager.html#SecretsManager.Client.update_secret_version_stage
* Add an integration test for Secrets Manager & Lambda
* Support passing `ClientRequestToken` to `put_secret_value`
By passing `ClientRequestToken` to `put_secret_value` within
the lambda function invoked by calling `rotate_secret`, one
can update the value associated with the existing (pending)
version, without causing a new secret version to be created.
* Add application logic for `AWSPENDING`
The rotation function must end with the versions of the secret
in one of two states:
- The `AWSPENDING` and `AWSCURRENT` staging labels are
attached to the same version of the secret, or
- The `AWSPENDING` staging label is not attached to any
version of the secret.
If the `AWSPENDING` staging label is present but not attached
to the same version as `AWSCURRENT` then any later invocation
of RotateSecret assumes that a previous rotation request is
still in progress and returns an error.
* Update `default_version_id` after Lambda rotation concludes
Call `set_default_version_id` directly, rather than going
through `reset_default_version` as the Lambda function is
responsible for moving the version labels around, not `rotate_secret`.
* Run `black` over changed files
* Fix Python 2.7 compatibility
* Add additional test coverage for Secrets Manager
* Fix bug found by tests
AWSPENDING + AWSCURRENT check wasn't using `version_stages`.
Also tidy up the AWSCURRENT moving in `update_secret_version_stage`
to remove AWSPREVIOUS it from the new stage.
* Run `black` over changed files
* Add additional `rotate_secret` tests
* Skip `test_rotate_secret_lambda_invocations` in test server mode
* Add test for invalid Lambda ARN
* implement user pool mfa actions
* Add messages to errors
Add messages to errors
Fix error message
* Change exception type
* fix validation & add more tests
Co-authored-by: George Lewis <glewis@evertz.com>
* add test that fails with FilterNotImplementedError
* describe_network_acls: add support for owner-id filter
Co-authored-by: Kevin Neal <Kevin_Neal@intuit.com>
* `nextToken` value in `logs:describeLogStreams` response
Modified the pagination for FilterLogEvents to more closely follow
the real AWS behaviour.
* Make assertions work in py2 and py3.
* Fix : Remove Invalid request error
* Fix test and review commanets
* Remove duplicate put_integration_template
* API Gateway - Body can be None or empty
Co-authored-by: Bert Blommers <info@bertblommers.nl>
* Fix: `nextToken` value in `logs:FilterLogEvents` response
Plagiarizing freely from @bpandola and his PR #3398, I have
modified the pagination for FilterLogEvents to more closely follow
the real AWS behaviour.
Fixes#3882
* Black reformatted my code.
* Remove timezone for python2.7 compatibility.
* Hopefully fix python2.7 compatibility for real.
* Additional test for a non-matching log group name in the nextToken.
* Implemented list instances in EMR
* removed import from tests
* make format
* fix W291 trailing whitespace
* removed to work for py2.7
* Storing only ec2_id and instance group in Fake instance
Co-authored-by: J <jdeepe@147dda1b0833.ant.amazon.com>
* Create SageMaker EndpointConfig with CloudFormation
Implement attributes for SM Endpoint Configs with CloudFormation
Delete SM Endpoint Configs with CloudFormation
Update SM Endpoint Configs with CloudFormation
* Fix typos in SM CF Model update test and refactor helper function for CF stack outputs
* Fixup weird commas in SM CF Test Configs from using black
* Create SageMaker Endpoints with CloudFormation
* Fix typos in SM CF update tests
* Create a formal interface for SM Cloudformation test configurations
* Create SageMaker Models with CloudFormation
* Utilize six for adding metaclass to TestConfig
* Update SM backend to return Model objects instead of response objects
* Create SageMaker Notebook Instance Lifecycle Configs with CloudFormation
Implement attributes for SM Notebook Instance Lifecycle Config in CloudFormation
Delete SM Notebook Instance Lifecycle Configs with CloudFormation
Update SM Notebook Instance Lifecycle Configs with CloudFormation
Also fixed error in create_from method where the properties where not
being referenced when setting OnCreate and OnStart.
Factor out template for SM Notebook Lifecycle Config CF tests
* Refactor SM CloudFormation create tests to use pytest.mark.parametrize
* Refactor SM CloudFormation get_attr tests to use pytest.mark.parametrize
Also update the NotebookInstance template function to use Name and Arn
for the output IDs so that the parametrization is easier.
* Refactor SM CloudFormation delete tests to use pytest.mark.parametrize
* Move event pattern validation into EventPattern class and apply enhanced pattern logic to all Rules
* Fix exists filtering logic to only match leaf nodes in event
* Apply black formatting
* Replace JSONDecodeError with ValueError for Python2 compatibility
* Update unit test names
* Move event pattern tests into test_event_pattern.py
* Apply black formatting
Co-authored-by: TSNoble <tom.noble@bjss.com>
* Make it possible to customize the ACM cert validation wait time.
Signed-off-by: Kai Xia <kaix+github@fastmail.com>
* address PR comments & change requests.
Signed-off-by: Kai Xia <kaix+github@fastmail.com>
* make tests work.
Signed-off-by: Kai Xia <kaix+github@fastmail.com>
* Create SageMaker Notebook Instances with CloudFormation
* Implement attributes for SageMaker notebook instance in Cloudformation
* Delete SageMaker Notebook Instances with CloudFormation
* Update SageMaker Notebook Instances with CloudFormation
* Factor out template into function for SageMaker notebook instance tests
* Py3: use unittest.mock instead of mock
* noqa
* oops
* just pull in patch()
* ignore RuntimeError when stopping patch
* ignore RuntimeError from default_session_mock.stop()
* Add prefix and numeric filtering logic for Archive EventPattern
* Pull EventPattern logic out into class and test logic more directly
* Apply black formatting
Co-authored-by: Tom Noble <tom.noble@bjss.com>
* Add exists filtering functionality to Archive. Add test case and refactor existing Archive EventPattern test cases
* Apply black formatting
* Change NotImplementedError to warning
* Simplify unimplemented warning for filters
* Change str check to six.string_types check for python2.7
Co-authored-by: Tom Noble <tom.noble@bjss.com>
