When comparing old and new values when doing a nested item update, the
`!=` implementation fails when the value being compared is `None`. This
results in an exception when trying to insert a new item into a nested
map. So just do a quick check that the original value is exists before
doing the comparison, as the `None` default is what is tripping this.
* IAM User Cloudformation Enhancements: update, delete, getatt.
* AWS::IAM::Policy Support
* Added unit tests for AWS:IAM:Policy for roles and groups. Fixed bug related to groups.
* AWS:IAM:AccessKey CloudFormation support.
* Refactor of CloudFormation parsing.py methods to simplify and standardize how they call to the models. Adjusted some models accordingly.
* Further model CloudFormation support changes to align with revised CloudFormation logic. Mostly avoidance of getting resoure name from properties.
* Support for Kinesis Stream RetentionPeriodHours param.
* Kinesis Stream Cloudformation Tag Support.
* Added omitted 'region' param to boto3.client() calls in new tests.
Co-authored-by: Joseph Weitekamp <jweite@amazon.com>
* change line position for uuid and last_modified because they're not input parameters
* add event_source_arn validator and setter
* refactor batch_size as setter
* add helper function to parse arn and return source service
* fix for EventSource's create_from_cfn, there was no reference in the lambda object for the esm if created by cfn
* add esm deletion by cloudformation
* remove unused variable in test
* add cfn's update
* add complete implementation of delete_from_cfn
* blacked changed files
* fix test with invalid batchsize for sqs
* Dynamodb2 Table - Bugfix for localindex and implemented get_cfn_attributes
* Dynamodb2 eventsource - fix test to use StreamArn attribute
* Lambda Test - fix test_update_event_source_mapping
When using the ADD syntax to sum up different components
the path that is provided is allowed to be non-existent.
In such a case DynamoDB will initialize it depending on
the type of the value.
If it is a number it will be initialized with 0.
If it is a set it will be initialized with an empty set.
Part of structured approach for UpdateExpressions:
1) Expression gets parsed into a tokenlist (tokenized)
2) Tokenlist get transformed to expression tree (AST)
3) The AST gets validated (full semantic correctness)
4) AST gets processed to perform the update -> this commit
This commit uses the AST to execute the UpdateExpression.
All the existing tests pass. The only tests that have been
updated are in test_dynamodb_table_with_range_key.py because
they wrongly allow adding a set to a path that doesn't exist.
This has been alligend to correspond to the behavior of AWS
DynamoDB.
This commit will resolve https://github.com/spulec/moto/issues/2806
Multiple tests have been implemented that verify this.
This commit puts AST validation on the execution path. This means updates get
validated prior to being executed. There were quite a few tests that were not
working against Amazon DDB. These tests I considered broken and as such this
commit adapts them such that they pass against Amazon DDB.
test_update_item_on_map()
=> One of the SET actions would try to set a nested element by specifying the nesting on the path
rather than by putting a map as a value for a non-existent key. This got changed.
test_item_size_is_under_400KB
=> Used the keyword "item" which DDB doesn't like. Change to cont in order to keep the same sizings.
=> Secondly the size error messages differs a bit depending whether it is part of the update or part
of a put_item. For an update it should be:
Item size to update has exceeded the maximum allowed size
otherwise it is
Item size has exceeded the maximum allowed size'
test_remove_top_level_attribute
=> Used a keyword item. Use ExpressionAttributeNames
test_update_item_double_nested_remove
=> Used keywords name & first. Migrated to non-deprecated API and use ExpressionAttributeNames
test_update_item_set &
test_boto3_update_item_conditions_pass &
test_boto3_update_item_conditions_pass_because_expect_not_exists &
test_boto3_update_item_conditions_pass_because_expect_not_exists_by_compare_to_null &
test_boto3_update_item_conditions_pass_because_expect_exists_by_compare_to_not_null &
test_boto3_update_item_conditions_fail &
test_boto3_update_item_conditions_fail_because_expect_not_exists &
test_boto3_update_item_conditions_fail_because_expect_not_exists_by_compare_to_null
=> Were broken tests which had string literal instead of value placeholder
Part of structured approach for UpdateExpressions:
1) Expression gets parsed into a tokenlist (tokenized)
2) Tokenlist get transformed to expression tree (AST)
3) The AST gets validated (full semantic correctness) -> this commit
4) AST gets processed to perform the update
This commit uses the AST to perform validation. Validation makes sure the
nodes encounterd have valid values and they will also resolve values for
references that refer to item state or values passed into the expression.
Part of structured approach for UpdateExpressions:
1) Expression gets parsed into a tokenlist (tokenized)
2) Tokenlist get transformed to expression tree (AST) -> This commit
3) The AST gets validated (full semantic correctness)
4) AST gets processed to perform the update
This commit uses the tokenlist to build an expression tree. This tree is not
yet used. Still it allows to raise additional Validation Exceptions which
previously were missed silently therefore it allows tests to catch these type of
ValidationException. For that reason DDB UpdateExpressions will be parsed
already. It also makes sure we won't break existing tests.
One of the existing tests had to be changed in order to still pass:
- test_dynamodb_table_with_range_key.test_update_item_with_expression
This test passed in a numeric literal which is not supported by DynamoDB
and with the current tokenization it would get the same error as in AWS
DynamoDB.
Currently the mock for DynamoDB has adhoc code to implement
its updateExpression functionality. This series will
transform the logic such that Update Expressions are processed
as follows:
1) Expression gets parsed into a tokenlist (tokenized) -> This commit
2) Tokenlist get transformed to expression tree (AST)
3) The AST gets validated (full semantic correctness)
4) AST gets processed to perform the update
This alows for a more realistic mocking. It will throw exceptions much
more aggressively avoiding situations where a test passes against the
mock but fails with an exception when running against AWS.
Introduction of step 3 also allows to have the update expression as an
atomic unit of work. So updates at the start of the expression cannot
be performed if there is an error further down the expression.
This specific commit will tokenize expressions but the tokenlist is not
yet used. It is purely to keep clear boundaries. It does do a minor
refactoring of the exceptions to allow more re-use and to ease testing.
This series of changes is to aid providing a long-term solution for
https://github.com/spulec/moto/issues/2806.
By having models.py as one big file it causes to
easily create circular dependencies. With the current
setup it is not possible to re-use DynamoType. This
refactor moves it out to its own file while trying to
keep the structure as much as it is.
