Yasuke/moto
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'master' into dynamodb-gsi-projectiontype

Browse Source
This commit is contained in:
Bert Blommers 2020-07-03 14:08:32 +01:00
commit 06b390b493
8 changed files with 198 additions and 99 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

178
README.md
View File

@ -58,95 +58,95 @@ With the decorator wrapping the test, all the calls to s3 are automatically mock
It gets even better! Moto isn't just for Python code and it isn't just for S3. Look at the [standalone server mode](https://github.com/spulec/moto#stand-alone-server-mode) for more information about running Moto with other languages. Here's the status of the other AWS services implemented: It gets even better! Moto isn't just for Python code and it isn't just for S3. Look at the [standalone server mode](https://github.com/spulec/moto#stand-alone-server-mode) for more information about running Moto with other languages. Here's the status of the other AWS services implemented:
```gherkin ```gherkin
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------|-----------------------------|
| Service Name | Decorator | Development Status | | Service Name | Decorator | Development Status | Comment |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| ACM | @mock_acm | all endpoints done | | ACM | @mock_acm | all endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| API Gateway | @mock_apigateway | core endpoints done | | API Gateway | @mock_apigateway | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Autoscaling | @mock_autoscaling | core endpoints done | | Autoscaling | @mock_autoscaling | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Cloudformation | @mock_cloudformation | core endpoints done | | Cloudformation | @mock_cloudformation | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Cloudwatch | @mock_cloudwatch | basic endpoints done | | Cloudwatch | @mock_cloudwatch | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| CloudwatchEvents | @mock_events | all endpoints done | | CloudwatchEvents | @mock_events | all endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Cognito Identity | @mock_cognitoidentity | basic endpoints done | | Cognito Identity | @mock_cognitoidentity | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Cognito Identity Provider | @mock_cognitoidp | basic endpoints done | | Cognito Identity Provider | @mock_cognitoidp | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Config | @mock_config | basic endpoints done | | Config | @mock_config | basic endpoints done | |
| | | core endpoints done | | | | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Data Pipeline | @mock_datapipeline | basic endpoints done | | Data Pipeline | @mock_datapipeline | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| DynamoDB | @mock_dynamodb | core endpoints done | | DynamoDB | @mock_dynamodb | core endpoints done | API 20111205. Deprecated. |
| DynamoDB2 | @mock_dynamodb2 | all endpoints + partial indexes | | DynamoDB2 | @mock_dynamodb2 | all endpoints + partial indexes | API 20120810 (Latest) |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| EC2 | @mock_ec2 | core endpoints done | | EC2 | @mock_ec2 | core endpoints done | |
| - AMI | | core endpoints done | | - AMI | | core endpoints done | |
| - EBS | | core endpoints done | | - EBS | | core endpoints done | |
| - Instances | | all endpoints done | | - Instances | | all endpoints done | |
| - Security Groups | | core endpoints done | | - Security Groups | | core endpoints done | |
| - Tags | | all endpoints done | | - Tags | | all endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| ECR | @mock_ecr | basic endpoints done | | ECR | @mock_ecr | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| ECS | @mock_ecs | basic endpoints done | | ECS | @mock_ecs | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| ELB | @mock_elb | core endpoints done | | ELB | @mock_elb | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| ELBv2 | @mock_elbv2 | all endpoints done | | ELBv2 | @mock_elbv2 | all endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| EMR | @mock_emr | core endpoints done | | EMR | @mock_emr | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Glacier | @mock_glacier | core endpoints done | | Glacier | @mock_glacier | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| IAM | @mock_iam | core endpoints done | | IAM | @mock_iam | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| IoT | @mock_iot | core endpoints done | | IoT | @mock_iot | core endpoints done | |
| | @mock_iotdata | core endpoints done | | | @mock_iotdata | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Kinesis | @mock_kinesis | core endpoints done | | Kinesis | @mock_kinesis | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| KMS | @mock_kms | basic endpoints done | | KMS | @mock_kms | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Lambda | @mock_lambda | basic endpoints done, requires | | Lambda | @mock_lambda | basic endpoints done, requires | |
| | | docker | | | | docker | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Logs | @mock_logs | basic endpoints done | | Logs | @mock_logs | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Organizations | @mock_organizations | some core endpoints done | | Organizations | @mock_organizations | some core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Polly | @mock_polly | all endpoints done | | Polly | @mock_polly | all endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| RDS | @mock_rds | core endpoints done | | RDS | @mock_rds | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| RDS2 | @mock_rds2 | core endpoints done | | RDS2 | @mock_rds2 | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Redshift | @mock_redshift | core endpoints done | | Redshift | @mock_redshift | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| Route53 | @mock_route53 | core endpoints done | | Route53 | @mock_route53 | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| S3 | @mock_s3 | core endpoints done | | S3 | @mock_s3 | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| SecretsManager | @mock_secretsmanager | basic endpoints done | | SecretsManager | @mock_secretsmanager | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| SES | @mock_ses | all endpoints done | | SES | @mock_ses | all endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| SNS | @mock_sns | all endpoints done | | SNS | @mock_sns | all endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| SQS | @mock_sqs | core endpoints done | | SQS | @mock_sqs | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| SSM | @mock_ssm | core endpoints done | | SSM | @mock_ssm | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| STS | @mock_sts | core endpoints done | | STS | @mock_sts | core endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| SWF | @mock_swf | basic endpoints done | | SWF | @mock_swf | basic endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------| |
| X-Ray | @mock_xray | all endpoints done | | X-Ray | @mock_xray | all endpoints done | |
|-------------------------------------------------------------------------------------| |-------------------------------------------------------------------------------------|
``` ```

13
moto/cloudformation/responses.py
View File

@ -351,18 +351,21 @@ class CloudFormationResponse(BaseResponse):
return template.render(exports=exports, next_token=next_token) return template.render(exports=exports, next_token=next_token)
def validate_template(self): def validate_template(self):
cfn_lint = self.cloudformation_backend.validate_template( template_body = self._get_param("TemplateBody")
self._get_param("TemplateBody") template_url = self._get_param("TemplateURL")
) if template_url:
template_body = self._get_stack_from_s3_url(template_url)
cfn_lint = self.cloudformation_backend.validate_template(template_body)
if cfn_lint: if cfn_lint:
raise ValidationError(cfn_lint[0].message) raise ValidationError(cfn_lint[0].message)
description = "" description = ""
try: try:
description = json.loads(self._get_param("TemplateBody"))["Description"] description = json.loads(template_body)["Description"]
except (ValueError, KeyError): except (ValueError, KeyError):
pass pass
try: try:
description = yaml.load(self._get_param("TemplateBody"))["Description"] description = yaml.load(template_body)["Description"]
except (yaml.ParserError, KeyError): except (yaml.ParserError, KeyError):
pass pass
template = self.response_template(VALIDATE_STACK_RESPONSE_TEMPLATE) template = self.response_template(VALIDATE_STACK_RESPONSE_TEMPLATE)

16
moto/core/models.py
View File

@ -10,6 +10,7 @@ import six
import types import types
from io import BytesIO from io import BytesIO
from collections import defaultdict from collections import defaultdict
from botocore.config import Config
from botocore.handlers import BUILTIN_HANDLERS from botocore.handlers import BUILTIN_HANDLERS
from botocore.awsrequest import AWSResponse from botocore.awsrequest import AWSResponse
from six.moves.urllib.parse import urlparse from six.moves.urllib.parse import urlparse
@ -416,6 +417,13 @@ class ServerModeMockAWS(BaseMockAWS):
import mock import mock
def fake_boto3_client(*args, **kwargs): def fake_boto3_client(*args, **kwargs):
region = self._get_region(*args, **kwargs)
if region:
if "config" in kwargs:
kwargs["config"].__dict__["user_agent_extra"] += " region/" + region
else:
config = Config(user_agent_extra="region/" + region)
kwargs["config"] = config
if "endpoint_url" not in kwargs: if "endpoint_url" not in kwargs:
kwargs["endpoint_url"] = "http://localhost:5000" kwargs["endpoint_url"] = "http://localhost:5000"
return real_boto3_client(*args, **kwargs) return real_boto3_client(*args, **kwargs)
@ -463,6 +471,14 @@ class ServerModeMockAWS(BaseMockAWS):
if six.PY2: if six.PY2:
self._httplib_patcher.start() self._httplib_patcher.start()
def _get_region(self, *args, **kwargs):
if "region_name" in kwargs:
return kwargs["region_name"]
if type(args) == tuple and len(args) == 2:
service, region = args
return region
return None
def disable_patching(self): def disable_patching(self):
if self._client_patcher: if self._client_patcher:
self._client_patcher.stop() self._client_patcher.stop()

14
moto/core/responses.py
View File

@ -188,6 +188,9 @@ class BaseResponse(_TemplateEnvironmentMixin, ActionAuthenticatorMixin):
default_region = "us-east-1" default_region = "us-east-1"
# to extract region, use [^.] # to extract region, use [^.]
region_regex = re.compile(r"\.(?P<region>[a-z]{2}-[a-z]+-\d{1})\.amazonaws\.com") region_regex = re.compile(r"\.(?P<region>[a-z]{2}-[a-z]+-\d{1})\.amazonaws\.com")
region_from_useragent_regex = re.compile(
r"region/(?P<region>[a-z]{2}-[a-z]+-\d{1})"
)
param_list_regex = re.compile(r"(.*)\.(\d+)\.") param_list_regex = re.compile(r"(.*)\.(\d+)\.")
access_key_regex = re.compile( access_key_regex = re.compile(
r"AWS.*(?P<access_key>(?<![A-Z0-9])[A-Z0-9]{20}(?![A-Z0-9]))[:/]" r"AWS.*(?P<access_key>(?<![A-Z0-9])[A-Z0-9]{20}(?![A-Z0-9]))[:/]"
@ -272,9 +275,14 @@ class BaseResponse(_TemplateEnvironmentMixin, ActionAuthenticatorMixin):
self.response_headers = {"server": "amazon.com"} self.response_headers = {"server": "amazon.com"}
def get_region_from_url(self, request, full_url): def get_region_from_url(self, request, full_url):
match = self.region_regex.search(full_url) url_match = self.region_regex.search(full_url)
if match: user_agent_match = self.region_from_useragent_regex.search(
region = match.group(1) request.headers.get("User-Agent", "")
)
if url_match:
region = url_match.group(1)
elif user_agent_match:
region = user_agent_match.group(1)
elif ( elif (
"Authorization" in request.headers "Authorization" in request.headers
and "AWS4" in request.headers["Authorization"] and "AWS4" in request.headers["Authorization"]

4
moto/elbv2/responses.py
View File

@ -158,7 +158,7 @@ class ELBV2Response(BaseResponse):
condition = {} condition = {}
condition["field"] = _condition["field"] condition["field"] = _condition["field"]
values = sorted( values = sorted(
[e for e in _condition.items() if e[0].startswith("values.member")], [e for e in _condition.items() if "values.member" in e[0]],
key=lambda x: x[0], key=lambda x: x[0],
) )
condition["values"] = [e[1] for e in values] condition["values"] = [e[1] for e in values]
@ -356,7 +356,7 @@ class ELBV2Response(BaseResponse):
condition = {} condition = {}
condition["field"] = _condition["field"] condition["field"] = _condition["field"]
values = sorted( values = sorted(
[e for e in _condition.items() if e[0].startswith("values.member")], [e for e in _condition.items() if "values.member" in e[0]],
key=lambda x: x[0], key=lambda x: x[0],
) )
condition["values"] = [e[1] for e in values] condition["values"] = [e[1] for e in values]

19
tests/test_cloudformation/test_validate.py
View File

@ -96,6 +96,25 @@ def test_boto3_yaml_validate_successful():
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200 assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
@mock_cloudformation
@mock_s3
def test_boto3_yaml_validate_template_url_successful():
s3 = boto3.client("s3")
s3_conn = boto3.resource("s3", region_name="us-east-1")
s3_conn.create_bucket(Bucket="foobar")
s3_conn.Object("foobar", "template-key").put(Body=yaml_template)
key_url = s3.generate_presigned_url(
ClientMethod="get_object", Params={"Bucket": "foobar", "Key": "template-key"}
)
cf_conn = boto3.client("cloudformation", region_name="us-east-1")
response = cf_conn.validate_template(TemplateURL=key_url)
assert response["Description"] == "Simple CloudFormation Test Template"
assert response["Parameters"] == []
assert response["ResponseMetadata"]["HTTPStatusCode"] == 200
@mock_cloudformation @mock_cloudformation
def test_boto3_yaml_invalid_missing_resource(): def test_boto3_yaml_invalid_missing_resource():
cf_conn = boto3.client("cloudformation", region_name="us-east-1") cf_conn = boto3.client("cloudformation", region_name="us-east-1")

32
tests/test_cognitoidp/test_cognitoidp.py
View File

@ -1243,6 +1243,38 @@ def test_change_password():
result["AuthenticationResult"].should_not.be.none result["AuthenticationResult"].should_not.be.none
@mock_cognitoidp
def test_change_password__using_custom_user_agent_header():
# https://github.com/spulec/moto/issues/3098
# As the admin_initiate_auth-method is unauthenticated, we use the user-agent header to pass in the region
# This test verifies this works, even if we pass in our own user-agent header
from botocore.config import Config
my_config = Config(user_agent_extra="more/info", signature_version="v4")
conn = boto3.client("cognito-idp", "us-west-2", config=my_config)
outputs = authentication_flow(conn)
# Take this opportunity to test change_password, which requires an access token.
newer_password = str(uuid.uuid4())
conn.change_password(
AccessToken=outputs["access_token"],
PreviousPassword=outputs["password"],
ProposedPassword=newer_password,
)
# Log in again, which should succeed without a challenge because the user is no
# longer in the force-new-password state.
result = conn.admin_initiate_auth(
UserPoolId=outputs["user_pool_id"],
ClientId=outputs["client_id"],
AuthFlow="ADMIN_NO_SRP_AUTH",
AuthParameters={"USERNAME": outputs["username"], "PASSWORD": newer_password},
)
result["AuthenticationResult"].should_not.be.none
@mock_cognitoidp @mock_cognitoidp
def test_forgot_password(): def test_forgot_password():
conn = boto3.client("cognito-idp", "us-west-2") conn = boto3.client("cognito-idp", "us-west-2")

21
tests/test_elbv2/test_elbv2.py
View File

@ -994,12 +994,17 @@ def test_handle_listener_rules():
priority = 100 priority = 100
host = "xxx.example.com" host = "xxx.example.com"
path_pattern = "foobar" path_pattern = "foobar"
pathpatternconfig_pattern = "foobar2"
created_rule = conn.create_rule( created_rule = conn.create_rule(
ListenerArn=http_listener_arn, ListenerArn=http_listener_arn,
Priority=priority, Priority=priority,
Conditions=[ Conditions=[
{"Field": "host-header", "Values": [host]}, {"Field": "host-header", "Values": [host]},
{"Field": "path-pattern", "Values": [path_pattern]}, {"Field": "path-pattern", "Values": [path_pattern]},
{
"Field": "path-pattern",
"PathPatternConfig": {"Values": [pathpatternconfig_pattern]},
},
], ],
Actions=[ Actions=[
{"TargetGroupArn": target_group.get("TargetGroupArn"), "Type": "forward"} {"TargetGroupArn": target_group.get("TargetGroupArn"), "Type": "forward"}
@ -1017,6 +1022,10 @@ def test_handle_listener_rules():
Conditions=[ Conditions=[
{"Field": "host-header", "Values": [host]}, {"Field": "host-header", "Values": [host]},
{"Field": "path-pattern", "Values": [path_pattern]}, {"Field": "path-pattern", "Values": [path_pattern]},
{
"Field": "path-pattern",
"PathPatternConfig": {"Values": [pathpatternconfig_pattern]},
},
], ],
Actions=[ Actions=[
{"TargetGroupArn": target_group.get("TargetGroupArn"), "Type": "forward"} {"TargetGroupArn": target_group.get("TargetGroupArn"), "Type": "forward"}
@ -1031,6 +1040,10 @@ def test_handle_listener_rules():
Conditions=[ Conditions=[
{"Field": "host-header", "Values": [host]}, {"Field": "host-header", "Values": [host]},
{"Field": "path-pattern", "Values": [path_pattern]}, {"Field": "path-pattern", "Values": [path_pattern]},
{
"Field": "path-pattern",
"PathPatternConfig": {"Values": [pathpatternconfig_pattern]},
},
], ],
Actions=[ Actions=[
{ {
@ -1079,11 +1092,16 @@ def test_handle_listener_rules():
# modify rule partially # modify rule partially
new_host = "new.example.com" new_host = "new.example.com"
new_path_pattern = "new_path" new_path_pattern = "new_path"
new_pathpatternconfig_pattern = "new_path2"
modified_rule = conn.modify_rule( modified_rule = conn.modify_rule(
RuleArn=first_rule["RuleArn"], RuleArn=first_rule["RuleArn"],
Conditions=[ Conditions=[
{"Field": "host-header", "Values": [new_host]}, {"Field": "host-header", "Values": [new_host]},
{"Field": "path-pattern", "Values": [new_path_pattern]}, {"Field": "path-pattern", "Values": [new_path_pattern]},
{
"Field": "path-pattern",
"PathPatternConfig": {"Values": [new_pathpatternconfig_pattern]},
},
], ],
)["Rules"][0] )["Rules"][0]
@ -1092,6 +1110,9 @@ def test_handle_listener_rules():
modified_rule.should.equal(obtained_rule) modified_rule.should.equal(obtained_rule)
obtained_rule["Conditions"][0]["Values"][0].should.equal(new_host) obtained_rule["Conditions"][0]["Values"][0].should.equal(new_host)
obtained_rule["Conditions"][1]["Values"][0].should.equal(new_path_pattern) obtained_rule["Conditions"][1]["Values"][0].should.equal(new_path_pattern)
obtained_rule["Conditions"][2]["Values"][0].should.equal(
new_pathpatternconfig_pattern
)
obtained_rule["Actions"][0]["TargetGroupArn"].should.equal( obtained_rule["Actions"][0]["TargetGroupArn"].should.equal(
target_group.get("TargetGroupArn") target_group.get("TargetGroupArn")
) )