Yasuke/moto
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
7693d77333
moto/tests/test_emr/test_emr.py

664 lines
23 KiB
Python
Raw Normal View History

from __future__ import unicode_literals
2014-08-27 15:17:06 +00:00
from __future__ import unicode_literals
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
import time
from datetime import datetime
feature emr: set_termination_protection call implementation feature emr: support for instance_groups param in run_jobflow
2015-07-08 07:26:58 +00:00
basic emr done
2013-08-08 00:32:29 +00:00
import boto
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
import pytz
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
from boto.emr.bootstrap_action import BootstrapAction
basic emr done
2013-08-08 00:32:29 +00:00
from boto.emr.instance_group import InstanceGroup
from boto.emr.step import StreamingStep
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
basic emr done
2013-08-08 00:32:29 +00:00
import sure # noqa
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
from moto import mock_emr_deprecated
job_flow_role is not available in boto 2.7
2013-08-08 14:55:02 +00:00
from tests.helpers import requires_boto_gte
basic emr done
2013-08-08 00:32:29 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
run_jobflow_args = dict(
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_flow_role="EMR_EC2_DefaultRole",
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
keep_alive=True,
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
log_uri="s3://some_bucket/jobflow_logs",
master_instance_type="c1.medium",
name="My jobflow",
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
num_instances=2,
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
service_role="EMR_DefaultRole",
slave_instance_type="c1.medium",
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
)
basic emr done
2013-08-08 00:32:29 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
input_instance_groups = [
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
InstanceGroup(1, "MASTER", "c1.medium", "ON_DEMAND", "master"),
InstanceGroup(3, "CORE", "c1.medium", "ON_DEMAND", "core"),
InstanceGroup(6, "TASK", "c1.large", "SPOT", "task-1", "0.07"),
InstanceGroup(10, "TASK", "c1.xlarge", "SPOT", "task-2", "0.05"),
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
]
basic emr done
2013-08-08 00:32:29 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_describe_cluster():
job_flow_role is not available in boto 2.7
2013-08-08 14:55:02 +00:00
conn = boto.connect_emr()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
args = run_jobflow_args.copy()
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
args.update(
dict(
api_params={
"Applications.member.1.Name": "Spark",
"Applications.member.1.Version": "2.4.2",
"Configurations.member.1.Classification": "yarn-site",
"Configurations.member.1.Properties.entry.1.key": "someproperty",
"Configurations.member.1.Properties.entry.1.value": "somevalue",
"Configurations.member.1.Properties.entry.2.key": "someotherproperty",
"Configurations.member.1.Properties.entry.2.value": "someothervalue",
"Instances.EmrManagedMasterSecurityGroup": "master-security-group",
"Instances.Ec2SubnetId": "subnet-8be41cec",
},
availability_zone="us-east-2b",
ec2_keyname="mykey",
job_flow_role="EMR_EC2_DefaultRole",
keep_alive=False,
log_uri="s3://some_bucket/jobflow_logs",
name="My jobflow",
service_role="EMR_DefaultRole",
visible_to_all_users=True,
)
)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
cluster_id = conn.run_jobflow(**args)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
input_tags = {"tag1": "val1", "tag2": "val2"}
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
conn.add_tags(cluster_id, input_tags)
Add visibletoallusers and normalizedinstancehours to EMR
2013-08-11 00:03:57 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
cluster = conn.describe_cluster(cluster_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster.applications[0].name.should.equal("Spark")
cluster.applications[0].version.should.equal("2.4.2")
cluster.autoterminate.should.equal("true")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# configurations appear not be supplied as attributes?
attrs = cluster.ec2instanceattributes
# AdditionalMasterSecurityGroups
# AdditionalSlaveSecurityGroups
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
attrs.ec2availabilityzone.should.equal(args["availability_zone"])
attrs.ec2keyname.should.equal(args["ec2_keyname"])
attrs.ec2subnetid.should.equal(args["api_params"]["Instances.Ec2SubnetId"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# EmrManagedMasterSecurityGroups
# EmrManagedSlaveSecurityGroups
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
attrs.iaminstanceprofile.should.equal(args["job_flow_role"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# ServiceAccessSecurityGroup
cluster.id.should.equal(cluster_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster.loguri.should.equal(args["log_uri"])
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
cluster.masterpublicdnsname.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster.name.should.equal(args["name"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
int(cluster.normalizedinstancehours).should.equal(0)
# cluster.release_label
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster.shouldnt.have.property("requestedamiversion")
cluster.runningamiversion.should.equal("1.0.0")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# cluster.securityconfiguration
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster.servicerole.should.equal(args["service_role"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster.status.state.should.equal("TERMINATED")
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
cluster.status.statechangereason.message.should.be.a(str)
cluster.status.statechangereason.code.should.be.a(str)
cluster.status.timeline.creationdatetime.should.be.a(str)
# cluster.status.timeline.enddatetime.should.be.a(str)
# cluster.status.timeline.readydatetime.should.be.a(str)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
dict((item.key, item.value) for item in cluster.tags).should.equal(input_tags)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster.terminationprotected.should.equal("false")
cluster.visibletoallusers.should.equal("true")
basic emr done
2013-08-08 00:32:29 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_describe_jobflows():
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
conn = boto.connect_emr()
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
args = run_jobflow_args.copy()
expected = {}
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
Get standalone server mode working for all tests.
2017-02-24 00:43:48 +00:00
for idx in range(4):
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster_name = "cluster" + str(idx)
args["name"] = cluster_name
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
cluster_id = conn.run_jobflow(**args)
expected[cluster_id] = {
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
"id": cluster_id,
"name": cluster_name,
"state": "WAITING",
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
}
# need sleep since it appears the timestamp is always rounded to
# the nearest second internally
time.sleep(1)
timestamp = datetime.now(pytz.utc)
time.sleep(1)
Get standalone server mode working for all tests.
2017-02-24 00:43:48 +00:00
for idx in range(4, 6):
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster_name = "cluster" + str(idx)
args["name"] = cluster_name
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
cluster_id = conn.run_jobflow(**args)
conn.terminate_jobflow(cluster_id)
expected[cluster_id] = {
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
"id": cluster_id,
"name": cluster_name,
"state": "TERMINATED",
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
}
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
jobs = conn.describe_jobflows()
Get standalone server mode working for all tests.
2017-02-24 00:43:48 +00:00
jobs.should.have.length_of(6)
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
for cluster_id, y in expected.items():
resp = conn.describe_jobflows(jobflow_ids=[cluster_id])
resp.should.have.length_of(1)
resp[0].jobflowid.should.equal(cluster_id)
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
resp = conn.describe_jobflows(states=["WAITING"])
Get standalone server mode working for all tests.
2017-02-24 00:43:48 +00:00
resp.should.have.length_of(4)
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
for x in resp:
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.state.should.equal("WAITING")
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
resp = conn.describe_jobflows(created_before=timestamp)
Get standalone server mode working for all tests.
2017-02-24 00:43:48 +00:00
resp.should.have.length_of(4)
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
resp = conn.describe_jobflows(created_after=timestamp)
Get standalone server mode working for all tests.
2017-02-24 00:43:48 +00:00
resp.should.have.length_of(2)
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_describe_jobflow():
basic emr done
2013-08-08 00:32:29 +00:00
conn = boto.connect_emr()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
args = run_jobflow_args.copy()
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
args.update(
dict(
ami_version="3.8.1",
api_params={
#'Applications.member.1.Name': 'Spark',
#'Applications.member.1.Version': '2.4.2',
#'Configurations.member.1.Classification': 'yarn-site',
#'Configurations.member.1.Properties.entry.1.key': 'someproperty',
#'Configurations.member.1.Properties.entry.1.value': 'somevalue',
#'Instances.EmrManagedMasterSecurityGroup': 'master-security-group',
"Instances.Ec2SubnetId": "subnet-8be41cec"
},
ec2_keyname="mykey",
hadoop_version="2.4.0",
name="My jobflow",
log_uri="s3://some_bucket/jobflow_logs",
keep_alive=True,
master_instance_type="c1.medium",
slave_instance_type="c1.medium",
num_instances=2,
availability_zone="us-west-2b",
job_flow_role="EMR_EC2_DefaultRole",
service_role="EMR_DefaultRole",
visible_to_all_users=True,
)
)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
cluster_id = conn.run_jobflow(**args)
jf = conn.describe_jobflow(cluster_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.amiversion.should.equal(args["ami_version"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
jf.bootstrapactions.should.equal(None)
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
jf.creationdatetime.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.should.have.property("laststatechangereason")
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
jf.readydatetime.should.be.a(str)
jf.startdatetime.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.state.should.equal("WAITING")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.ec2keyname.should.equal(args["ec2_keyname"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# Ec2SubnetId
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.hadoopversion.should.equal(args["hadoop_version"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
int(jf.instancecount).should.equal(2)
for ig in jf.instancegroups:
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
ig.creationdatetime.should.be.a(str)
# ig.enddatetime.should.be.a(str)
ig.should.have.property("instancegroupid").being.a(str)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
int(ig.instancerequestcount).should.equal(1)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
ig.instancerole.should.be.within(["MASTER", "CORE"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
int(ig.instancerunningcount).should.equal(1)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
ig.instancetype.should.equal("c1.medium")
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
ig.laststatechangereason.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
ig.market.should.equal("ON_DEMAND")
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
ig.name.should.be.a(str)
ig.readydatetime.should.be.a(str)
ig.startdatetime.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
ig.state.should.equal("RUNNING")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.keepjobflowalivewhennosteps.should.equal("true")
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
jf.masterinstanceid.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.masterinstancetype.should.equal(args["master_instance_type"])
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
jf.masterpublicdnsname.should.be.a(str)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
int(jf.normalizedinstancehours).should.equal(0)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.availabilityzone.should.equal(args["availability_zone"])
jf.slaveinstancetype.should.equal(args["slave_instance_type"])
jf.terminationprotected.should.equal("false")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
jf.jobflowid.should.equal(cluster_id)
# jf.jobflowrole.should.equal(args['job_flow_role'])
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.loguri.should.equal(args["log_uri"])
jf.name.should.equal(args["name"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# jf.servicerole.should.equal(args['service_role'])
jf.steps.should.have.length_of(0)
list(i.value for i in jf.supported_products).should.equal([])
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
jf.visibletoallusers.should.equal("true")
basic emr done
2013-08-08 00:32:29 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_list_clusters():
basic emr done
2013-08-08 00:32:29 +00:00
conn = boto.connect_emr()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
args = run_jobflow_args.copy()
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
expected = {}
for idx in range(40):
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster_name = "jobflow" + str(idx)
args["name"] = cluster_name
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
cluster_id = conn.run_jobflow(**args)
expected[cluster_id] = {
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
"id": cluster_id,
"name": cluster_name,
"normalizedinstancehours": "0",
"state": "WAITING",
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
}
# need sleep since it appears the timestamp is always rounded to
# the nearest second internally
time.sleep(1)
timestamp = datetime.now(pytz.utc)
time.sleep(1)
for idx in range(40, 70):
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster_name = "jobflow" + str(idx)
args["name"] = cluster_name
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
cluster_id = conn.run_jobflow(**args)
conn.terminate_jobflow(cluster_id)
expected[cluster_id] = {
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
"id": cluster_id,
"name": cluster_name,
"normalizedinstancehours": "0",
"state": "TERMINATED",
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
}
args = {}
while 1:
resp = conn.list_clusters(**args)
clusters = resp.clusters
len(clusters).should.be.lower_than_or_equal_to(50)
for x in clusters:
y = expected[x.id]
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.id.should.equal(y["id"])
x.name.should.equal(y["name"])
x.normalizedinstancehours.should.equal(y["normalizedinstancehours"])
x.status.state.should.equal(y["state"])
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.status.timeline.creationdatetime.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
if y["state"] == "TERMINATED":
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.status.timeline.enddatetime.should.be.a(str)
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
else:
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.status.timeline.shouldnt.have.property("enddatetime")
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.status.timeline.readydatetime.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
if not hasattr(resp, "marker"):
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
break
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
args = {"marker": resp.marker}
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
resp = conn.list_clusters(cluster_states=["TERMINATED"])
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
resp.clusters.should.have.length_of(30)
for x in resp.clusters:
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.status.state.should.equal("TERMINATED")
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
resp = conn.list_clusters(created_before=timestamp)
resp.clusters.should.have.length_of(40)
resp = conn.list_clusters(created_after=timestamp)
resp.clusters.should.have.length_of(30)
basic emr done
2013-08-08 00:32:29 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_run_jobflow():
basic emr done
2013-08-08 00:32:29 +00:00
conn = boto.connect_emr()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
args = run_jobflow_args.copy()
job_id = conn.run_jobflow(**args)
job_flow = conn.describe_jobflow(job_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_flow.state.should.equal("WAITING")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
job_flow.jobflowid.should.equal(job_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_flow.name.should.equal(args["name"])
job_flow.masterinstancetype.should.equal(args["master_instance_type"])
job_flow.slaveinstancetype.should.equal(args["slave_instance_type"])
job_flow.loguri.should.equal(args["log_uri"])
job_flow.visibletoallusers.should.equal("false")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
int(job_flow.normalizedinstancehours).should.equal(0)
job_flow.steps.should.have.length_of(0)
basic emr done
2013-08-08 00:32:29 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_run_jobflow_in_multiple_regions():
regions = {}
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
for region in ["us-east-1", "eu-west-1"]:
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
conn = boto.emr.connect_to_region(region)
args = run_jobflow_args.copy()
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
args["name"] = region
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
cluster_id = conn.run_jobflow(**args)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
regions[region] = {"conn": conn, "cluster_id": cluster_id}
basic emr done
2013-08-08 00:32:29 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
for region in regions.keys():
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
conn = regions[region]["conn"]
jf = conn.describe_jobflow(regions[region]["cluster_id"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
jf.name.should.equal(region)
Add visibletoallusers and normalizedinstancehours to EMR
2013-08-11 00:03:57 +00:00
Tests: Tweaked two tests to be version-gated to Boto 2.8+.
2014-09-10 20:29:45 +00:00
@requires_boto_gte("2.8")
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_run_jobflow_with_new_params():
# Test that run_jobflow works with newer params
Add visibletoallusers and normalizedinstancehours to EMR
2013-08-11 00:03:57 +00:00
conn = boto.connect_emr()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
conn.run_jobflow(**run_jobflow_args)
Add visibletoallusers and normalizedinstancehours to EMR
2013-08-11 00:03:57 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
@requires_boto_gte("2.8")
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_run_jobflow_with_visible_to_all_users():
conn = boto.connect_emr()
for expected in (True, False):
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_id = conn.run_jobflow(visible_to_all_users=expected, **run_jobflow_args)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
job_flow = conn.describe_jobflow(job_id)
job_flow.visibletoallusers.should.equal(str(expected).lower())
Add visibletoallusers and normalizedinstancehours to EMR
2013-08-11 00:03:57 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
@requires_boto_gte("2.8")
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_run_jobflow_with_instance_groups():
input_groups = dict((g.name, g) for g in input_instance_groups)
conn = boto.connect_emr()
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_id = conn.run_jobflow(instance_groups=input_instance_groups, **run_jobflow_args)
Add visibletoallusers and normalizedinstancehours to EMR
2013-08-11 00:03:57 +00:00
job_flow = conn.describe_jobflow(job_id)
Lints.
2017-02-24 02:37:43 +00:00
int(job_flow.instancecount).should.equal(
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
sum(g.num_instances for g in input_instance_groups)
)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
for instance_group in job_flow.instancegroups:
expected = input_groups[instance_group.name]
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
instance_group.should.have.property("instancegroupid")
int(instance_group.instancerunningcount).should.equal(expected.num_instances)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
instance_group.instancerole.should.equal(expected.role)
instance_group.instancetype.should.equal(expected.type)
instance_group.market.should.equal(expected.market)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
if hasattr(expected, "bidprice"):
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
instance_group.bidprice.should.equal(expected.bidprice)
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
feature emr: set_termination_protection call implementation feature emr: support for instance_groups param in run_jobflow
2015-07-08 07:26:58 +00:00
@requires_boto_gte("2.8")
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
feature emr: set_termination_protection call implementation feature emr: support for instance_groups param in run_jobflow
2015-07-08 07:26:58 +00:00
def test_set_termination_protection():
conn = boto.connect_emr()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
job_id = conn.run_jobflow(**run_jobflow_args)
feature emr: set_termination_protection call implementation feature emr: support for instance_groups param in run_jobflow
2015-07-08 07:26:58 +00:00
job_flow = conn.describe_jobflow(job_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_flow.terminationprotected.should.equal("false")
feature emr: set_termination_protection call implementation feature emr: support for instance_groups param in run_jobflow
2015-07-08 07:26:58 +00:00
conn.set_termination_protection(job_id, True)
job_flow = conn.describe_jobflow(job_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_flow.terminationprotected.should.equal("true")
feature emr: set_termination_protection call implementation feature emr: support for instance_groups param in run_jobflow
2015-07-08 07:26:58 +00:00
conn.set_termination_protection(job_id, False)
job_flow = conn.describe_jobflow(job_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_flow.terminationprotected.should.equal("false")
feature emr: set_termination_protection call implementation feature emr: support for instance_groups param in run_jobflow
2015-07-08 07:26:58 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
@requires_boto_gte("2.8")
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_set_visible_to_all_users():
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
conn = boto.connect_emr()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
args = run_jobflow_args.copy()
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
args["visible_to_all_users"] = False
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
job_id = conn.run_jobflow(**args)
job_flow = conn.describe_jobflow(job_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_flow.visibletoallusers.should.equal("false")
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
conn.set_visible_to_all_users(job_id, True)
job_flow = conn.describe_jobflow(job_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_flow.visibletoallusers.should.equal("true")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
conn.set_visible_to_all_users(job_id, False)
job_flow = conn.describe_jobflow(job_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
job_flow.visibletoallusers.should.equal("false")
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_terminate_jobflow():
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
conn = boto.connect_emr()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
job_id = conn.run_jobflow(**run_jobflow_args)
flow = conn.describe_jobflows()[0]
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
flow.state.should.equal("WAITING")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
conn.terminate_jobflow(job_id)
flow = conn.describe_jobflows()[0]
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
flow.state.should.equal("TERMINATED")
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# testing multiple end points for each feature
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_bootstrap_actions():
bootstrap_actions = [
BootstrapAction(
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
name="bs1",
path="path/to/script",
bootstrap_action_args=["arg1", "arg2&arg3"],
),
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
BootstrapAction(
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
name="bs2", path="path/to/anotherscript", bootstrap_action_args=[]
),
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
]
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
conn = boto.connect_emr()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
cluster_id = conn.run_jobflow(
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
bootstrap_actions=bootstrap_actions, **run_jobflow_args
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
jf = conn.describe_jobflow(cluster_id)
for x, y in zip(jf.bootstrapactions, bootstrap_actions):
x.name.should.equal(y.name)
x.path.should.equal(y.path)
list(o.value for o in x.args).should.equal(y.args())
resp = conn.list_bootstrap_actions(cluster_id)
for i, y in enumerate(bootstrap_actions):
x = resp.actions[i]
x.name.should.equal(y.name)
x.scriptpath.should.equal(y.path)
list(arg.value for arg in x.args).should.equal(y.args())
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_instance_groups():
input_groups = dict((g.name, g) for g in input_instance_groups)
conn = boto.connect_emr()
args = run_jobflow_args.copy()
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
for key in ["master_instance_type", "slave_instance_type", "num_instances"]:
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
del args[key]
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
args["instance_groups"] = input_instance_groups[:2]
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
job_id = conn.run_jobflow(**args)
jf = conn.describe_jobflow(job_id)
base_instance_count = int(jf.instancecount)
conn.add_instance_groups(job_id, input_instance_groups[2:])
jf = conn.describe_jobflow(job_id)
Lints.
2017-02-24 02:37:43 +00:00
int(jf.instancecount).should.equal(
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
sum(g.num_instances for g in input_instance_groups)
)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
for x in jf.instancegroups:
y = input_groups[x.name]
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
if hasattr(y, "bidprice"):
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
x.bidprice.should.equal(y.bidprice)
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.creationdatetime.should.be.a(str)
# x.enddatetime.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.should.have.property("instancegroupid")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
int(x.instancerequestcount).should.equal(y.num_instances)
x.instancerole.should.equal(y.role)
int(x.instancerunningcount).should.equal(y.num_instances)
x.instancetype.should.equal(y.type)
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.laststatechangereason.should.be.a(str)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
x.market.should.equal(y.market)
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.name.should.be.a(str)
x.readydatetime.should.be.a(str)
x.startdatetime.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.state.should.equal("RUNNING")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
for x in conn.list_instance_groups(job_id).instancegroups:
y = input_groups[x.name]
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
if hasattr(y, "bidprice"):
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
x.bidprice.should.equal(y.bidprice)
# Configurations
# EbsBlockDevices
# EbsOptimized
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.should.have.property("id")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
x.instancegrouptype.should.equal(y.role)
x.instancetype.should.equal(y.type)
x.market.should.equal(y.market)
x.name.should.equal(y.name)
int(x.requestedinstancecount).should.equal(y.num_instances)
int(x.runninginstancecount).should.equal(y.num_instances)
# ShrinkPolicy
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.status.state.should.equal("RUNNING")
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.status.statechangereason.code.should.be.a(str)
x.status.statechangereason.message.should.be.a(str)
x.status.timeline.creationdatetime.should.be.a(str)
# x.status.timeline.enddatetime.should.be.a(str)
x.status.timeline.readydatetime.should.be.a(str)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
igs = dict((g.name, g) for g in jf.instancegroups)
conn.modify_instance_groups(
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
[igs["task-1"].instancegroupid, igs["task-2"].instancegroupid], [2, 3]
)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
jf = conn.describe_jobflow(job_id)
int(jf.instancecount).should.equal(base_instance_count + 5)
igs = dict((g.name, g) for g in jf.instancegroups)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
int(igs["task-1"].instancerunningcount).should.equal(2)
int(igs["task-2"].instancerunningcount).should.equal(3)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_steps():
input_steps = [
StreamingStep(
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
name="My wordcount example",
mapper="s3n://elasticmapreduce/samples/wordcount/wordSplitter.py",
reducer="aggregate",
input="s3n://elasticmapreduce/samples/wordcount/input",
output="s3n://output_bucket/output/wordcount_output",
),
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
StreamingStep(
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
name="My wordcount example & co.",
mapper="s3n://elasticmapreduce/samples/wordcount/wordSplitter2.py",
reducer="aggregate",
input="s3n://elasticmapreduce/samples/wordcount/input2",
output="s3n://output_bucket/output/wordcount_output2",
),
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
]
# TODO: implementation and test for cancel_steps
conn = boto.connect_emr()
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
cluster_id = conn.run_jobflow(steps=[input_steps[0]], **run_jobflow_args)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
jf = conn.describe_jobflow(cluster_id)
jf.steps.should.have.length_of(1)
conn.add_jobflow_steps(cluster_id, [input_steps[1]])
jf = conn.describe_jobflow(cluster_id)
jf.steps.should.have.length_of(2)
for step in jf.steps:
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
step.actiononfailure.should.equal("TERMINATE_JOB_FLOW")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
list(arg.value for arg in step.args).should.have.length_of(8)
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
step.creationdatetime.should.be.a(str)
# step.enddatetime.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
step.jar.should.equal("/home/hadoop/contrib/streaming/hadoop-streaming.jar")
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
step.laststatechangereason.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
step.mainclass.should.equal("")
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
step.name.should.be.a(str)
# step.readydatetime.should.be.a(str)
# step.startdatetime.should.be.a(str)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
step.state.should.be.within(["STARTING", "PENDING"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
expected = dict((s.name, s) for s in input_steps)
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
steps = conn.list_steps(cluster_id).steps
for x in steps:
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
y = expected[x.name]
# actiononfailure
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
list(arg.value for arg in x.config.args).should.equal(
[
"-mapper",
y.mapper,
"-reducer",
y.reducer,
"-input",
y.input,
"-output",
y.output,
]
)
x.config.jar.should.equal("/home/hadoop/contrib/streaming/hadoop-streaming.jar")
x.config.mainclass.should.equal("")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# properties
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.should.have.property("id").should.be.a(str)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
x.name.should.equal(y.name)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.status.state.should.be.within(["STARTING", "PENDING"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# x.status.statechangereason
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.status.timeline.creationdatetime.should.be.a(str)
# x.status.timeline.enddatetime.should.be.a(str)
# x.status.timeline.startdatetime.should.be.a(str)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
x = conn.describe_step(cluster_id, x.id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
list(arg.value for arg in x.config.args).should.equal(
[
"-mapper",
y.mapper,
"-reducer",
y.reducer,
"-input",
y.input,
"-output",
y.output,
]
)
x.config.jar.should.equal("/home/hadoop/contrib/streaming/hadoop-streaming.jar")
x.config.mainclass.should.equal("")
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# properties
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.should.have.property("id").should.be.a(str)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
x.name.should.equal(y.name)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
x.status.state.should.be.within(["STARTING", "PENDING"])
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
# x.status.statechangereason
Remove Py2 support (#3915)
2021-07-26 06:40:39 +00:00
x.status.timeline.creationdatetime.should.be.a(str)
# x.status.timeline.enddatetime.should.be.a(str)
# x.status.timeline.startdatetime.should.be.a(str)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
@requires_boto_gte("2.39")
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
def test_list_steps_with_states():
# boto's list_steps prior to 2.39 has a bug that ignores
# step_states argument.
steps = conn.list_steps(cluster_id).steps
step_id = steps[0].id
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
steps = conn.list_steps(cluster_id, step_states=["STARTING"]).steps
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
steps.should.have.length_of(1)
steps[0].id.should.equal(step_id)
Run black on moto & test directories.
2019-10-31 15:44:26 +00:00
Implement filters and pagers for some EMR end points
2016-10-18 23:47:02 +00:00
test_list_steps_with_states()
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
Testing new version of decorator.
2017-02-16 03:35:45 +00:00
@mock_emr_deprecated
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
def test_tags():
input_tags = {"tag1": "val1", "tag2": "val2"}
conn = boto.connect_emr()
cluster_id = conn.run_jobflow(**run_jobflow_args)
conn.add_tags(cluster_id, input_tags)
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
cluster = conn.describe_cluster(cluster_id)
cluster.tags.should.have.length_of(2)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
dict((t.key, t.value) for t in cluster.tags).should.equal(input_tags)
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
conn.remove_tags(cluster_id, list(input_tags.keys()))
Add EMR clusters and tagging.
2015-02-14 17:00:14 +00:00
cluster = conn.describe_cluster(cluster_id)
Better EMR coverage and boto3 request/response handling This revision includes: - A handler for requests for which content-type is JSON (from boto3). - A decorator (generate_boto3_response) to convert XML responses to JSON (for boto3). This way, existing response templates for boto can be shared for generating boto3 response. - Utility class/functions to use botocore's service specification data (accessible under botocore.data) for type casting, from query parameters to Python objects and XML to JSON. - Updates to response handlers/models to cover more EMR end points and mockable parameters
2016-09-22 03:59:19 +00:00
cluster.tags.should.have.length_of(0)
Reference in New Issue Copy Permalink