moto/tests
Brian Pandola f7467164e4
Fix Race Condition in batch:SubmitJob (#3480)
* Extract Duplicate Code into Helper Method

DRY up the tests and replace the arbitrary `sleep()` calls with a more
explicit check before progressing.

* Improve Testing of batch:TerminateJob

The test now confirms that the job was terminated by sandwiching a `sleep`
command between two `echo` commands.  In addition to the original checks
of the terminated job status/reason, the test now asserts that only the
first echo command succeeded, confirming that the job was indeed terminated
while in progress.

* Fix Race Condition in batch:SubmitJob

The `test_submit_job` in `test_batch.py` kicks off a job, calls `describe_jobs`
in a loop until the job status returned is SUCCEEDED, and then asserts against
the logged events.

The backend code that runs the submitted job does so in a separate thread. If
the job was successful, the job status was being set to SUCCEEDED *before* the
event logs had been written to the logging backend.

As a result, it was possible for the primary thread running the test to detect
that the job was successful immediately after the secondary thread had updated
the job status but before the secondary thread had written the logs to the
logging backend.  Under the right conditions, this could cause the subsequent
logging assertions in the primary thread to fail.

Additionally, the code that collected the logs from the container was using
a "dodgy hack" of time.sleep() and a modulo-based conditional that was
ultimately non-deterministic and could result in log messages being dropped
or duplicated in certain scenarios.

In order to address these issues, this commit does the following:

* Carefully re-orders any code that sets a job status or timestamp
  to avoid any obvious race conditions.
* Removes the "dodgy hack" in favor of a much more straightforward
  (and less error-prone) method of collecting logs from the container.
* Removes arbitrary and unnecessary calls to time.sleep()

Before applying any changes, the flaky test was failing about 12% of the
time.  Putting a sleep() call between setting the `job_status` to SUCCEEDED
and collecting the logs, resulted in a 100% failure rate.  Simply moving
the code that sets the job status to SUCCEEDED to the end of the code block,
dropped the failure rate to ~2%.  Finally, removing the log collection
hack allowed the test suite to run ~1000 times without a single failure.

Taken in aggregate, these changes make the batch backend more deterministic
and should put the nail in the coffin of this flaky test.

Closes #3475
2020-11-18 10:49:25 +00:00
..
test_acm Linting 2020-11-11 15:55:37 +00:00
test_apigateway Linting 2020-11-11 15:55:37 +00:00
test_applicationautoscaling Linting 2020-11-11 15:55:37 +00:00
test_athena Linting 2020-11-11 15:55:37 +00:00
test_autoscaling Linting 2020-11-11 15:55:37 +00:00
test_awslambda fixed issue in update_configuration for lambda when setting VPC config property (#3479) 2020-11-18 08:45:31 +00:00
test_batch Fix Race Condition in batch:SubmitJob (#3480) 2020-11-18 10:49:25 +00:00
test_cloudformation Linting 2020-11-11 15:55:37 +00:00
test_cloudwatch Finish porting from nose to pytest. 2020-11-10 08:25:05 +01:00
test_codecommit Finish porting from nose to pytest. 2020-11-10 08:25:05 +01:00
test_codepipeline Linting 2020-11-11 15:55:37 +00:00
test_cognitoidentity Finish porting from nose to pytest. 2020-11-10 08:25:05 +01:00
test_cognitoidp Linting 2020-11-11 15:55:37 +00:00
test_config Linting 2020-11-11 15:55:37 +00:00
test_core Linting 2020-11-11 15:55:37 +00:00
test_datapipeline Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_datasync Finish porting from nose to pytest. 2020-11-10 08:25:05 +01:00
test_dynamodb Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_dynamodb2 Add support for empty strings in non-key dynamo attributes (#3467) 2020-11-17 09:12:39 +00:00
test_dynamodbstreams Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_ec2 Improve ec2:DescribeSubnets filtering (#3457) 2020-11-16 08:17:36 +00:00
test_ec2instanceconnect Improve implementation coverage (and layout) 2020-05-12 14:58:35 +01:00
test_ecr Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_ecs Linting 2020-11-11 15:55:37 +00:00
test_elasticbeanstalk Linting 2020-11-11 15:55:37 +00:00
test_elb Back to Black 2020-11-10 14:12:38 +01:00
test_elbv2 Finish porting from nose to pytest. 2020-11-10 08:25:05 +01:00
test_emr Added support for EMR Security Configurations and Kerberos Attributes. (#3456) 2020-11-17 10:54:34 +00:00
test_events EventBridge: put_rule and list_rules should store and retrieve EventBusName property (#3472) 2020-11-17 15:36:17 +00:00
test_forecast Linting 2020-11-11 15:55:37 +00:00
test_glacier Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_glue Back to Black 2020-11-10 14:12:38 +01:00
test_iam Merge pull request #3451 from bpandola/fix-3450 2020-11-15 11:25:10 -06:00
test_iot Linting 2020-11-11 15:55:37 +00:00
test_iotdata Finish porting from nose to pytest. 2020-11-10 08:25:05 +01:00
test_kinesis Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_kinesisvideo Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_kinesisvideoarchivedmedia Linting 2020-11-11 15:55:37 +00:00
test_kms Linting 2020-11-11 15:55:37 +00:00
test_logs Linting 2020-11-11 15:55:37 +00:00
test_managedblockchain Linting 2020-11-11 15:55:37 +00:00
test_opsworks Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_organizations Linting 2020-11-11 15:55:37 +00:00
test_packages Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_polly Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_ram Finish porting from nose to pytest. 2020-11-10 08:25:05 +01:00
test_rds Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_rds2 Fix: TagList missing in rds:DescribeDBInstance response (#3459) 2020-11-16 09:30:53 +00:00
test_redshift Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_resourcegroups Run black on moto & test directories. 2019-10-31 10:36:05 -07:00
test_resourcegroupstaggingapi Add ec2.vpc resource support to Tagging API (#3375) 2020-10-10 19:05:21 +01:00
test_route53 Fix XML encoding in Route53 JInja2 Templates #3469 (#3473) 2020-11-18 07:23:49 +00:00
test_s3 Linting 2020-11-11 15:55:37 +00:00
test_s3bucket_path Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_sagemaker Back to Black 2020-11-10 14:12:38 +01:00
test_secretsmanager Linting 2020-11-11 15:55:37 +00:00
test_ses Linting 2020-11-11 15:55:37 +00:00
test_sns Linting 2020-11-11 15:55:37 +00:00
test_sqs Linting 2020-11-11 15:55:37 +00:00
test_ssm Linting 2020-11-11 15:55:37 +00:00
test_stepfunctions Linting 2020-11-11 15:55:37 +00:00
test_sts Finish porting from nose to pytest. 2020-11-10 08:25:05 +01:00
test_swf Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
test_transcribe Linting 2020-11-11 15:55:37 +00:00
test_utilities Use TaggingService for S3 Objects 2020-03-31 12:04:04 +01:00
test_xray Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
__init__.py Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00
helpers.py Port test suite from nose to pytest. 2020-11-10 08:23:44 +01:00