run tox w/ a matrix of ansible versions

[ryanpetrello]

No description provided.

[ryanpetrello]

this'll require some zuul updates to point at the new targets:

~/dev/ansible-runner tox -a
linters
py27-ansible27
py27-ansible28
py27-ansibledev
py36-ansible27
py36-ansible28
py36-ansibledev
py37-ansible27
py37-ansible28
py37-ansibledev

[ansible-zuul]

Build failed.

• ansible-tox-py36 : FAILURE in 4m 18s
• ansible-tox-py37 : FAILURE in 4m 09s
• ansible-build-python-tarball : SUCCESS in 1m 35s
• ansible-tox-linters : SUCCESS in 4m 14s
• ansible-tox-py27 : FAILURE in 6m 35s

[pabelanger]

Hello! So, we can actually create a .zuul.yaml file here with these new entry points. For example:

https://github.com/ansible-network/sandbox/pull/31/files#diff-7415f5ff7beee2cdf9ffe31e12e4c086R1

So, if this is a common pattern, we could standardize these entry points for tox, and we can add them to ansible/ansible-zuul-jobs, then we can have:

ansible-tox-py36-ansible26
ansible-tox-py36-ansible27
etc

[pabelanger]

also, I'm happy to work on .zuul.yaml file if you'd like, given this is the first time adding it to ansible-runner. Just let me know!

[ryanpetrello]

Hey @pabelanger,

That would be awesome if you don't mind :)

[ansible-zuul]

Build failed.

• ansible-tox-py36 : FAILURE in 4m 18s
• ansible-tox-py37 : FAILURE in 4m 12s
• ansible-build-python-tarball : SUCCESS in 1m 33s
• ansible-tox-linters : SUCCESS in 4m 10s
• ansible-tox-py27 : FAILURE in 8m 19s

[pabelanger]

sure, let me get up a PR with example jobs

[AlanCoding]

FYI, I'm seeing at least 2 more failures with Ansible devel, probably due to event ordering. Doing this is sure to reveal some issues, and I can help to address them.

[ryanpetrello]

@AlanCoding I noticed the same - that these tests don't actually work in 2.8 due to events coming in differently.

I figured once @pabelanger had a PR up for the zuul config, we could wrap this up by making the tests work in 2.8, too (and also verifying it's not actually a 2.8 regression, of course).

[matburt]

What is the order difference we are seeing?

[AlanCoding]

I don't have a clean run of it, but here are failures which appeared to be from Ansible versioning:

https://gist.github.com/AlanCoding/c4a1e22db9447184166a212b5b343dfa

[matburt]

Oh, these test assumptions probably just need to be updated conditionally based on the version of Ansible being used. Pre 2.8:

...
playbook_on_task_start
runner_on_ok
...

2.8+

...
playbook_on_task_start
runner_on_start
runner_on_ok
...

[ryanpetrello]

Yep, exactly. I just haven't gotten around to it because I'm waiting on a PR from @pabelanger for zuul config.

[privateip]

@ryanpetrello @pabelanger any update here?

[pabelanger]

Sorry, this fell to the way side once hold on migrating more things to zuul was put on hold. I can whip up an example for Friday.

[AlanCoding]

Link #336 for some version-dependent test fixes

[ryanpetrello]

Haven't found time for this, might reinvestigate later.