Add a linter and reformatter for python infra and test code

[thejohnjansen]

I'm doing line by line comparisons to try to figure out why Edge is not able to run the WebDriver tests and see subtle differences in formatting that make it difficult to see differences. @andreastt

[andreastt]

I would expect wptrunner has some linting infrastructure already set up, and could piggyback on this.

[domenic]

A linter already exists, but it doesn't enforce formatting preferences on the test writer (which is IMO a good thing).

In the past when Edge has tried to contribute changes they have come along with a large variety of stylistic changes. See e.g. #6499. Perhaps the issue is that Edge has some reformatter that they apply, which they should be turning off when operating on a shared, collaborative codebase?

[foolip]

@thejohnjansen, can you elaborate on the original problem, what are the things going through "line by line comparison"?

Is the request that all of web-platform-tests have a consistent style to make it easier to read tests, that we have an optional reformatter to reduce the number of choices one has to make when writing tests, or something else?

[kereliuk]

@thejohnjansen By line by line, are you talking about when you do wpt run and there is an error thrown but the output from WPT doesn't say where or give you enough of a stack trace to figure it out? (Even if its a syntax error, this happens)

[andreastt]

To shed some more light on the matter, @thejohnjansen was complaining about stylistic differences in the Python wdspec tests (under /webdriver), and not generally for every other test type. I don’t think it makes sense to enforce a particular style for other test types.

It is true that there is a lot of stylistic inconsistency under /webdriver that a linter would help prevent. I think however that there are other, more compelling reasons to enable linting of Python tests:

• Detect unnecessary, unused imports
• Correct misuse of == for comparing True, False, and None
• Detect other programming errors

I wouldn’t proclaim that I know a great deal about the best Python testing practices, but it seems we can add tox and flake8, which is used for wptrunner, quite easily without introducing any new dependencies. I have a large-ish patch locally that does this and fixes an array of small problems.

[foolip]

Renaming and adding some labels based on #8154 (comment). @thejohnjansen, if the issue is now narrower than you hoped for, please shout :)

[thejohnjansen]

Yes, the scope of this issue is accurate. I'm sorry for the initial confusion (and for not responding, I'm apparently not getting GitHub notifications). I was looking at a test file in the WebDriver directory and clicked to file a new issue, so I thought the issue would include the context of what I was looking at. It did not :-)

[foolip]

We currently run flake8 as part of tox, together with pytest. I have found this a bit hard to understand. @jgraham, what would you think of having a dedicated test job for flake8, invoking it once?

[foolip]

I've sent #13865 to do that.

[thiagowfx]

what would you think of having a dedicated test job for flake8, invoking it once?

#14893

[thiagowfx]

Also, for a more recent discussion about the topic, see web-platform-tests/rfcs#171