WIP: Split bear execution by runtime #1386

jayvdb · 2017-02-01T14:12:05Z

gitmate-bot · 2017-02-01T14:12:21Z

Thanks for your contribution!

Reviewing pull requests take really a lot of time and we're all volunteers. Please make sure you go through the following check list and complete them all before pinging someone for a review.

I have rebased properly. Please see this for a tutorial.
I have gone through the commit guidelines and I've followed them:
I have followed the guidelines for the commit shortlog.
I have followed the guidelines for the commit body.
I have included the issue URL.
I have used the Fixes keyword if the commit fixes a real bug and the Closes keyword if it adds a feature/enhancement. See more here.
I have run all the tests and they all pass. That includes that all CI (below) is green - if GitMate issues appear you will have to amend your commits, pushing new commits on top is not sufficient!

As you learn things over your Pull Request please help others on the chat and on PRs to get their stuff right as well!

jayvdb · 2017-02-01T14:38:01Z

This is mostly a WIP,, as it needs to be split into multiple commits and some of the voodoo bash stuff needs to be refined, and some additional Requirement need to be added for the current algorithm to correctly split Python bears from non-Python bears.

However, the concept behind it, of not testing non-Python bears on each version of Python, needs some arch review.
The main performance improvement is that any job which doesnt require sudo can run start (and finish) much faster, which gives better feedback for many times of -bear work. Due to the PPAs, some jobs need to use sudo, and these wont run any faster, so the actual elapsed completion time wont be reduced, at least not much.

There are many further improvements that can be made, such as splitting node and ruby and python bears into separate jobs (these are already manageable by Requirement classes), and R and Haskell and Java should follow soon with Requirement classes being added. That will allow testing using multiple versions of each runtime concurrently.

sils · 2017-02-01T19:02:38Z

@jayvdb gorgeous idea! Put th concept explanation into the commit message please or maybe right into the code as a comment.

jayvdb · 2017-02-02T00:55:20Z

.ci/bears.prune.sh

+# Delete empty bear directories
+
+for dir in bears/*/; do
+  if [[ -z "$(ls $dir/ 2>/dev/null | egrep -v '(__|.xml)')" ]]; then


So this looks for directories without any Bear's in it..

The __ is for __init__ and __pycache__
.xml is because of bears/scala/scalastyle_config.xml

I think a much better approach would be to look for one file matching *Bear.py

Oh, the problem with *Bear is we have directories with only subdirectories, like bears/vcs. Solvable, and probably still clearer to take this approach.

jayvdb · 2017-02-02T00:58:06Z

.travis.yml

+    if [[ "$BEARS" != "all" ]]; then
+      rm $(comm -23 <(ls bears/*/[A-Za-z]*.py | grep -v general | sort) <(grep -l PipRequirement bears/*/*.py | sort))
+      rm bears/general/CPDBear.py
+      rm bears/generate_package.py tests/generate_packageTest.py


the special case for bears/generate_package.py -> tests/generate_packageTest.py could be added to tests.prune.sh, instead of here in .travis.yml

jayvdb · 2017-02-02T00:59:28Z

.travis.yml

+  # Only keep non-Python bears on Python 3.4
+  - >
+    if [[ "$BEARS" != "all" ]]; then
+      rm $(comm -23 <(ls bears/*/[A-Za-z]*.py | grep -v general | sort) <(grep -l PipRequirement bears/*/*.py | sort))


grep -v general -- all bears/general/ are Python language bears without any dependencies, except CPDBear (see next line).

This needs to be tidied up somehow.

jayvdb · 2017-02-02T01:01:30Z

bears/ruby/RuboCopBear.py

@@ -19,8 +19,7 @@ class RuboCopBear:
    """

    LANGUAGES = {'Ruby'}
-    REQUIREMENTS = {GemRequirement('rubocop'),
-                    PipRequirement('pyyaml', '3.12')}


This was done because the shell foo in .travis.yml needs to find non-Python bears. It is using the presence of PipRequirement, and there are two non-Python bears which use PipRequirement.

.travis.yml logic needs to be rewritten, and then this isnt necessary.

jayvdb · 2017-02-02T01:02:52Z

.ci/generate_bear_requirements.py

@@ -81,6 +82,15 @@ def get_inherited_requirements():
    in_inherited = False
    with open(os.path.join(PROJECT_DIR, 'requirements.txt'), 'r') as file:
        for line in file.read().splitlines():
+            line = line.strip()
+            if not line.startswith('#'):
+                m = re.match('^[A-Za-z._-]+', line)


Adding more requirements parsing ;-( Should be using an existing tool for this.
This entire modification to this script wont be necessary when the pyyaml related changes are backed-out.

gitmate-bot · 2017-02-02T15:19:10Z

Comment on e320695.

This commit seems to be marked as work in progress and should not be used in production. Treat carefully.

GitCommitBear, severity NORMAL, section commit.

gitmate-bot · 2017-02-03T17:22:05Z

Comment on 1b8458a.

This commit seems to be marked as work in progress and should not be used in production. Treat carefully.

GitCommitBear, severity NORMAL, section commit.

gitmate-bot · 2017-02-03T23:05:05Z

Comment on 1b8458a.

This commit seems to be marked as work in progress and should not be used in production. Treat carefully.

GitCommitBear, severity NORMAL, section commit.

jayvdb · 2017-02-04T00:37:31Z

Note bca64a0 was an intentional breakage to show codecov reports less than 100% coverage when one bear doesnt work. See it on
https://codecov.io/gh/jayvdb/coala-bears/branch/travis-matrix-python-speedup/commits

jayvdb · 2017-02-06T04:57:45Z

The circle breakage has been addressed in #1406 , which only solves part of this as it is easier to test that part by itself.

gitmate-bot · 2017-02-13T11:46:23Z

Hey! This pull request hasn't been updated for a while :/ It would be nice if we could get this going again!

gitmate-bot · 2017-07-22T01:57:15Z

Hey! This pull request hasn't been updated for a while :/ It would be nice if we could get this going again!

saksham189 · 2018-04-12T15:45:57Z

.ci/bears.prune.sh

+go_requirement_bears=$(grep -m 1 -l GoRequirement $requirement_bears)
+npm_requirement_bears=$(grep -m 1 -l NpmRequirement $requirement_bears)
+rscript_requirement_bears=$(grep -m 1 -l RscriptRequirement $requirement_bears)
+


Need to add luarocks_requirement_bears and julia_requirement_bears

Ref: #1650

saksham189 · 2018-04-12T15:49:23Z

.ci/bears.prune.sh

+
+dart_bears=$(ls bears/dart/*Bear.py)
+julia_bears=$(ls bears/julia/*Bear.py)
+lua_bears=$(ls bears/lua/*Bear.py)


can remove lua_bears and julia_bears

saksham189 · 2018-04-12T16:12:25Z

.ci/bears.prune.sh

+
+non_python_bears=$(comm -23 <(ls $bears) <(ls $python_bears))
+
+cabal_requirement_bears="$cabal_requirement_bears bears/haskell/HaskellLintBear.py bears/shell/ShellCheckBear.py"


ShellCheckBear has a Cabal Requirement now

Ref: #1496

Also this needs to be solved: #1358 so, that we can remove HaskellLintBear

saksham189 · 2018-04-12T16:50:54Z

.ci/bears.dirs.prune.sh

+  if [[ -z "$bears""$subdirs" ]]; then
+    echo Removing $dir
+    rm -rf $dir
+  fi


Why is it important to delete the empty directories? What if we left them as it is?

I forget.

probably another bear will complain about a missing __init__.py, or we have missing coverage of those __init__.py

jayvdb · 2018-05-12T07:53:29Z

Godeps/Godeps.json

@@ -0,0 +1,25 @@
+{


jayvdb · 2018-05-12T07:59:40Z

DESCRIPTION

@@ -0,0 +1,3 @@
+Package: coala


jayvdb · 2018-05-12T08:02:00Z

Makefile.PL

@@ -0,0 +1,6 @@
+use ExtUtils::MakeMaker;


jayvdb · 2018-05-12T08:03:43Z

.ci/bears.dirs.prune.sh

+  if [[ -z "$bears""$subdirs" ]]; then
+    echo Removing $dir
+    rm -rf $dir
+  fi


I forget.

probably another bear will complain about a missing __init__.py, or we have missing coverage of those __init__.py

gitmate-bot · 2018-11-12T08:39:13Z

.ci/bears.prune.sh

+
+yield_result_bears=$(grep -m 1 -l 'yield Result' $bears)
+
+non_yield_result_bears=$(comm -23 <(ls $bears) <(ls $yield_result_bears))


non_yield_result_bears appears unused. Verify it or export it. [SC2034]

Origin: ShellCheckBear, Section: bash.

gitmate-bot · 2018-11-12T08:39:14Z

.ci/bears.prune.sh

+fi
+
+pip_only_requirement_bears=$(comm -23 <(ls $pip_requirement_bears) <(ls $non_pip_runtime_requirement_bears))
+pip_plus_requirement_bears=$(comm -23 <(ls $pip_requirement_bears) <(ls $pip_only_requirement_bears))


pip_plus_requirement_bears appears unused. Verify it or export it. [SC2034]

Origin: ShellCheckBear, Section: bash.

gitmate-bot · 2018-11-12T08:39:15Z

.ci/bears.prune.sh

+clang_bears=''
+other_bears=''
+for bear in $no_requirement_bears; do
+  if [[ ${bear/Clang/} != $bear ]]; then


Quote the rhs of != in [[ ]] to prevent glob matching. [SC2053]

Origin: ShellCheckBear, Section: bash.

gitmate-bot · 2018-11-12T09:13:35Z

.ci/bears.prune.sh

+clang_bears=''
+other_bears=''
+for bear in $no_requirement_bears; do
+  if [[ ${bear/Clang/} != ${bear} ]]; then


Quote the rhs of != in [[ ]] to prevent glob matching. [SC2053]

Origin: ShellCheckBear, Section: bash.

jayvdb · 2019-05-26T14:08:10Z

.travis.yml

+    - language: ruby
+      rvm: 2.0
+
+    - language: haskell


this job isnt working yet in the new PR #2910

the TRAVIS_GHC_ROOT=/opt/ghc env setting below might be relevant.

jayvdb · 2019-05-26T14:10:25Z

setup.py

+
+config = configparser.ConfigParser()
+config.read('setup.cfg')
+if 'setup:custom' in config.sections():


This is a separate issue, and hasnt been fixed yet.

gitmate-bot added process/pending review size/L review carefully! labels Feb 1, 2017

jayvdb added the process/wip label Feb 1, 2017

jayvdb commented Feb 2, 2017

View reviewed changes

jayvdb removed the process/pending review label Feb 2, 2017

jayvdb force-pushed the travis-matrix-python-speedup branch from 4474917 to e320695 Compare February 2, 2017 15:18

gitmate-bot added process/pending review size/XL and removed process/wip size/L labels Feb 2, 2017

jayvdb force-pushed the travis-matrix-python-speedup branch from e320695 to 1b8458a Compare February 3, 2017 17:21

gitmate-bot added size/XXL and removed size/XL labels Feb 3, 2017

jayvdb mentioned this pull request Feb 4, 2017

DistributionRequirement enhancements #1403

Merged

jayvdb changed the title ~~Only test non-Python bears on Python 3.6~~ Split bear execution per runtime Feb 4, 2017

jayvdb changed the title ~~Split bear execution per runtime~~ Split bear execution by runtime Feb 4, 2017

jayvdb force-pushed the travis-matrix-python-speedup branch 2 times, most recently from 585ccdb to 921ebc6 Compare February 4, 2017 01:10

jayvdb added process/wip and removed process/pending review labels Feb 6, 2017

gitmate-bot added the status/STALE label Aug 31, 2017

jayvdb requested a review from saksham189 April 9, 2018 13:28

saksham189 reviewed Apr 12, 2018

View reviewed changes

jayvdb commented May 12, 2018

View reviewed changes

Godeps/Godeps.json

@@ -0,0 +1,25 @@

{

Copy link

Member Author

jayvdb May 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#2481

jayvdb commented May 12, 2018

View reviewed changes

DESCRIPTION Outdated

@@ -0,0 +1,3 @@

Package: coala

Copy link

Member Author

jayvdb May 12, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#2482

jayvdb commented May 12, 2018

View reviewed changes

jayvdb mentioned this pull request Jun 19, 2018

.travis.yml: Add sentinel job #2541

Merged

2 tasks

jayvdb mentioned this pull request Oct 23, 2018

Add paprika bear #2347

Open

jayvdb force-pushed the travis-matrix-python-speedup branch from c07e6b2 to 9fe134a Compare November 12, 2018 08:36

gitmate-bot added size/L area/lintbears difficulty/medium importance/high and removed size/XXL labels Nov 12, 2018

gitmate-bot assigned jayvdb Nov 12, 2018

gitmate-bot reviewed Nov 12, 2018

View reviewed changes

jayvdb force-pushed the travis-matrix-python-speedup branch from 9fe134a to 041f05b Compare November 12, 2018 09:10

gitmate-bot reviewed Nov 12, 2018

View reviewed changes

jayvdb force-pushed the travis-matrix-python-speedup branch 3 times, most recently from d56fc7f to d28fca8 Compare November 12, 2018 09:39

WIP: Split tests by runtime

d28fca8

jayvdb mentioned this pull request Nov 12, 2018

coalib/: Use trollius for 3.4.x support coala/coala#5892

Closed

jayvdb mentioned this pull request May 17, 2019

Declarative mapping of tests to source carljm/pytest-smartcov#3

Open

jayvdb mentioned this pull request May 26, 2019

Use tox #2910

Merged

jayvdb commented May 26, 2019

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WIP: Split bear execution by runtime #1386

WIP: Split bear execution by runtime #1386

jayvdb commented Feb 1, 2017 •

edited

Loading

gitmate-bot commented Feb 1, 2017

jayvdb commented Feb 1, 2017

sils commented Feb 1, 2017

jayvdb Feb 2, 2017

jayvdb Feb 2, 2017

jayvdb Feb 2, 2017

jayvdb Feb 2, 2017

jayvdb Feb 2, 2017

jayvdb Feb 2, 2017

gitmate-bot commented Feb 2, 2017

gitmate-bot commented Feb 3, 2017

gitmate-bot commented Feb 3, 2017

jayvdb commented Feb 4, 2017

jayvdb commented Feb 6, 2017

gitmate-bot commented Feb 13, 2017

gitmate-bot commented Jul 22, 2017

saksham189 Apr 12, 2018

saksham189 Apr 12, 2018

saksham189 Apr 12, 2018

saksham189 Apr 12, 2018 •

edited

Loading

saksham189 Apr 12, 2018

jayvdb May 12, 2018

jayvdb May 12, 2018

jayvdb May 12, 2018

jayvdb May 12, 2018

jayvdb May 12, 2018

gitmate-bot Nov 12, 2018

gitmate-bot Nov 12, 2018

gitmate-bot Nov 12, 2018

gitmate-bot Nov 12, 2018

jayvdb May 26, 2019

jayvdb May 26, 2019


		non_python_bears=$(comm -23 <(ls $bears) <(ls $python_bears))

		cabal_requirement_bears="$cabal_requirement_bears bears/haskell/HaskellLintBear.py bears/shell/ShellCheckBear.py"


		yield_result_bears=$(grep -m 1 -l 'yield Result' $bears)

		non_yield_result_bears=$(comm -23 <(ls $bears) <(ls $yield_result_bears))

WIP: Split bear execution by runtime #1386

Are you sure you want to change the base?

WIP: Split bear execution by runtime #1386

Conversation

jayvdb commented Feb 1, 2017 • edited Loading

gitmate-bot commented Feb 1, 2017

jayvdb commented Feb 1, 2017

sils commented Feb 1, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gitmate-bot commented Feb 2, 2017

gitmate-bot commented Feb 3, 2017

gitmate-bot commented Feb 3, 2017

jayvdb commented Feb 4, 2017

jayvdb commented Feb 6, 2017

gitmate-bot commented Feb 13, 2017

gitmate-bot commented Jul 22, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saksham189 Apr 12, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jayvdb commented Feb 1, 2017 •

edited

Loading

saksham189 Apr 12, 2018 •

edited

Loading