Ensuring library compatibility with CircuitPython version

[dhalbert]

In each major version of CircuitPython (major.x.x), we may make incompatible API changes. In each minor version (x.minor.x), we may add additional API functionality. Our current policy to support two major versions of CircuitPython: a stable version and an upcoming release that is in alpha or beta. The libraries have to track CircuitPython as it evolves. How can we do this in an automated way, so that a user is confident they have the latest compatible library version?

Currently we package the same version of a library in all bundles we make, for all versions of CircuitPython. If there are incompatible changes between CircuitPython versions, we add code to each libraries that is affected;:that code does version-checking or failover to support the API differences. As we move the supported version window, we can clean up these work-arounds.

We could instead have different versions of a library for different CircuitPython versions, as necessary. For each version of CircuitPython, we would bundle the latest version of the library that supported that version of CPy. If an incompatible change was necessary, we would release a new library version, but keep the old version for the older bundle. For instance, a library might work for 3.x and 4.x, but not for 5.x without an incompatible change. We'd bundle the latest version for 5.x, and bundle the highest compatible older version for 3.x and 4.x.

To support multiple, we could have multiple branches for the CPy versions, and fetch the version from the appropriate branch. Or, we could track the minimum CPy version in, say __init__.py as a constant, something like __minimum_cpy_version__ = '3.0.0'. The bundle builder would scan the releases of a library to find the latest one for the bundle being built. This minimum version info could be encoded in other ways, such as a suffix to the library version, or a filename.

Once we drop support for a CPy version, we no longer build bundles for that version. We should consider keeping an archive version of the last bundle, for those who cannot upgrade due to older applications or boards we've stopped updating (e.g., ESP8266).

Further comments welcome. This issue was first discussed in the "In the weeds" section of the CIrcuitPython weekly meeting on August 19, 2019: https://www.youtube.com/watch?v=eqIY1M6TvaA.

[anecdata]

Would be nice to have a facility to see, on-disk and at runtime, what bundle version is installed and in use. VERSIONS.txt shows the bundle date (and of course all of the individual versions), but I'm not aware of a snapshot where I can see on CIRCUITPY (as a human or as a .py) if I have the 4.x or 5.x bundle for that date installed.

I suppose one could copy / parse the parent folder on the target device, adafruit-circuitpython-bundle-5.x-mpy-20190910/lib/*, but a standard solution that everyone could count on would be good.

[sommersoft]

I finally got some cycles to put my random thoughts, and some brainstorming, down.

1. Using branches:
   • Linear expansion of repo maintenance, based on supported versions. At minimum, that's roughly 320 branches to maintain. This isn't completely a negative point though (see "transition code" comment below).
   • Does allow for easy git-fu when bundling, by just checking out the appropriate branch and grabbing the latest release.
   • Very cleanly erases the need for "transition code" to accommodate multiple CircuitPython versions.

---

2. __minimum_cpy_version__
   • Placement could mirror the current __version__ and __repo__, to minimize required work in package and non-package libraries.
   • Could be used by the new CircUp tool (if desired).
   • Requires iterating through each tagged commit, opening the appropriate file, getting info, and then processing based on the condition.

---

3. Semantic Version Metadata (https://semver.org/#spec-item-10)
   • Appends minimum CircuitPython version to the release tag. Example: 1.0.0+5.0.0 = lib version 1.0.0, with minimum CirPy version of 5.0.0. Although, including minor/patch in the metadata is probably superfluous.
   • Also allows easy git-fu when bundling, with use of glob pattern searching.
   • Could introduce confusion in changing the semver format.
   • Could actually facilitate number 2 above, the same way __version__ is accomplished.

[dhalbert]

@sommersoft Thanks for the detailed analysis, which shows up a number of things that weren't obvious. The base issue keeps coming up for people who are confused about which BLE library to get.

---

1. We could make master always be used, unless a branch exists that matches the CPy version we're targeting. Existing libraries will still work without having to add new branches. That way we can peel off branches for older versions and move up the implementations in master without disturbing older ones. Looking for branches is probably cheap.

Once we create a CPy-specific branch, we have to create branches for previous versions as well. For instance, just having 4.x and master doesn't make it clear what to use for 3.x, so if we create a 4.x branch, we need to create a 3.x branch as well (which might be the same as the 4.x branch.

Or we could just create all these branches right away, on every library: it's just a mechanical thing. We would create 1.x, 2.x, 3.x, and 4.x branches.

Note #175, "Finding latest release assumes latest tag is on master", which we'll have to deal with.

---

2. "iterating through each tagged commit, opening the appropriate file, getting info, and then processing based on the condition" sounds expensive.

---

3. This does seem like the simplest, since it just involves scanning tags. Maybe we can just list all the tags, sort them by semver, and then range match on the +x.x.x part. If there's no +x.x.x, then we fall back to using the latest tag. This would also provide a way of solving Finding latest release assumes latest tag is on master #175.

[dhalbert]

@ntoll Do you have any opinion here wrt CircUp? We discussed this yesterday in the weekly meeting, and felt the option 3 was the easiest, though it goes beyond what semver has in mind for how to use the metadata part of the version string.

[ntoll]

@dhalbert thanks for pinging me about this -- much appreciated. I concur that option 3 is likely the easiest solution and can see how I could easily update Circup to facilitate this convention. 👍

[caternuson]

Could the other release fields be used at all as a place to store the additional meta data? I'm thinking the "Release Title" and "Release Description" that also get filled out in addition to the tag.

[tannewt]

FWIW, I'd really like to avoid this problem if we can. I'd rather only support BLE with 5.0.0. Dependency tracking is a huge rat hole where tons of time can be spent. (Python is still spending this time.)

Instead, I'd rather packages throw friendly exceptions for unsupported versions and always support the newest version.