Move the cache watch API from channels to interfaces

[jpeach]

When you register a watch on a cache entry using CreateWatch the caller has to pass in a channel to receive the notification. This approach makes both the caller and the cache responsible for channel buffer management (i.e. it will deadlock if the buffer fiils), and precludes notification designs that aren't channels (e.g. appending to a queue).

Consider changing the API to allow the caller to pass in a notification interface. This means that if a caller chooses to implement notifications using channels it can, and the cache doesn't need to care about that. If a caller needs a different kind of notification, it can do that too.

UInfortunately, this would break everyone's code, so needs some thought about migration and perhaps a default implementation.

xref #503

[alecholmez]

I like the thought @jpeach. Can you drop a comment here with what the notification interface might look like?

[jpeach]

I like the thought @jpeach. Can you drop a comment here with what the notification interface might look like?

Sure, probably next week. In the slack commentary someone mentioned that the Java xDS library used a notification interface, so it might be worth following that.

[github-actions]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.

[github-actions]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as "help wanted" or "no stalebot". Thank you for your contributions.