Implement discovery protocol on v3 api

[serathius]

Motivation removal of V2 API in #12913

Context Etcd supports Discovery protocol https://etcd.io/docs/v3.4/dev-internal/discovery_protocol/ based on V2 API. In v3.6 we are removing V2 API so we need a replacement that will work in v3.6+ versions. In v3.6 discovery v2 should still be able to do on older vclusters

Goal:

• Add --discovery-version flag that accepts v2 (default) and v3 value. Using v3 will switch Etcd to use discovery protocol using V3 API
• Update documentation https://etcd.io/docs/v3.4/dev-internal/discovery_protocol/
• Mark for v2 discovery as deprecated

Deprecation plan:

• Make v3 default in v3.7
• Remove v2 option in v3.8

[serathius]

cc @ptabor

[serathius]

@ahrtr would you be interested in implementing this?

[ahrtr]

@ahrtr would you be interested in implementing this?

@serathius Sure. Based on the Plans for v3.6.0 release, I assume this should be completed by the end of this month or the very early of next month?

[serathius]

@ahrtr would you be interested in implementing this?

@serathius Sure. Based on the Plans for v3.6.0 release, I assume this should be completed by the end of this month or the very early of next month?

Yes, I hope be done with deprecation and downgrades by then.

[ahrtr]

@ahrtr would you be interested in implementing this?

@serathius Sure. Based on the Plans for v3.6.0 release, I assume this should be completed by the end of this month or the very early of next month?

Yes, I hope be done with deprecation and downgrades by then.

No problem.

[ahrtr]

@serathius It took me a couple of hours to go through the existing discovery source code and try out the v2discovery using the public endpoint https://discovery.etcd.io. It's pretty simple.

But the original discovery project discovery.etcd.io has already been archived, and obviously is out of date. So we have two directions:

1. Create a new project to provide the similar functionality as discovery.etcd.io for v3 discovery , and we can still use etcd as the backend store.
2. Get rid of discovery.etcd.io completely in v3 discovery. We just need to start an standalone etcd instance or an etcd cluster directly to serve the v3 discovery service/requests, the bootstraped etcd instances in a new cluster just communicate with the discovery etcd instance or cluster using clientv3 API. We may need to define some discovery protocol used between the bootstrapped etcd instances and the discovery etcd server.

I prefer to the second approach/direction. What do you think? @ptabor

EDIT: We don't need to depend on the discovery.etcd.io at all, because it uses exactly the same v2client API to communicate with etcd server. It just automatically generates a token(UUID) for the client, see new.go#L35-L43. Apart from this, it just proxy the request to the backend etcd server, see token.go#L32.

It seems that we just need to implement the similar functionality as previous v2http and of course the v3discovery (similar to v2discovery).

[ahrtr]

@serathius I'd like to breakdown this task into 3 three smaller tasks:

1. Implement the v3discovery;
2. Add unit test and (e2e or integration) test;
3. Update the documentation.

I am planning to submit a PR for #1 today or tomorrow, so as to:

1. get feedback before moving on.
2. reduce the burden on the reviewers to review a huge PR.

What do you think? cc @ptabor @spzala

[serathius]

Small PRs are highly encouraged! Thanks for proposing this.

[ptabor]

Thank you @ahrtr . We should not depend/support discovery.etcd.io in any way.

[Edited after reading the PR]

Ad. deprecation plan, is my understanding correct:

Assuming there is discovery protocol customer using:

1. v3.4 (or v3.5) discovery server not using the --enable-v2v3 mode (so v2 api state is kept in store v2)
2. etcd v3.5 using the discovery server using for bootstrapping.

If we consider upgrading scenario of the 2. keeping 1. as is.

• 1. upgrade to 3.6... according to the plan the v2 protocol is still being used. The state is persisted in 1. in v2store.
     the discovery keep working
• 2. upgrade to 3.7/3.8. If user switches to v3: The server does not finds the state in discovery service (as its not in store v3) so get's stuck till all size member will get rerun at the same time.

So upgrade to 3.7/3.8 disruptful and we pushed the problem forward to 3.7/3.8 but it will still hit users.

---

What options we have:

1. Make upgrade to 3.6 require explicit user action without migration path

• v3.6 only supports --discovery=v3. The flag is mandatory.
• The flag description contains warnings that switching to v3 protocol will require concurrent restart of all members with the --v3 flag.
• This is acceptable option if discovery usage is small (that I hope for) - as serious production deployment should not depend on external service on bootstrap.

2. Make upgrade to 3.7 require explicit user action without migration path

This is similar to above... but user does not need to coordinate the 'concurrent' recreate with v3.5->v3.6 transition and can make it at any point during 3.6 lifecycle.

3. Make upgrade to 3.8 require the switch as proposed.

I think that 2. is good enough. If we could cloud-source whether discovery has meaningful prod adoption we could consider 1.

---

Orthogonally we could consider some tooling like etcdctl discover-clone-v2v3 that migrates the config from v2store to v3 such that update without coordinated restart of all members is possible.

[ahrtr]

Thanks for the comment. Since v2 discovery will be decommissioned in 3.8, so the tool should only be available in 3.5 ~ 3.7?

Orthogonally we could consider some tooling like etcdctl discover-clone-v2v3 that migrates the config from v2store to v3 such that update without coordinated restart of all members is possible.

[ahrtr]

The implementation should have already been done in PR pull/13635, please take a look. @serathius @ptabor @spzala

With regard to the documentation, my understanding is we only need to update document for 3.6, any the previous document should be keep as it is. But I do not see the sub-directory v3.6 in content/en/docs. My understanding is the sub-directory v3.6 will not be added until etcd v3.6 is released.

Could you please provide guide on how & where should I update the discovery document?

[serathius]

Two follow ups from the #13635:

• Make V3 discovery and clientv3 to reuse the same client config;
• Change v3 discovery url to endpoints so as to support failover.

[serathius]

Any work left?

[ahrtr]

Just raised etcd-io/website#653 to track the left documentation work