Introducing the ESI API
The EVE Swagger Interface is a new framework developed by Team Tech Co to underpin the new ESI API. If you don't know what this is, you should go here to read the blog Introducing ESI - A new API for EVE Online.
This blog will dive a little deeper into the design and technical specifics of ESI, going to get into the base design of ESI to explain a few basic principles of what's going on behind the scenes. Most of this is transparent to the front-end, but may be of interest to some capsuleers regardless.
Fundamentals
Multi-tenant to the core
ESI was designed to be a single interface to all information EVE related. As such, if ESI and the cluster agree on configuration, you can change the datasource query parameter on any route (including /<version>/swagger.json) to the server name of your choice.
You will see this in the ESI UI as a select menu in the header/nav bar, and in the datasource enum in the spec itself.
As a (potentially) interesting side-note here, there are actually three ESI environments running, an esi-dev, esi-test, and the production esi. With esi-dev or esi-test and a bit of configuration (which will be replaced with tooling), EVE developers can actually query their local development server through ESI.
Authentication
Auth is still handled by SSO. There are new scopes which ESI uses, you can make new developer keys (or alter your existing) to make use of the new scopes. If an ESI endpoint requires authentication, you will see a red exclamation mark on the route description in the swagger UI. Mouse hovering over this icon will tell you what scope the endpoint requires.
ESI will handle redirecting your authentication header to the correct SSO for verification (depending on the datasource query string argument). SSO is not multi-tenant though, so you will need to create app keys on each SSO backend you intend to use.
Versioning
ESI itself will have an API version (as defined in the spec), that version number however, is mostly irrelevant to API consumers (it's the version of ESI-lib). Instead, ESI versions each route individually. This allows us to make much faster changes and avoid the awkward situation of global API versioning.
As an example, let's say you have a route /v1/hello/<string>/, and you want to change the path parameter to accepting an integer instead. With a traditional API, the entire APIs basePath would have to be bumped. This is obviously a little less than ideal, considering there could be hundreds of other unchanged routes.
With ESI, you always (and only) get three complete APIs, /dev/, /latest/ and /legacy/. But each route is also given a numbered path, starting with /v1/. Alternate routes are mentioned in the route description (until something better comes along in the standards?).
/dev/ can change at any point, changes to /latest/ will be announced. After changes are made the previous /latest/ will be available as /legacy/, until the next version bump. If you want to avoid the return schema of your request suddenly changing, you can use the versioned alternate route. In that case, prudent developers may want to create unit tests (watch for the "warning: 199" headers) to notify them when endpoints are moved to legacy. Additionally, we will be updating a stickied thread in the EVE Technology Lab subforum as changes happen.
Additionally, endpoints may be deprecated. When an endpoint is deprecated, a line appears through it on the swagger UI, and it begins returning the "warning: 299" header. This is slightly different than a warning: 199 header, which you will receive if an endpoint was updated and there is now a newer version of it available. Deprecation is how Tech Co broadcasts an intent to delete a route. Deprecated endpoints may include a recommended alternate source or other message in the 299 warning, and you should move away from them immediately.
Because endpoints are versioned individually, the concept of an overall /v1/ (or 2 or 5) API is not very relevant, and no swagger-ui is provided for these routes. Browsing the API's capabilities should be done via /dev/, /latest/ or /legacy/. However, users who wish to indulge their curiosity may feed a /v1/swagger.json into their own Swagger UI to get an overview, if they wish.
As an aside, all ESI routes end with a /. The only exceptions are the /<version>/swagger.json routes and routes used by the swagger-ui, which are passed through ESI.
Rate limiting
There are no rate limits in place, the plan is to rely on caching more than rate limiting.
ESI returns standard caching headers if the data is cached. Applications should notice and make use of these headers (expires and last-modified) on routes where they are provided.
Decoupling
We briefly touched on this earlier, but one of the major motivating factors of this design was to decouple the API from the monolith itself. As a consumer of the API, you should not have to care at all about the database structure, how or where the data is retrieved. All you should need to know is what inputs are allowed/accepted, and what exactly is being returned.
ESI takes this one step further. There is a generic concept of a "data relay" inside ESI, built on top of Google's PubSub. Because of this decoupling of data and API, it allows the retrieving end to move out of the monolith gradually and transparently. As long as something somewhere is listening and responding to events via PubSub, ESI will not notice if the service leaves the monolith and becomes standalone.
Moar design!
In the ESI API system, the codebase is split into two separate yet equally important groups: the esi-routers, who proxy routes and aggregate specs; and the esi-[endpoint]s, who fulfill the requests. These are their designs.
ESI-router
The routers are where most of the magic happens. They are the ones integrating with the Kubernetes API. The router finds the endpoints, and from each one, requests all of its swagger.jsons. It then combines all of these partial specs into versioned global specs. It serves the global specs, proxies the swagger-uis, and proxies ESI routes to their appropriate endpoint.
The routers are also a cluster, and use Redis to elect a leader among themselves. The leader continually checks for changes to the distributed spec, and the other routers all check for changes on the leader.
All routers also maintain telemetry on the endpoints, measuring the latency of each call, as well as tracking who made the request.
ESI-[endpoint]s (there are many)
The endpoints can be considered mini, stand-alone ESIs. If you found a way past the router cluster and hit the endpoint directly, you'd find it serves a swagger.json for each version it has routes for (in addition to dev, latest and legacy), but does not have any other user interface. All authentication and caching happens on the endpoint level. Each endpoint serves one or more routes in its own little cluster (behind a Kubernetes ClusterIP service), able to horizontally scale at will.
Would you like to know more?
Tech Co developers are regularly available on the tweetfleet slack (@ccp_bartender, @ccp_snowedin, @ccp_aquarhead, @ccpchimichanga), and we will be monitoring Reddit and the EVE Online forum thread if you've got questions or comments. We're especially keen to talk about anything you need from us to port your existing applications over from the XML API, so please stop by for a chat!
To the glorious future!
• Team Tech Co