New CREST Resource for Bulk Market Orders
Hello everyone!
I wanted to inform you all about a new market resource in the CREST API. Until now, if you wanted to get all the market orders for Tranquility you had to request every type per region individually. That would generally be 50+ regions multiplied by upwards of 13,000 types. Even if it was just 13,000 types and 50 regions, that is 650,000 requests required to get all the market information. And if you wanted to get all the data in the 5-minute cache window, it would require almost 2,200 requests per second. While some of you came up with creative solutions to this problem such as prioritizing faster moving items, we decided we needed a better solution that was both easier to use and reduced load on our servers. As such, we have added a new market resource that is aimed at anyone who is trying to get all of the market data.
The new market resource can be found under /market/<regionID>/orders/all/ and returns all market orders for a given region paginated, with up to 10,000 results per page. For busy regions such as The Forge, you would expect around 30 pages of data based on the current trading volume. Here is a sample of the output:
{
"totalCount_str": "306700",
"pageCount": 31,
"items": [
{
"buy": false,
"issued": "2016-04-02T02:15:32",
"price": 5716.55,
"volumeEntered": 50000,
"stationID": 60003760,
"volume": 9756,
"range": "region",
"minVolume": 1,
"duration": 90,
"type": 21368,
"id": 4386193414
},
...
],
"next": {
"href": "https://api-sisi.testeveonline.com/market/10000002/orders/all/?page=2"
},
"totalCount": 306700,
"pageCount_str": "31"
}
One of the things you will note about this resource is that we have trimmed down each order to be very slim. We don't include a link to the type, instead we supply just a type ID. We don't include the _str attributes. Everything that could be stripped out has been in order to reduce this resource response in size. Be warned though, even in its stripped-down form each page is several megabytes in size.
This new resource is going live tomorrow on May 24th on TQ, and is currently live on Singularity. We are already working on some performance improvements to CREST and this resource, and are hoping that we can eventually increase the orders per page from 10k to 30k.
One of those changes we are making to help reach the speed and size we want for this resource is removing all the checks we do that verify the response matches the mediatype for this resource. The reason for that is that CREST spends a large amount of time verifying that and on a resource this size it ends up taking far too long. What this means to developers is that while we are mostly confident in the response being what it should, there is a chance that attributes might go missing or be of a type they shouldn't be with no warning. To be clear, this removal of response verification is only happening to this specific resource.
If you have any questions or concerns, please let us know. Hopefully this new resource helps elevate some of the work you are all having to do and makes life easier.
CCP FoxFour