Jump to content
Sign in to follow this  
Alex

3/15/2019 - API Usage Limitations

Recommended Posts

Hello everyone,

Even after doubling our server size recently, we are still struggling with excessive load times due to excessive load on the server. I believe that the source of this excessive load currently is due to heavy usage of the API endpoints. If you are unfamiliar with the game's API, then you can pretty much ignore the rest of this post as it's aimed at API users and will not impact you.

First, I understand that the API is poorly designed, poorly optimized, and all around needs to be completely re-done and expanded upon. It's near the top of my to-do list, I am finishing my undergraduate degree this semester and post-graduation I will be spending my time working on the game and will start getting big projects like re-factoring the API done.

However, in the meantime, the excessive load on the server is unacceptable. As a result, I am taking a blunt force action to curb the strain on the server until I can set up a proper API system.

Beginning at 12:00 AM server time 3/18/2019 all API endpoints will require your API key to work. This should give everyone a few days to get their ducks in a row before the change occurs. There may be bugs as it hasn't been thoroughly tested on all of the pages; if there are report them in the Tech Support subforum and I will have them quickly resolved.

API endpoints that already require your API key are not being changed by this update.

I am also reducing the number of API requests per user per day. The standard number of calls has been reduced from 10,000 to 2,000 and the number of calls for VIP users has been reduced from 999,999 to 5,000. When the API is refactored this will be changed to a rate limit instead of a daily limit, however I am simply working with the existing interface because I do not have time to rewrite the entire system currently.

I understand this is a significant decrease in the number of calls allowed. However, there are 15 API endpoint pages, if you call each of them every 5 minutes every hour per day, that's only 4,320 calls per day, which easily falls within the VIP limit. Realistically, you shouldn't need to call each page every 5 minutes anyway. As-is, the two highest request users are just barely reaching over 2,000 calls per day, with an average of the top 10 users being 764.7 calls, a median of 507, and a range of 1962. I understand not all calls are being recorded here, but the point is that for the average API user this should not be a significant limitation.

The point of this update is to curb usage of the API, and by creating a scarcity I am hoping to discourage people from making too many calls and causing ridiculous load times for regular users. Yes, it's a poor fix, but as I've stated it is a temporary solution until I have the time to properly refactor the API. I did not want to have to do this, but it seems to me a necessary sacrifice given the conditions the game is currently operating under.

If you have any questions, please ask below.

 

  • Upvote 7
  • Downvote 2

Share this post


Link to post
Share on other sites

Maybe I am missing something here, but you clearly stated that top two people are only making 4,320 calls, and the Avg person 780 calls, so will dropping the limits to 5,000 and 2,000 really don't do anything if most are not even close to those number of calls?.

think you hit it on the head when you clearly stated its poorly made, cutting the number of calls people can make to a number no one is reaching, I do not see how this fix the problem.

If I am not wrong most of the calls are about the market prices and such, you really want to change it then stop people being able to pull the market via the API and watch the lag stop.

  • Upvote 2

Share this post


Link to post
Share on other sites

tVKJgLh.png

Ofc, jokes aside, it's an understandable change and I can't fault you on this too much. If you think the servers are going to have a lot less load on them via this change, it'll generally be the best thing for the casual playerbase that doesn't really use the api. If the game loads faster, than I'm perfectly willing to compromise and have the cap.

What this might influence a lot is things like recruitment bots and of course Frawley's awesome stat tracker, so you might want to see if you can make a compromise for those 2, and maybe some other things. Recruitment bots are great because they pick up newer players which than have a lot larger retention than players that don't ever join alliances. Maybe to make that compromise you could add an option where people can use alliance keys instead that have a larger cap, maybe 10,000 or something.

Edited by Radoje
  • Upvote 2

Share this post


Link to post
Share on other sites
2 hours ago, Radoje said:

What this might influence a lot is things like recruitment bots and of course Frawley's awesome stat tracker, so you might want to see if you can make a compromise for those 2, and maybe some other things

Would a certain trade bot fall under "some other things"? 

  • Upvote 1
  • Downvote 1

Share this post


Link to post
Share on other sites
4 hours ago, MoonShadow said:

Maybe I am missing something here, but you clearly stated that top two people are only making 4,320 calls, and the Avg person 780 calls, so will dropping the limits to 5,000 and 2,000 really don't do anything if most are not even close to those number of calls?.

think you hit it on the head when you clearly stated its poorly made, cutting the number of calls people can make to a number no one is reaching, I do not see how this fix the problem.

Not all calls presently require a key and as such they are not tracked. 

  • Upvote 1

Share this post


Link to post
Share on other sites
2 hours ago, Darth Revan said:

Would a certain trade bot fall under "some other things"? 

Yes, a trade bot that called the api and showed you mistrades would fall under those other things. As would your military notification bot, a battle simulator sheet etc.

  • Upvote 1

Share this post


Link to post
Share on other sites

I think one of the underlying issues is less about the amount of calls to the server and more likely the amount of redundant calls to the server. Things like trade, war and other notification bots access the server constantly and pull data. As a community we have more than enough resources to be able to address this concern for Alex in the meantime.

 

In order to accomplish this we could simply implement the following:

 

  • A proper caching layer
    • Things like nation data and such. It changes but not extremely regularly
  • A distributed push notification system with a single system calling the server -- This data is relatively static and would only need to be pulled once from the server on a simple timed schedule
    • Trade Bots
    • War Bots
    • War Attack Bots

 

If this is something the community would be interested in I could work on it in the near future.

  • Upvote 1

Share this post


Link to post
Share on other sites
3 hours ago, Legoboyvdlp said:

Not all calls presently require a key and as such they are not tracked. 

aaahhh forgot about the general api, guess that will have a massive impact on the game 

Share this post


Link to post
Share on other sites
9 hours ago, Alex said:

I understand this is a significant decrease in the number of calls allowed. However, there are 15 API endpoint pages, if you call each of them every 5 minutes every hour per day, that's only 4,320 calls per day, which easily falls within the VIP limit. Realistically, you shouldn't need to call each page every 5 minutes anyway. As-is, the two highest request users are just barely reaching over 2,000 calls per day, with an average of the top 10 users being 764.7 calls, a median of 507, and a range of 1962. I understand not all calls are being recorded here, but the point is that for the average API user this should not be a significant limitation.

I'm not sure exactly where to start with your bad maths. Yes you might have 15 api endpoints, but those endpoints are not all equal. There are ~260 Alliances, ~10.7k Nations and ~61.9k Cities that are accessible from those endpoints.

Calling all those, even once per day, gives you a maximum of 73k potential calls per day.

Which is the root of the problem. It is impossible to get the information required for even the most basic of stuff, without delving into the individual API endpoints.

If you want to a proper short term solution, merge /alliance into /alliances, /nation and /city into /nations. Sure they'll be massive, especially in terms of nations, but hopefully no one should be fetching it more than once per day (and you can possibly rate limit it when you have time).

Good luck with your degree and if you ever want to discuss the API, feel free to DM me on Discord or tag me in #api

  • Upvote 1
  • Downvote 1

Share this post


Link to post
Share on other sites
30 minutes ago, Shakyr said:

I'm not sure exactly where to start with your bad maths. Yes you might have 15 api endpoints, but those endpoints are not all equal. There are ~260 Alliances, ~10.7k Nations and ~61.9k Cities that are accessible from those endpoints.

Calling all those, even once per day, gives you a maximum of 73k potential calls per day.

Which is the root of the problem. It is impossible to get the information required for even the most basic of stuff, without delving into the individual API endpoints.

If you want to a proper short term solution, merge /alliance into /alliances, /nation and /city into /nations. Sure they'll be massive, especially in terms of nations, but hopefully no one should be fetching it more than once per day (and you can possibly rate limit it when you have time).

Good luck with your degree and if you ever want to discuss the API, feel free to DM me on Discord or tag me in #api

"I'm not sure exactly where to start with your bad maths. Yes you might have 15 api endpoints, but those endpoints are not all equal. There are ~260 Alliances, ~10.7k Nations and ~61.9k Cities that are accessible from those endpoints.

Calling all those, even once per day, gives you a maximum of 73k potential calls per day."

no, no, annnnnnd no

I can agree with the merging of apis tho, having all the info of a nation/alliance/war/etc with one call is well worth having to sort on my end

  • Downvote 1

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
Sign in to follow this  

  • Recently Browsing   0 members

    No registered users viewing this page.

×

Important Information

By using this site, you agree to our Terms of Use and the Guidelines of the game and community.