Administrators Popular Post Alex Posted March 15, 2019 Administrators Popular Post Share Posted March 15, 2019 Hello everyone, Even after doubling our server size recently, we are still struggling with excessive load times due to excessive load on the server. I believe that the source of this excessive load currently is due to heavy usage of the API endpoints. If you are unfamiliar with the game's API, then you can pretty much ignore the rest of this post as it's aimed at API users and will not impact you. First, I understand that the API is poorly designed, poorly optimized, and all around needs to be completely re-done and expanded upon. It's near the top of my to-do list, I am finishing my undergraduate degree this semester and post-graduation I will be spending my time working on the game and will start getting big projects like re-factoring the API done. However, in the meantime, the excessive load on the server is unacceptable. As a result, I am taking a blunt force action to curb the strain on the server until I can set up a proper API system. Beginning at 12:00 AM server time 3/18/2019 all API endpoints will require your API key to work. This should give everyone a few days to get their ducks in a row before the change occurs. There may be bugs as it hasn't been thoroughly tested on all of the pages; if there are report them in the Tech Support subforum and I will have them quickly resolved. API endpoints that already require your API key are not being changed by this update. I am also reducing the number of API requests per user per day. The standard number of calls has been reduced from 10,000 to 2,000 and the number of calls for VIP users has been reduced from 999,999 to 5,000. When the API is refactored this will be changed to a rate limit instead of a daily limit, however I am simply working with the existing interface because I do not have time to rewrite the entire system currently. I understand this is a significant decrease in the number of calls allowed. However, there are 15 API endpoint pages, if you call each of them every 5 minutes every hour per day, that's only 4,320 calls per day, which easily falls within the VIP limit. Realistically, you shouldn't need to call each page every 5 minutes anyway. As-is, the two highest request users are just barely reaching over 2,000 calls per day, with an average of the top 10 users being 764.7 calls, a median of 507, and a range of 1962. I understand not all calls are being recorded here, but the point is that for the average API user this should not be a significant limitation. The point of this update is to curb usage of the API, and by creating a scarcity I am hoping to discourage people from making too many calls and causing ridiculous load times for regular users. Yes, it's a poor fix, but as I've stated it is a temporary solution until I have the time to properly refactor the API. I did not want to have to do this, but it seems to me a necessary sacrifice given the conditions the game is currently operating under. If you have any questions, please ask below. 11 3 Is there a bug? Report It | Not understanding game mechanics? Ask About It | Got a good idea? Suggest ItForums Rules | Game Link Link to comment Share on other sites More sharing options...
Tymoteusz Posted March 15, 2019 Share Posted March 15, 2019 Oh come on Link to comment Share on other sites More sharing options...
Doughnuts Posted March 15, 2019 Share Posted March 15, 2019 Maybe I am missing something here, but you clearly stated that top two people are only making 4,320 calls, and the Avg person 780 calls, so will dropping the limits to 5,000 and 2,000 really don't do anything if most are not even close to those number of calls?. think you hit it on the head when you clearly stated its poorly made, cutting the number of calls people can make to a number no one is reaching, I do not see how this fix the problem. If I am not wrong most of the calls are about the market prices and such, you really want to change it then stop people being able to pull the market via the API and watch the lag stop. 2 Link to comment Share on other sites More sharing options...
Guest Frawley Posted March 15, 2019 Share Posted March 15, 2019 Erm, I might pm you about the stats site. Link to comment Share on other sites More sharing options...
True King Posted March 15, 2019 Share Posted March 15, 2019 Seems like a big decrease to give VIP users half what standard ones were getting. Link to comment Share on other sites More sharing options...
Radoje Posted March 15, 2019 Share Posted March 15, 2019 (edited) Ofc, jokes aside, it's an understandable change and I can't fault you on this too much. If you think the servers are going to have a lot less load on them via this change, it'll generally be the best thing for the casual playerbase that doesn't really use the api. If the game loads faster, than I'm perfectly willing to compromise and have the cap. What this might influence a lot is things like recruitment bots and of course Frawley's awesome stat tracker, so you might want to see if you can make a compromise for those 2, and maybe some other things. Recruitment bots are great because they pick up newer players which than have a lot larger retention than players that don't ever join alliances. Maybe to make that compromise you could add an option where people can use alliance keys instead that have a larger cap, maybe 10,000 or something. Edited March 15, 2019 by Radoje 5 Link to comment Share on other sites More sharing options...
True King Posted March 15, 2019 Share Posted March 15, 2019 I think for VIP, he should make the cap 10k if possible. So not as big of a decrease & would encourage VIP more. 1 1 Link to comment Share on other sites More sharing options...
Darth Revan Posted March 15, 2019 Share Posted March 15, 2019 2 hours ago, Radoje said: What this might influence a lot is things like recruitment bots and of course Frawley's awesome stat tracker, so you might want to see if you can make a compromise for those 2, and maybe some other things. Would a certain trade bot fall under "some other things"? 2 1 Link to comment Share on other sites More sharing options...
True King Posted March 15, 2019 Share Posted March 15, 2019 I think he should just flat out boost it to 10k for VIP; no compromises on specific uses. 1 Link to comment Share on other sites More sharing options...
mksinky Posted March 15, 2019 Share Posted March 15, 2019 4 hours ago, MoonShadow said: Maybe I am missing something here, but you clearly stated that top two people are only making 4,320 calls, and the Avg person 780 calls, so will dropping the limits to 5,000 and 2,000 really don't do anything if most are not even close to those number of calls?. think you hit it on the head when you clearly stated its poorly made, cutting the number of calls people can make to a number no one is reaching, I do not see how this fix the problem. Not all calls presently require a key and as such they are not tracked. 3 Link to comment Share on other sites More sharing options...
Radoje Posted March 16, 2019 Share Posted March 16, 2019 2 hours ago, Darth Revan said: Would a certain trade bot fall under "some other things"? Yes, a trade bot that called the api and showed you mistrades would fall under those other things. As would your military notification bot, a battle simulator sheet etc. 1 Link to comment Share on other sites More sharing options...
PDunny Posted March 16, 2019 Share Posted March 16, 2019 I think one of the underlying issues is less about the amount of calls to the server and more likely the amount of redundant calls to the server. Things like trade, war and other notification bots access the server constantly and pull data. As a community we have more than enough resources to be able to address this concern for Alex in the meantime. In order to accomplish this we could simply implement the following: A proper caching layer Things like nation data and such. It changes but not extremely regularly A distributed push notification system with a single system calling the server -- This data is relatively static and would only need to be pulled once from the server on a simple timed schedule Trade Bots War Bots War Attack Bots If this is something the community would be interested in I could work on it in the near future. 4 Link to comment Share on other sites More sharing options...
Doughnuts Posted March 16, 2019 Share Posted March 16, 2019 3 hours ago, Legoboyvdlp said: Not all calls presently require a key and as such they are not tracked. aaahhh forgot about the general api, guess that will have a massive impact on the game Link to comment Share on other sites More sharing options...
katashimon13 Posted March 16, 2019 Share Posted March 16, 2019 laughs in cache rawr 1 Link to comment Share on other sites More sharing options...
Shakyr Posted March 16, 2019 Share Posted March 16, 2019 9 hours ago, Alex said: I understand this is a significant decrease in the number of calls allowed. However, there are 15 API endpoint pages, if you call each of them every 5 minutes every hour per day, that's only 4,320 calls per day, which easily falls within the VIP limit. Realistically, you shouldn't need to call each page every 5 minutes anyway. As-is, the two highest request users are just barely reaching over 2,000 calls per day, with an average of the top 10 users being 764.7 calls, a median of 507, and a range of 1962. I understand not all calls are being recorded here, but the point is that for the average API user this should not be a significant limitation. I'm not sure exactly where to start with your bad maths. Yes you might have 15 api endpoints, but those endpoints are not all equal. There are ~260 Alliances, ~10.7k Nations and ~61.9k Cities that are accessible from those endpoints. Calling all those, even once per day, gives you a maximum of 73k potential calls per day. Which is the root of the problem. It is impossible to get the information required for even the most basic of stuff, without delving into the individual API endpoints. If you want to a proper short term solution, merge /alliance into /alliances, /nation and /city into /nations. Sure they'll be massive, especially in terms of nations, but hopefully no one should be fetching it more than once per day (and you can possibly rate limit it when you have time). Good luck with your degree and if you ever want to discuss the API, feel free to DM me on Discord or tag me in #api 2 1 Link to comment Share on other sites More sharing options...
Verin Posted March 16, 2019 Share Posted March 16, 2019 30 minutes ago, Shakyr said: I'm not sure exactly where to start with your bad maths. Yes you might have 15 api endpoints, but those endpoints are not all equal. There are ~260 Alliances, ~10.7k Nations and ~61.9k Cities that are accessible from those endpoints. Calling all those, even once per day, gives you a maximum of 73k potential calls per day. Which is the root of the problem. It is impossible to get the information required for even the most basic of stuff, without delving into the individual API endpoints. If you want to a proper short term solution, merge /alliance into /alliances, /nation and /city into /nations. Sure they'll be massive, especially in terms of nations, but hopefully no one should be fetching it more than once per day (and you can possibly rate limit it when you have time). Good luck with your degree and if you ever want to discuss the API, feel free to DM me on Discord or tag me in #api "I'm not sure exactly where to start with your bad maths. Yes you might have 15 api endpoints, but those endpoints are not all equal. There are ~260 Alliances, ~10.7k Nations and ~61.9k Cities that are accessible from those endpoints. Calling all those, even once per day, gives you a maximum of 73k potential calls per day." no, no, annnnnnd no I can agree with the merging of apis tho, having all the info of a nation/alliance/war/etc with one call is well worth having to sort on my end 1 Link to comment Share on other sites More sharing options...
Popular Post Sketchy Posted March 16, 2019 Popular Post Share Posted March 16, 2019 @Alex if you are going to do this, can you at least give each person 2 seperate api keys, one for public apis (war, alliances/nations etc) and one for private apis (alliance members, bank)? I can't grant edit access to my sheets to anyone now because they'll just be able to open up the script editor and look directly at my api key, then use it to pull whatever private alliance api data they like. 2 14 Link to comment Share on other sites More sharing options...
Artifex Posted March 18, 2019 Share Posted March 18, 2019 (edited) Remove captcha. Edited March 18, 2019 by Malleator 1 1 Love you Link to comment Share on other sites More sharing options...
Dusty Posted March 22, 2019 Share Posted March 22, 2019 On 3/16/2019 at 2:17 AM, Sketchy said: @Alex if you are going to do this, can you at least give each person 2 seperate api keys, one for public apis (war, alliances/nations etc) and one for private apis (alliance members, bank)? I can't grant edit access to my sheets to anyone now because they'll just be able to open up the script editor and look directly at my api key, then use it to pull whatever private alliance api data they like. Honestly this would be immensely useful 1 Link to comment Share on other sites More sharing options...
Recommended Posts