Status of SWARFARM after update v8.0.9

With the newest patch, I'm forced to make some changes in how SWARFARM will function in the future in reference to the news I've made 2 days ago. In short, we are no longer able to store data logs, or update dungeon monster stats. Xzandro explained the reason for these changes, so let me quote him.

Quote from Xzandro

After today's patch, com2us introduced new encryption keys/methods on pretty much all game events. Only event that is unaffected is the one, where you directly export your profile data file, to use in for example SWOP and SWARFARM. This seems to be a deliberate decision by com2us, because there is no way, that this is coincidence. You could think that this is good news, but it is not really. Although SWOP is widely unaffected, SWARFARM's data logs will not work currently and pretty much all other 3rd party tools, that need automatic data other than the profile file provided by SWEX will not work. [...] Release of SWEX 0.0.55 I released a new version, that removed the Disabled Merge sealed monster storage option and set it to false by default, because this option currently prevents SWEX from exporting your profile data, since com2us uses new encryption there too. So either manually disable the Disabled Merge sealed monster storage option in the SWEX settings or update to the newest version.

As you can see, if you download newest SWEX version (or disable merging sealed monster shrine storage) you should still be able to export your profile data, but that's all.

What works and what not?

1. Bestiary data, f.e. skill multipliers - WORKS; SWARFARM wasn't using SWEX to obtain that data and its source hasn't changed
2. Profile data, f.e. list of monsters, fusion - (PARTIALLY) WORKS; as mentioned by Xzandro, profile command is still functioning properly and the only thing which may be missing (I didn't check that yet) is list of monsters located in sealed monster shrine storage
3. API - WORKS; strictly connected to Bestiary, works as intended
4. Data logs - FROZEN UNTIL FURTHER NOTICE; with commands from all events we've been gathering (dungeon runs, shop refreshes, wishes etc.) being encrypted, we can no longer extract the detailed info from them; it means that we no longer can keep an eye of drop rates changes made by com2us; it's probably biggest hit taken by SWARFARM in this update; basically 1/3rd of the site becomes obsolete
5. Boss & wave stats - FROZEN UNTIL FURTHER NOTICE IN v.8.0.8; this data was being parsed from command at the start of dungeon run; since we no longer can get that data, there is nothing I can do; reason same as for data logs
6. Personal logs - FROZEN UNTIL FURTHER NOTICE; same as Data logs

What about SWARFARM changes announced 2 days ago?

1. Optimization of profile parsing - UNAFFECTED
2. Database optimization - UNAFFECTED
3. Archive system - UNDECIDED
4. Browsing archived data - FROZEN UNTIL FURTHER NOTICE; there will be nothing to compare to
5. Fixing deadlocks - UNAFFECTED
6. Migrating to front-end framework - SPED UP
7. Creating more optimized API v3 - UNAFFECTED
8. Feedback & github issues cleanup - UNAFFECTED
9. Live sync - FROZEN UNTIL FURTHER NOTICE

The priority list has changed a little bit. Optimizing data logs seems quite meaningless after the update.

Final speech

I'm really sad that a lot of tools people were relying on will stop working. Community devs dedicated their personal time to make something for other members and it has been rendered useless by just one change. Com2us knew what they did, which they showed by profile parsing still being intact. Instead of coming to terms with some of the community's ideas, popular tools and starting to live in synergy with us, their actions have taken many of us back to the days of Excel. Most of you will probably not see any problem in actions taken by Com2us, because bestiary has been unaffected and most popular tool (SWOP) still works and it is ok, understandable. Right now, when Com2us makes some change to drop rates, we won't be able to collect as big simple size as before. People will need to start manually record their runs, which will result in... 2k, 5k? In comparison to SWARFARM's 2-weeks timespan 200k+ (as of writing this news, GAH has over 260k records). With Beast Horn drop rate being incredibely low (~1/2000), one person can not collect reliable data.

We'll see how it goes, but I don't expect Com2us to put resources into providing reliable and secure API for community tools. Neither of starting to show drop rates or skill multipliers in game either.

Status of SWARFARM

It's gotten a little dry here, huh? Don't worry, it's not a goodbye message. Right?RIGHT?

Introduction

Since the Reloaded Update, I've been adding new stuff which may or may not be useful for you as users, for ex. generating reports by filtered date or new "Rune Builds" menu option in Profile. Some I announced, some did not. But... I've neglected the site's performance for too long. You know that, I know that, everyone knows that. The site was running poorly, there was a lot of API errors, people have been (and still are) seeing "Server Error (500)". Lately I've heard a lot of phrases like "Great, SWARFARM is down AGAIN" and probably it's the push (or rather slap in my face) I needed.

With that in mind, I would like to apologize for not taking care of SWARFARM in terms of performance and stability. All I did in last one and a half, or even two years, was updating Bestiary with balance patches and new monsters. Would I call it strange? It depends. I'm not playing Summoners War for even longer, but I'm still responsible for this site. That lack of maintenance led to the problem I'm facing today. In order to resolve it, I'm currently working at a lot of things you may not even notice.

Status

1. Optimization of profile parsing - IN PROGRESS: as of today 4x faster locally, aiming to 10x; on server it will be even faster because I'm cutting off thousands of SQL queries (hi fellow IT guys) which doesn't have that much of an impact on local database with only one ongoing connection
2. Database optimization - IN PROGRESS: as of today 15x faster queries, aiming to... as fast as possible
3. Archive system - IN PROGRESS: instead of storing everything in one place, old data will be moved to other ("archived") tables; it should speed up generating reports
4. Browsing archived data - PROTOTYPING: I'm still thinking of the best solution to allow seeing old reports, for ex. from B12 before Reloaded update or B10 year ago
5. Fixing deadlocks - PLANNING: IT guys will know what I mean, but basically it's the situation when you are in a store and want to take milk from the shelf, but there is a man standing near it, so you pretend to look at the eggs until he goes; but of course, he waits until you go so he can take the eggs from shelf you are standing next to; you are both waiting for each other to move; the same happens with database connections in some cases
6. Migrating to front-end framework - RESEARCH PHASE: basically, Django Template Language isn't the best. Also SWARFARM code may be labelled as legacy; it still uses Django 2.2 where there is official 4.2 moving into LTS and 5.0 in development. Using proper framework would definitely improve user experience, but sadly I'm not a graphic designer and it's quite hard for me to come up with good design. Also, upgrading backend libraries could potentially increase the performance (with Python 3.11)
7. Creating more optimized API v3 - RESEARCH PHASE: at first I need to collect as much data as possible about how often API v1 & v2 are being used, which one is inefficient, what data is missing; of course v1 and v2 will remain functional for the sake of backward compatibility
8. Feedback & github issues cleanup - TO DO: basically, fixing errors reported there, adding new stuff based on feedback
9. Monster reports - ON HALT: some of old users may remember that SWARFARM started asking for consent to use your data in (un)anonymous reports about monster builds; something similar to old SWStats or in-game stats provided by Com2Us; the modal still shows for new users, but unfortunately I'm not planning to work on that, at least for now. Right now generating such reports is inefficient; maybe I'll get back to it once I do the performance & stability stuff; no promises here
10. Live Sync - TURNED OFF/ON HALT: too much workload for database right now, we'll see after performance & stability stuff

Update? When?

No date announced. I don't want to work under pressure of time. I want to make this update as good as possible.

Known limitation

Server in which SWARFARM is running still uses HDD instead of SSD. It drastically decreases the capabilites of our database. The best option would be to upgrade from current dedicated server to newer one, with SSD and better parameters, buuuuut... It's a lot of time from planning, through testing to executing to set everything up so you won't notice any downtime. Money is also a factor, infrastructure is not free. Porksmash's policy was to make SWARFARM completely free no matter what. I stand by that. So don't worry, there won't be any paywall, neither ads. That being said, all that stuff I'm working on is so we don't need to move to better/newer dedicated server. First software, then hardware. The only "income" for SWARFARM is Patreon, which still being taken care of by Porksmash. From what I know, current patreons donate enough to make the server pay for itself with a slight excess. As you may be guessing, it's not enough to get better/newer server. In my country, something great for SWARFARM costs about 90$/month, not including backup storage etc. As I mentioned earlier, I'm not going to implement any paywall or add ads on the site. I'm grateful to you all for supporting this project and just wanted to let you know what limitations we are facing. Thus, I would like to thank all the patreons for support you provide.

Some numbers for nerds like me

1. Almost 600 000 000 records in data log tables since March 2019; these include runs, shop refreshes, wishes, drops; basically everything we collect automatically through SWEX profile
2. In just september (~12days), we've collected over 7 000 000 more data logs, which means that SWARFARM gets at least 7 requests from SWEX per second; not to mention visitors, auto profile upload, and all these other community tools (like SWOP) getting data directly from us
3. We store almost 10 000 000 monster instances, from which "only" ~130 000 are LD5's (incl. farmable)
4. Monsters? Who cares... Runes is what's important in this game - currently approx. 30 000 000 in our database
5. What about you? Over freaking 50 000 registered users!

* I will share more data after performance & stability update

Worth mentioning

• Xzandro - for being... Xzandro, everyone knows him. On a serious note, for all the help he gives
• Cerusa - for taking care of all skill effects, which need to be updated manually every balance patch
• Seiishizo - for providing a good feedback and promoting site

Pepperidge farm remembers

Did you know that Porksmash was working on SWARFARM UI version back in 2019? It hasn't been touched since then

Last words (feel free to ignore)

Because of my professional life, I forgot how fun it is to work on side project in which you don't have deadlines (except site-breaking updates) and are in full control. I've been burned out because of stuff I had to do and had no control over no matter how stupid it was. Happily it changed and now the fun of working as a programmer is back. So I hope that I'll be working on this site for a little longer.

By the way, is anyone reading these news? lol