Status of SWARFARM

It's gotten a little dry here, huh? Don't worry, it's not a goodbye message. Right?RIGHT?

Introduction

Since the Reloaded Update, I've been adding new stuff which may or may not be useful for you as users, for ex. generating reports by filtered date or new "Rune Builds" menu option in Profile. Some I announced, some did not. But... I've neglected the site's performance for too long. You know that, I know that, everyone knows that. The site was running poorly, there was a lot of API errors, people have been (and still are) seeing "Server Error (500)". Lately I've heard a lot of phrases like "Great, SWARFARM is down AGAIN" and probably it's the push (or rather slap in my face) I needed.

With that in mind, I would like to apologize for not taking care of SWARFARM in terms of performance and stability. All I did in last one and a half, or even two years, was updating Bestiary with balance patches and new monsters. Would I call it strange? It depends. I'm not playing Summoners War for even longer, but I'm still responsible for this site. That lack of maintenance led to the problem I'm facing today. In order to resolve it, I'm currently working at a lot of things you may not even notice.

Status

1. Optimization of profile parsing - IN PROGRESS: as of today 4x faster locally, aiming to 10x; on server it will be even faster because I'm cutting off thousands of SQL queries (hi fellow IT guys) which doesn't have that much of an impact on local database with only one ongoing connection
2. Database optimization - IN PROGRESS: as of today 15x faster queries, aiming to... as fast as possible
3. Archive system - IN PROGRESS: instead of storing everything in one place, old data will be moved to other ("archived") tables; it should speed up generating reports
4. Browsing archived data - PROTOTYPING: I'm still thinking of the best solution to allow seeing old reports, for ex. from B12 before Reloaded update or B10 year ago
5. Fixing deadlocks - PLANNING: IT guys will know what I mean, but basically it's the situation when you are in a store and want to take milk from the shelf, but there is a man standing near it, so you pretend to look at the eggs until he goes; but of course, he waits until you go so he can take the eggs from shelf you are standing next to; you are both waiting for each other to move; the same happens with database connections in some cases
6. Migrating to front-end framework - RESEARCH PHASE: basically, Django Template Language isn't the best. Also SWARFARM code may be labelled as legacy; it still uses Django 2.2 where there is official 4.2 moving into LTS and 5.0 in development. Using proper framework would definitely improve user experience, but sadly I'm not a graphic designer and it's quite hard for me to come up with good design. Also, upgrading backend libraries could potentially increase the performance (with Python 3.11)
7. Creating more optimized API v3 - RESEARCH PHASE: at first I need to collect as much data as possible about how often API v1 & v2 are being used, which one is inefficient, what data is missing; of course v1 and v2 will remain functional for the sake of backward compatibility
8. Feedback & github issues cleanup - TO DO: basically, fixing errors reported there, adding new stuff based on feedback
9. Monster reports - ON HALT: some of old users may remember that SWARFARM started asking for consent to use your data in (un)anonymous reports about monster builds; something similar to old SWStats or in-game stats provided by Com2Us; the modal still shows for new users, but unfortunately I'm not planning to work on that, at least for now. Right now generating such reports is inefficient; maybe I'll get back to it once I do the performance & stability stuff; no promises here
10. Live Sync - TURNED OFF/ON HALT: too much workload for database right now, we'll see after performance & stability stuff

Update? When?

No date announced. I don't want to work under pressure of time. I want to make this update as good as possible.

Known limitation

Server in which SWARFARM is running still uses HDD instead of SSD. It drastically decreases the capabilites of our database. The best option would be to upgrade from current dedicated server to newer one, with SSD and better parameters, buuuuut... It's a lot of time from planning, through testing to executing to set everything up so you won't notice any downtime. Money is also a factor, infrastructure is not free. Porksmash's policy was to make SWARFARM completely free no matter what. I stand by that. So don't worry, there won't be any paywall, neither ads. That being said, all that stuff I'm working on is so we don't need to move to better/newer dedicated server. First software, then hardware. The only "income" for SWARFARM is Patreon, which still being taken care of by Porksmash. From what I know, current patreons donate enough to make the server pay for itself with a slight excess. As you may be guessing, it's not enough to get better/newer server. In my country, something great for SWARFARM costs about 90$/month, not including backup storage etc. As I mentioned earlier, I'm not going to implement any paywall or add ads on the site. I'm grateful to you all for supporting this project and just wanted to let you know what limitations we are facing. Thus, I would like to thank all the patreons for support you provide.

Some numbers for nerds like me

1. Almost 600 000 000 records in data log tables since March 2019; these include runs, shop refreshes, wishes, drops; basically everything we collect automatically through SWEX profile
2. In just september (~12days), we've collected over 7 000 000 more data logs, which means that SWARFARM gets at least 7 requests from SWEX per second; not to mention visitors, auto profile upload, and all these other community tools (like SWOP) getting data directly from us
3. We store almost 10 000 000 monster instances, from which "only" ~130 000 are LD5's (incl. farmable)
4. Monsters? Who cares... Runes is what's important in this game - currently approx. 30 000 000 in our database
5. What about you? Over freaking 50 000 registered users!

* I will share more data after performance & stability update

Worth mentioning

• Xzandro - for being... Xzandro, everyone knows him. On a serious note, for all the help he gives
• Cerusa - for taking care of all skill effects, which need to be updated manually every balance patch
• Seiishizo - for providing a good feedback and promoting site

Pepperidge farm remembers

Did you know that Porksmash was working on SWARFARM UI version back in 2019? It hasn't been touched since then

Last words (feel free to ignore)

Because of my professional life, I forgot how fun it is to work on side project in which you don't have deadlines (except site-breaking updates) and are in full control. I've been burned out because of stuff I had to do and had no control over no matter how stupid it was. Happily it changed and now the fun of working as a programmer is back. So I hope that I'll be working on this site for a little longer.

By the way, is anyone reading these news? lol

Rift Beast filter by Damage, up Personal Logs max threshold

Hey! Today I've added option to filter Rift Beasts reports by damage (score), next to filter by date. You can use both, only one at the same time or neither. Keep in mind that it generates new reports, which may take some time. Don't spam the "Filter" button, thanks.

Also, small QoL for personal data logs. I've increased maximum threshold from 2k to 4k. I'll keep observing how long it takes to load data. If server can keep up, there is a possibility to increase the cap even further.

New Rune Builds section & uncovering RTA Builds

Hello everyone. Today I have 2* new updates for you.

1. Some of you may know that for a loooong time, while checking monster details, there was RTA section hidden behind blur with "IN PROGRESS" message. Guess what, it's now visible. Basically, it shows monsters' equipped RTA build. The only difference between RTA & normal build is that you can't edit RTA build yet... And I don't know when (if) it'll be available.

2. By checking your Profile, you should find new option in the side menu, under "Artifacts", named "Rune builds". It shows your rune build stats WITHOUT taking into consideration monster data - it sums runes and artifact bonuses, then displays it. Basically, you can compare different builds in term of hard data without worrying about monster base stats. It also adds set effects, so having active Rage set will result in +40% CRIT DMG, Swift set in +25% SPD etc.

* Also, some other fixes you wouldn't even notice.

I hope you find it helpful. Let me know if you experience some issues with new features.

[BETA] Date filter for dungeon reports

Hello everyone. Since Com2Us decided to (once more) change drop rates in new dungeons, even before 2-week timespan (which we use as a standard in SWARFARM), I've added a way for you to check drop rates by given time frame. In order to use this feature, just go to given dungeon (i.e. GBA2 [sadly, it's not Game Boy Advance 2]), scroll to the section with sample size and you'll see quite simple form filter. Just select dates and filter.

FYI, 2-week is still our standard timeframe, and you can't go higher than that. If you filter by more than 2 weeks, it'll shrink in respect to end date.

It ain't much, but it's honest work.

RELOADED Update (v8.0.0)

All changes related to RELOADED update has been implemented. New dungeons, new runes, new artifact substats. By using combo 5+Intangible, 3+2+Intangible or 2+2+1+Intangible, rune set should be set properly, i.e. 3 Violent + 2 Will + Intangible should result in Violent + Will.

Skill effects for new and changed monsters will be adjusted in the near future.

As for now, I'm leaving upgrade level next to artifacts. It may help people find single not refined artifacts in their storage.