April 7, 2019

Personal Data Logs Now Available

Yes, the Data Logs button disappeared from the top bar. No, don't panic! There is a new section available when viewing your profile to access your personal data logs. At the moment this is simply a table view of data logs, with no statistical reports. If you care, check your logs for accuracy and please report any issues in the feedback section.

Here's the short-term roadmap, which covers all the steps involved in item #3 in the previous post:

  1. Completed Table based views for personal data logs
  2. Personal log statistical reports
  3. Integration of dungeon data into bestiary
  4. Globally aggregated statistical reports integrated into bestiary.
March 9, 2019

Status of Data Logging

If you've opened up SW-Exporter with the SwarfarmLogger plugin enabled, you'll see that instead of a big red error message, there is a notice that SWARFARM is now logging a few in-game events. The new logging framework is being slowly rolled out over the next few weeks. There will be at least 3 phases before the logging is fully functional again:

  1. Gradual reintroduction of all data logging and debugging DONE
  2. Reset logs (if necessary) and start permanent data logging NOT REQUIRED
  3. Introduction of new reports
  4. SW-Exporter plugin updates

The new reports will come along with an update to the bestiary to add dungeon information. The logs will be boiled down into static reports generated weekly or every 2 weeks (depending on the volume of data) and you will be able to scroll back in time week by week. The data will be presented alongside other dungeon information such as enemy waves.

Personal logs will now be integrated into your profile section of the website and have a slightly different format, including a tabular view of the actual data from your runs. Personal logs will also continue to be fully dynamic and not subject to any caching or static report generation.

Currently Logging: EVERYTHING!

  • Summoning
  • Scenarios
  • Caiross Dungeons
  • Magic Shop refreshes
  • Wishes
  • Rune Crafting
  • Magic Box Crafting
  • Elemental Rifts
  • World Boss
  • Raids

Still in the works: NOTHING!

Feb. 13, 2019

What's changed lately

After coming back online from the loss of the old server, the first order of business was ensuring that the loss of all data could never happen again. This was accomplished by:

  1. Owning all server resources directly
  2. Nightly full database snapshots, retained for 90 days
  3. Hot-Standby database server running in a different datacenter
  4. Backup job reporting and alerts

As I detailed previously, the server resources used to be completely donated and all I had to pay out of pocket for was the domain name. That is quite a different story now, but well worth it to ensure that the rug won't be pulled out from under us. To cover these costs I have set up a Patreon account. I greatly appreciate those who have contributed to the past through donations, and Patreon will help me continue to devote resources and time to development in addition to covering the monthly costs.

Despite outward appearances, development has never really stopped on SWARFARM. The majority of recent work has been in the background. I provide an API for monster and skill data, which powers many Discord bots, monster data in the SWOP rune optimizer (<3 Xzandro), and even other websites. This API has been expanded to include most of the data available on the website, including user profiles, and will eventually power the complete rewrite of SWARFARM, which can be previewed here.

I am still working to bring the data logging back online as the first priority. The loss of all of the data provided a bit of an opportunity to correct a lot of early decisions that proved to be difficult to work with.

Jan. 31, 2019

Back in business

After a whirlwind of code updates and a bit of shopping around for a new host, SWARFARM is back. Given the nature of the data loss experienced, you will have to re-register and import your profiles again. A small note to keep in mind is that you can no longer register the same username with different case (e.g. porksmash and Porksmash cannot both be registered). Thank you to everyone who was supportive and patient during this time.

Data logs are still currently disabled. In about a week the logging will be turned back on, and then a couple weeks after that the reports will go live.

Jan. 28, 2019

Unmitigated Disaster

For those who did not know, the server resources for the last year and a half or so were donated. That virtual server has either gone offline or been wiped out. I have been unable to contact the person who donated that server and am moving forward assuming it is permanently gone.

Unfortunately, the backups for SWARFARM have not been functioning for quite some time. It is embarrassing to admit my complacency has resulted in the loss of all data. I will have to bring the site back to life in a brand new state. The only comfort is that your user profiles (except teams) can be quickly restored thanks to the import process. Losing the data logs is a disaster, though, as I know many people used it for personal tracking and the global statistics were referenced quite frequently.

I appreciate the numerous offers for help from people on reddit and Discord. At this point I have everything I need to bring the site back in the next couple days, and then quickly move towards my own hosting which I am in full control of. In between now and this weekend I am going to resolve some long-standing bad decisions I made very early on in the development process, and then SWARFARM will be reopened. Until then, the read-only portions of the site such as the Bestiary will remain available to browse and via API for tools such as the Discord bot.