Gaiscioch Select Chapter
POPULAR ADVENTURES:



ACTIVE ADVENTURES:





ADVENTURES:
Skull and Bones
Enshrouded
Once Human
Conan Exiles
Baldur's Gate 3
Ashes of Creation
Chrono Odyssey
Pax Dei
Dune Awakening
Guild Wars
Myth of Empires
- Full List -
CHAPTERS:
Chapter 8:
Conqueror's Blade (2019)
Chapter 7:
New World (2021)
Chapter 6:
World of Warcraft: Classic (2019)
Chapter 5:
Elder Scrolls Online (2014)
Chapter 4:
Guild Wars 2 (2012)
Chapter 3:
RIFT (2011)
Chapter 2:
Warhammer Online (2008)
Chapter 1:
Dark Age of Camelot (2001)
Community
Events
CHARITY:

LEGACY EVENTS:


Search Gaiscioch.com:
137 Tuatha Guilds:
8,334 Members:
14,100 Characters:
11,709 Items:
  • Views: 2,496
  • Replies: 7

Mythic Speaks About Badlands

Taoiseach de na Arach Glas
Foghladha
Taoiseach de na Arach Glas
  • GW2: Foghladha.2506
  • ESO: @Foley
Posted On: 10/01/2010 at 03:49 PM
  • Steam
  • PSN
  • XBOX
  • Twitch
  • Twitch
  • Twitter

It’s no secret that there are performance issues on Badlands, I know it’s frustrating as a player and it’s especially frustrating for us as developers. We have pride in our product and want you all to be able to enjoy it, so it’s been especially stressful on us trying to address the issues on Badlands and not seeing the results we want. I know ultimately what you all want is a solution, and we promise to continue working on it until it’s resolved. I want to give you all some insight into what we’ve done so far to address the issues, some of the success we’ve had and what we’ve got coming up to slay the demons plaguing WAR, and in particular Badlands.

We started recognizing and receiving issue reports from Badlands shortly after the reclamation of our European customers. There was such a breadth of reports it wasn’t immediately clear what was to blame so we begun to dig in. Quickly a few major culprits arose out of the reports: items getting lost in mail and from live events, some new client crashes we hadn’t seen before and the most serious and disruptive of all, getting kicked to character or server select when moving between areas in the game.

There were a lot of leads to follow, some issues crossed paths with other issues, some issues once analyzed were found to be a combination of two or three smaller issues, and some were dead ends. With each maintenance patch you’re seeing the results of our investigations. Honestly the maintenance patches of the past few months have been some of our most extensive maintenance patches that we’ve ever done for WAR in an effort to resolve the issues you all have been suffering as soon as possible. Right now I want to concentrate on what’s giving Badlands players and WAR developers heartburn lately.

It’s all about the data. We’ve now been operating WAR for over two years and during that time many characters and guilds have been created. We hold onto all that data and at this point we have in the MILLIONS of characters and thousands of guilds on Badlands alone. That’s a lot of data and it’s very important for us to keep it safe. It’s beyond just data, it’s the collective experiences of all of our players, their adventures, their heart pounding victories and their heroic defeats. It’s undoubtedly our most precious asset and we treat it as such.

Every day we perform backups on this precious data, usually in the early morning when server populations are at their lowest. On a server like Badlands this normally takes a few hours to run. However all this changed with the European reclamation, these days Badlands activity is almost always high at all times of the day. This is awesome in and of itself; however this means player activity is competing a lot more for disk time with our backups than it has in the past. Combined with how much data we have to backup on Badlands and the amount of action going on this is causing our backups to run for extended periods of time, it especially gets hairy when our Backups run into primetime which has been the case the past few weekends. Sometimes this is leading to backups that take longer than a day to finish.

What does all of this have to do with you getting kicked to character or server select when zoning? Zoning behind the scenes is a process which passes the player from one physical server to another physical server, as part of that process we save the character to disk before moving the player between servers and then load the character up on the other end when they arrive. When we are stuck in this state with backups and player activity competing for disk time queues begin to form and build up faster than the system can clear them. If your save doesn’t get processed within a certain amount of time the game believes something is wrong and removes the character from the game world as a precaution.

This has put us in a dilemma, we could simply turn off backups or modify them in some way and many of these issues would go away or be lessened significantly. However in doing that we put characters, guilds, campaign data and more at risk of being rolled back for multiple days or worse, being lost all together. Turning off backups also has a negative impact on the quality of service our Customer Service Department can deliver to our users which is very important to us. Turning off backups has not been an option for us, your characters are too important to us and to you to risk them in such a fashion.

So what have we been doing to try and fix this issue? Optimizing what data we save, how we save it and when we save it, removing unused code and data, adding diagnostics and logging, trying new hardware, trying new hardware configurations, limiting trial access to Badlands and that’s just the highlights on a long list. With each change that hasn’t solved the issue our next attempt at solving it has had an increased amount of risk and increase potential of return -- making changes that impact data and how we manage it, especially character data, is normally something we are extremely cautious about and try if at all possible to reserve for new versions instead of maintenance patches in order to get plenty of testing on it. For the most part all of our attempts at solving the issue have made improvements, but it hasn’t been enough.

We have more changes to address the issues that have been in testing and additional changes in production that we really believe will solve this issue once and for all. Some have had enough testing that we hope to provide them to you within the next few maintenance patches. As well we already have plans in motion in the case that they don’t solve the issues being experienced on Badlands.

I pulled some data to help put some perspective on just how much effort we’ve put behind resolving issues on the engineering side alone. We’ve made approximately 70 code changes directly addressing the various issues and forms of lag players have been reporting. To put that into perspective version 1.3.6 involved 156 code changes, and this is just the WAR engineering staff! Our server administrators, quality assurance, customer service and community teams have put countless hours into helping identify and address the issues on Badlands as well as everyone else working on WAR.

I apologize; we apologize for the fact that the issues haven’t been resolved at this point. It’s been our top priority for weeks at this point and will continue to be so until it is resolved.

Along this road there have been a number of successes as well two of which I want to call out in particular:
Client stability – Prior to version 1.3.6 going live we were okay with the stability of the client, we’d love for it to never crash and will not be happy until that goal is met. However we made a breakthrough a few weeks ago which has reduced by approximately half the number of client crashes we are seeing.

Server stability – Flash crowd region crashes are largely a thing of the past. We’ve seen crowds more than double what we could support just a few versions ago battling it out in Open RvR. This has been through an extensive process of optimizing the server code as well as client and server messaging. Today we’re seeing larger crowds than ever battle it out, all the while the server is hardly breaking a sweat. These larger crowds bring with them new issues unique to having so many players in an area and we acknowledge that even though the server doesn’t crash the play experience can suffer. During version 1.3.6 maintenance patches we’ve continued to improve the play experience in these situations and there’s more to come.

I know these successes pale in comparison to the issues Badlands players in particular have been experiencing lately, but it’s also a sign of what’s right around the corner once we get this sorted out – some of the best play experiences and biggest RvR battles you’ve ever seen.

We will continue to work on this issue and any other issue that severely impacts the WAR play experience and do so while working to be more open regarding communication going forward. As always, thank you for your support and we’ll see you on the battlefield.

John Cox
Development Manager
BioWare Mythic

» Edited on: 2010-10-01 16:12:00

"It's not the loot and accolades you walk away with, it's the memories and friendships that you cherish forever." - Foghladha
Awards & Achievements
Devotion Rank 20Valor Rank 20Fellowship Rank 10Explorer Rank 12Scholar Rank 15Artisan Rank 8Social Rank 9Mentorship Rank 6

Response:

Fili de na Griobhta
Dargron
Fili de na Griobhta
  • GW2: Dargron.3895
Replied On: 10/01/2010 at 05:24 PM PDT
  • Twitch

Good stuff... well, not that the issue is particularly problematic and persistant, but that EA/Bioware/Mythic are finally cluing us, the players, into what's going on behind the scenes. Reports like these really should be a regular feature. It instills alot of confidence into the playerbase just to know that they are aware of the problems and that bug reports aren't being ignored (or possibly never even recieved). When there's nothing but silence, that's when the conspiracy theorys start. ;)

Awards & Achievements
Devotion Rank 20Valor Rank 2Fellowship Rank 20Scholar Rank 4Artisan Rank 10Social Rank 1
Ridire de na Fhiaigh
Reclis
Ridire de na Fhiaigh
Replied On: 10/01/2010 at 06:29 PM PDT

I'll drink to that!! Stoats what kind of ale do you got fer this occasion??

Awards & Achievements
Devotion Rank 20Valor Rank 3Fellowship Rank 10Scholar Rank 4Artisan Rank 2
Caomhnoir de na Iomproidh
Docroc
Caomhnoir de na Iomproidh
Replied On: 10/01/2010 at 06:30 PM PDT
  • Steam

Reading this made me feel a lot better. I have even felt the improvement this week. Already resubed. :)

Awards & Achievements
Devotion Rank 20Valor Rank 7Fellowship Rank 11Scholar Rank 2
Curadh de na Ulchabhan
Bashir
Curadh de na Ulchabhan
Replied On: 10/01/2010 at 08:39 PM PDT

This makes a lot of sense to me. I support an SAP financial system and we had the similar system issues with data back ups running too long and causing performance issues. I just hope they can solve their problems faster than we solved ours.

Awards & Achievements
Devotion Rank 20Fellowship Rank 9Explorer Rank 3Scholar Rank 5Artisan Rank 3
Curadh de na Capall
Jaiice
Curadh de na Capall
Replied On: 10/02/2010 at 08:51 AM PDT
  • Twitch
  • Extra-Life

Great to finally hear some news. If they would have said something a month ago, it would have saved so many people alot of frustration knowing that they were working on it.

Awards & Achievements
Devotion Rank 20Fellowship Rank 6Scholar Rank 2Artisan Rank 3Social Rank 2
Caomhnoir de na Aracos
Stoat
Caomhnoir de na Aracos
  • GW2: Stoat.9671
Replied On: 10/02/2010 at 02:38 PM PDT

I'm glad to see that they are making progress, and have finally let us know that they are working on the problem. Dargron is right, that the wall-of-silence we were getting from BioWare / Mythic did nothing but breed conspiracy theories... _____ Ok, but to celebrate this new info... Lemme see what I've got behind the counter... [starts pouring shots of Mythic Mead] Hmmm... That won't get it by itelf [drags out a keg of Developer Draft] Thats it! [drops a shot glass into a full mug of brew] Step right up for some BioWare BoilerMakers! Reclis... you are up! Give it a try :)

» Edited on: 2010-10-02 14:39:42

Awards & Achievements
Devotion Rank 20Fellowship Rank 11Scholar Rank 5
Ridire de na Iomproidh
Tenebraeleo
Ridire de na Iomproidh
Replied On: 10/02/2010 at 06:43 PM PDT

[Downs a BioWare BoilerMaker] Mmmmm....Good Stuff there Stoat! [falls to floor unconscious] Note to self when I come to: Next time let Reclis go first, as Stoat directed.

Awards & Achievements
Devotion Rank 20Fellowship Rank 10
[0.1768]