Overnight Shard Hang

News and Announcements
Post Reply
User avatar
Derrick
Posts: 9004
Joined: Thu Dec 13, 2007 7:49 pm
Location: Cove
Contact:

Overnight Shard Hang

Post by Derrick »

Last night at about 11:30pm est (shortly after I went to bed) the shard hung up and didn't recover. We had a similar hang early in the day.

It's believed (but not yet proven) that these two hangs were both caused by the new "creature packs" AI in yesterdays patch. That code/feature has been disabled pending better analysis.

Usually when we do have a shard crash, which has been relatively infrequent in the history of UOSA, the shard will restart itself, with a possible slight revert (zero to 30 mins). However this is the second time in a week that we were down for extended period of time. We've not in the history of the shard had services outages of this magnitude and we are taking every precaution to assure that this doesn't happen again.

The shard crash Friday night that also took us offline for the night was a normal crash, and while the shard did attempt to restart itself, there was some new code that did not load properly. The problem that occurred Friday night that caused the crash was a code error that was fixed Saturday morning.

Crashes can be caused by a multitude of reasons. Most often the crashes on Second Age have been caused by code errors that have existed for a long long time before causing problems, typically triggered by an unusual situation or sequence of events. This was the case on Friday night, but the log revealed the source of the error and the code was able to be corrected right away. The second most common source of a crash is from new code and systems we introduce to the server. Although we do testing on all code before it's published, it's very hard to recreate the conditions of a live shard with hundreds of players in a test environment, so often any problems with new code are discovered on the live shard not long after the publish.

A shard hang as occurred last night is especially hard to troubleshoot becuase the shard does not crash and generate a log to help us identify the problem. So we are guessing this was new code, and the most complex of the code recently published is the pack AI.

This is a free shard, but we take very seriously our obligation to keep it up. While I and staff have put a lot of time into Second Age, we respect that players have too; we do not take this lightly and are committed to keeping Second Age stable and online.

User avatar
Nosferatu
Posts: 1
Joined: Wed Feb 11, 2009 3:25 am
Location: Carpathian Mountains, Transylvania

Re: Overnight Shard Hang

Post by Nosferatu »

Accidents happen dear shard admit, at least with yours, you didnt have to clean up any blood.

Kryptonical
UOSA Donor!!
UOSA Donor!!
Posts: 1209
Joined: Mon Jan 12, 2009 2:54 pm

Re: Overnight Shard Hang

Post by Kryptonical »

Thanks for all your hard work Derrick!

User avatar
Redbeard72
Posts: 92
Joined: Thu Sep 04, 2008 12:53 am
Location: Yew

Re: Overnight Shard Hang

Post by Redbeard72 »

You guys are the best !!!!! Always !

User avatar
Flash Hardstar
UOSA Donor!!
UOSA Donor!!
Posts: 1163
Joined: Thu Oct 09, 2008 4:52 pm
Location: Boston, Massachusetts

Re: Overnight Shard Hang

Post by Flash Hardstar »

on a positive note, thats the first i've seen it happen since last year!

Era Accuracy would be that happeneing quite often. :)

Shea
Posts: 565
Joined: Tue Feb 03, 2009 1:45 pm

Re: Overnight Shard Hang

Post by Shea »

Thanks for all of your hard work. Perfectly understandable :)

Ash
UOSA Donor!!
UOSA Donor!!
Posts: 138
Joined: Wed Jan 07, 2009 9:12 pm

Re: Overnight Shard Hang

Post by Ash »

Things happen, I really appreciate the time and dedication you and your staff puts towards this server.

Three Bones
Posts: 38
Joined: Fri Dec 05, 2008 11:50 pm

Re: Overnight Shard Hang

Post by Three Bones »

That is the final straw! I can't take it anymore...I want my money back.

Oh, what? You mean..I didn't pay 9.95 per month for this?

No seriously, totally free?

Well hell...I can't bitch then..

DrFaustus
UOSA Donor!!
UOSA Donor!!
Posts: 3151
Joined: Tue Oct 21, 2008 3:07 pm

Re: Overnight Shard Hang

Post by DrFaustus »

Nosferatu wrote:Accidents happen dear shard admit, at least with yours, you didnt have to clean up any blood.
Hint: No one can read your text.

Might be a good thing.

User avatar
Guerrilla
Posts: 660
Joined: Tue Jan 27, 2009 3:04 am
Location: Dirty South USA

Re: Overnight Shard Hang

Post by Guerrilla »

lol.... whatchu know bout that over night hang derrick?
:)

Post Reply

Return to “Shard News”