Been away while the last few days of fixes/updates/features have gone into place. Quick summation below.
1) Quick links -- "SQUEE!":http://tvtropes.org/pmwiki/pmwiki.php/Main/Squee
2) Campaigns Page -- FINALLY!!! What appears to be the single most-requested item has been returned!!! (Not perfect, but working well enough, and will improve with time.)
3) Navigation coloring -- Wasn't a huge problem with me, would prefer a partial reverse of colors (dark themes rule!) but it's definitely more legible on screens.
4) DST coloring -- I always forget that page exists...but at least it's unified in color scheme.
5) "desc" -- "BRILLIANT!":http://imgur.com/VkIANNz
Still getting 502/504 errors as well. Can maybe pull up a page once an hour. If I get lucky while trying it will continue to work for a few minutes before failure again.
The biggest issue I have with modern social software companies, is rollout dates. Rollout for new changes should always be on MONDAY, that way, you have all week to fix whatever you may have broke.
Rolling out later in the week just means your customers feel it the most, as they use it during the weekend, when you aren't at work.
While I agree completely that it sucks, at least andrew let us know in advance that they couldn't get things situated in advance and that the weekend/busy period was going to be impacted. Not ideal obviously, but at least I could plan accordingly.
Our session last night was severely impacted by the 502/504 gateway errors. My group uses OP heavily, from GM prep and notes, in-character discussions both during and after sessions, maps, images, character sheets, etc. I ended up DMing our 6-hour game with one NPC page that would load and straight memory. One player didn't have a physical character sheet, so they also just winged it as best as possible. I love the flexibility OP has and both the functionality and the ability to use custom CSS really makes it valuable to me (Ascendant membership for the last year or so), but last night burned me pretty bad, so I will most likely be going back to good old paper for critical information, or at least as a draft so enough of the material exists elsewhere. Unfortunately, wasn't seeing the gateway errors until it was too late and we were already into the game.
I posted an update in the "502 error thread":http://forums.obsidianportal.com/comments.php?DiscussionID=4065&page=2#Comment_35307 for those who had difficulty accessing the site over the peak Fri/Sat gaming period.
I don't want to shirk responsbility for the increased outage over this weekend, and we could have accelerated the hardware upgrade process.
After announcing the bug fixes/feature upgrades, our announcements seem to have viralized over social media or been posting to gaming groups, so that led to a higher-than-expected number of OP users, including new accounts, hitting the site over the weekend.
So instead of getting a better gaming experience, due to the fixes, many of you experienced 500/502 gateway errors instead.
I'll be meeting the tech team on Monday and working on improving our hardware by Wednesday, ahead of next weekend's heavy Fri/Sat gaming periods, so I am working towards minimizing this situations re-occurring.
Andreww, it is still happening. Not just Friday and Saturday. Sunday as well. Since Friday I am unable to access any of my sites to do anything. This is beyond frustrating and I hope it gets fixed tomorrow (Monday).
As of now, Obsidian Portal is unusuable
Keelah Se'lai,
Keryth
"Shadows Over New York":http://www.obsidianportal.com/campaigns/shadows-over-new-york
"2013 Campaign of The Year":http://blog.obsidianportal.com/2013-coty-shadows-over-new-york/
"Campaign of the Month July 2013":http://blog.obsidianportal.com/shadows-over-new-york-julys-campaign-of-the-month/
Now it is Sunday/Monday 12:30 am Pacific, 3:30 am Eastern, and 4:30pm Singapore time. Still not working, at or nearing end of work day Monday for the programmers.
Hi all, initial reports from the tech team is that during the hardware upgrade process, some of the links were broken within OP. They're working on restoring the site to full functionality now. Will post an update when it's done.
The techs have completed their diagnostic and repaired the links that were broken during the upgrade process. OP should be working as it should now, and you might experience a performance increase and fewer 500/502 errors when you're on the site, editing wikis.
I had pushed for the hardware upgrade on Friday to try to get the performance boost ahead of the weekend, so I apologize for the disruption to your gaming sessions over the weekend.
We'll be pushing our upgrades on Monday/Tuesday so that there'll be a lower chance of outages hitting in the later part of the week.
We've replenished our email quota, though the emails are stuck in a "scheduled email queue".
The last time Alex worked with our developers on this, the moment it was cleared, OP's email system sent out like 50k emails in a matter of minutes.
Obviously, we don't want this to recur, so our techs are working on resolving this in the next couple of hours and are working on releasing the "scheduled emails" into the immediate send queue.
This means that email notifications will return to normal once this is fixed.
We're working towards getting this done by Tuesday morning.
The techs and I have been testing campaign invites, wiki forum post notifications, update notifications, they seem to be working correctly.
There may be some bugs in specific OP accounts.
If you're still facing issues, post in the "support thread":http://help.obsidianportal.com/discussions/problems/19507-any-ideas-for-what-to-do-when-invites-dont-send and we'll have the tech look specifically at your account.
Note that they may send invites from your account, so please don't be alarmed if it looks like someone's been eating your porridge, sleeping in your bed, sending invites from your account. Cos they'll need to do that to troubleshoot your account.
Comments
- Kallak
1) Quick links -- "SQUEE!":http://tvtropes.org/pmwiki/pmwiki.php/Main/Squee
2) Campaigns Page -- FINALLY!!! What appears to be the single most-requested item has been returned!!! (Not perfect, but working well enough, and will improve with time.)
3) Navigation coloring -- Wasn't a huge problem with me, would prefer a partial reverse of colors (dark themes rule!) but it's definitely more legible on screens.
4) DST coloring -- I always forget that page exists...but at least it's unified in color scheme.
5) "desc" -- "BRILLIANT!":http://imgur.com/VkIANNz
~Mae
CotM Selection Committee
Just trying to help out.
- Kallak
Just trying to help out.
"504 Gateway Time-out | nginx/1.4.6 (Ubuntu)" errors all weekend. Very unreliable, and pretty much unusable as it is so touch and go.
B.
Rolling out later in the week just means your customers feel it the most, as they use it during the weekend, when you aren't at work.
-bort
Campaign of the Year - 2018
- Kallak
I posted an update in the "502 error thread":http://forums.obsidianportal.com/comments.php?DiscussionID=4065&page=2#Comment_35307 for those who had difficulty accessing the site over the peak Fri/Sat gaming period.
I don't want to shirk responsbility for the increased outage over this weekend, and we could have accelerated the hardware upgrade process.
After announcing the bug fixes/feature upgrades, our announcements seem to have viralized over social media or been posting to gaming groups, so that led to a higher-than-expected number of OP users, including new accounts, hitting the site over the weekend.
So instead of getting a better gaming experience, due to the fixes, many of you experienced 500/502 gateway errors instead.
I'll be meeting the tech team on Monday and working on improving our hardware by Wednesday, ahead of next weekend's heavy Fri/Sat gaming periods, so I am working towards minimizing this situations re-occurring.
As of now, Obsidian Portal is unusuable
Keelah Se'lai,
Keryth
"Shadows Over New York":http://www.obsidianportal.com/campaigns/shadows-over-new-york
"2013 Campaign of The Year":http://blog.obsidianportal.com/2013-coty-shadows-over-new-york/
"Campaign of the Month July 2013":http://blog.obsidianportal.com/shadows-over-new-york-julys-campaign-of-the-month/
Just trying to help out.
- Kallak
I had pushed for the hardware upgrade on Friday to try to get the performance boost ahead of the weekend, so I apologize for the disruption to your gaming sessions over the weekend.
We'll be pushing our upgrades on Monday/Tuesday so that there'll be a lower chance of outages hitting in the later part of the week.
Just trying to help out.
- Kallak
We're working on a fix.
We've replenished our email quota, though the emails are stuck in a "scheduled email queue".
The last time Alex worked with our developers on this, the moment it was cleared, OP's email system sent out like 50k emails in a matter of minutes.
Obviously, we don't want this to recur, so our techs are working on resolving this in the next couple of hours and are working on releasing the "scheduled emails" into the immediate send queue.
This means that email notifications will return to normal once this is fixed.
We're working towards getting this done by Tuesday morning.
Will post an update once this is done.
The techs and I have been testing campaign invites, wiki forum post notifications, update notifications, they seem to be working correctly.
There may be some bugs in specific OP accounts.
If you're still facing issues, post in the "support thread":http://help.obsidianportal.com/discussions/problems/19507-any-ideas-for-what-to-do-when-invites-dont-send and we'll have the tech look specifically at your account.
Note that they may send invites from your account, so please don't be alarmed if it looks like someone's been eating your porridge, sleeping in your bed, sending invites from your account. Cos they'll need to do that to troubleshoot your account.
OP functions on multiple servers, so we've upgraded our main server as we noticed it was getting close to 100% system resource utilization.
We're on a new, bigger server (we're on cloud hosting, so this is a virtual server carved out of our section of the cloud, not a physical server).
OP should be loading faster for most users, wikis editing/viewing should result in fewer 500/502 errors.
If you notice anything awry, please post a "support ticket":http:/help.obsidianportal.com
There was about 15minutes of downtime, at about 3am EST.
Having done 2 upgrades in the span of 10 days, we're getting better at smoothing out any potential issues.