Obsidian Portal - site update (Jan 2016)

245

Comments

  • Kallak
    Kallak
    Posts: 1,090
    that's a nice fix. I've run into that particular issue SEVERAL times
    All the best,
    - Kallak
  • Maesenko
    Maesenko
    Posts: 325
    Been away while the last few days of fixes/updates/features have gone into place. Quick summation below.

    1) Quick links -- "SQUEE!":http://tvtropes.org/pmwiki/pmwiki.php/Main/Squee
    2) Campaigns Page -- FINALLY!!! What appears to be the single most-requested item has been returned!!! (Not perfect, but working well enough, and will improve with time.)
    3) Navigation coloring -- Wasn't a huge problem with me, would prefer a partial reverse of colors (dark themes rule!) but it's definitely more legible on screens.
    4) DST coloring -- I always forget that page exists...but at least it's unified in color scheme.
    5) "desc" -- "BRILLIANT!":http://imgur.com/VkIANNz

    ~Mae

    CotM Selection Committee

  • Doc_the_grey
    Posts: 8
    Would love to look at them but I've been getting 504 and 502 errors for the last hour.
  • GamingMegaverse
    GamingMegaverse
    Posts: 2,998
    All morning, for the last 3 hours.

    Just trying to help out.

  • Kallak
    Kallak
    Posts: 1,090
    I've been getting them today trying to log in and view the forums on my phone. Took several tries to get past them.
    All the best,
    - Kallak
  • Lxcharon
    Lxcharon
    Posts: 189
    I haven't had any problems on the forums side but the OP site has been down for me all day.
  • GamingMegaverse
    GamingMegaverse
    Posts: 2,998
    Absolutely impossible to run a game- thank god I have a backup.

    Just trying to help out.

  • Glistam_
    Glistam_
    Posts: 11
    I was able to load pages after a few tries a couple hours ago, but now nothing is loading and I just keep getting "Bad Gateway" errors.
  • Khamul
    Khamul
    Posts: 3
    Nothing but 502!!! :(
  • Wynter
    Wynter
    Posts: 1
    Still getting 502/504 errors as well. Can maybe pull up a page once an hour. If I get lucky while trying it will continue to work for a few minutes before failure again.
  • LordGroan
    LordGroan
    Posts: 5
    I've been seeing:

    "504 Gateway Time-out | nginx/1.4.6 (Ubuntu)" errors all weekend. Very unreliable, and pretty much unusable as it is so touch and go.

    B.
  • Bortas
    Bortas
    Posts: 645
    The biggest issue I have with modern social software companies, is rollout dates. Rollout for new changes should always be on MONDAY, that way, you have all week to fix whatever you may have broke.

    Rolling out later in the week just means your customers feel it the most, as they use it during the weekend, when you aren't at work.

    -bort
  • Kallak
    Kallak
    Posts: 1,090
    While I agree completely that it sucks, at least andrew let us know in advance that they couldn't get things situated in advance and that the weekend/busy period was going to be impacted. Not ideal obviously, but at least I could plan accordingly.
    All the best,
    - Kallak
  • josh_little_5015
    josh_little_5015
    Posts: 2
    Our session last night was severely impacted by the 502/504 gateway errors. My group uses OP heavily, from GM prep and notes, in-character discussions both during and after sessions, maps, images, character sheets, etc. I ended up DMing our 6-hour game with one NPC page that would load and straight memory. One player didn't have a physical character sheet, so they also just winged it as best as possible. I love the flexibility OP has and both the functionality and the ability to use custom CSS really makes it valuable to me (Ascendant membership for the last year or so), but last night burned me pretty bad, so I will most likely be going back to good old paper for critical information, or at least as a draft so enough of the material exists elsewhere. Unfortunately, wasn't seeing the gateway errors until it was too late and we were already into the game.
  • andreww38
    andreww38
    Posts: 239
    Hi all,

    I posted an update in the "502 error thread":http://forums.obsidianportal.com/comments.php?DiscussionID=4065&page=2#Comment_35307 for those who had difficulty accessing the site over the peak Fri/Sat gaming period.

    I don't want to shirk responsbility for the increased outage over this weekend, and we could have accelerated the hardware upgrade process.

    After announcing the bug fixes/feature upgrades, our announcements seem to have viralized over social media or been posting to gaming groups, so that led to a higher-than-expected number of OP users, including new accounts, hitting the site over the weekend.

    So instead of getting a better gaming experience, due to the fixes, many of you experienced 500/502 gateway errors instead.

    I'll be meeting the tech team on Monday and working on improving our hardware by Wednesday, ahead of next weekend's heavy Fri/Sat gaming periods, so I am working towards minimizing this situations re-occurring.
  • Keryth987
    Keryth987
    Posts: 1,047
    Andreww, it is still happening. Not just Friday and Saturday. Sunday as well. Since Friday I am unable to access any of my sites to do anything. This is beyond frustrating and I hope it gets fixed tomorrow (Monday).

    As of now, Obsidian Portal is unusuable


    Keelah Se'lai,
    Keryth
    "Shadows Over New York":http://www.obsidianportal.com/campaigns/shadows-over-new-york
    "2013 Campaign of The Year":http://blog.obsidianportal.com/2013-coty-shadows-over-new-york/
    "Campaign of the Month July 2013":http://blog.obsidianportal.com/shadows-over-new-york-julys-campaign-of-the-month/

  • GamingMegaverse
    GamingMegaverse
    Posts: 2,998
    Now it is Sunday/Monday 12:30 am Pacific, 3:30 am Eastern, and 4:30pm Singapore time. Still not working, at or nearing end of work day Monday for the programmers.

    Just trying to help out.

  • andreww38
    andreww38
    Posts: 239
    Hi all, initial reports from the tech team is that during the hardware upgrade process, some of the links were broken within OP. They're working on restoring the site to full functionality now. Will post an update when it's done.
  • Kallak
    Kallak
    Posts: 1,090
    Thanks for the update (and apparently being up at 5:30 in the morning to do it)
    All the best,
    - Kallak
  • andreww38
    andreww38
    Posts: 239
    @kallak, I'm in Singapore, so I'm 13 hours ahead of EST. It's 7.54pm Monday here as I post this.
  • andreww38
    andreww38
    Posts: 239
    The techs have completed their diagnostic and repaired the links that were broken during the upgrade process. OP should be working as it should now, and you might experience a performance increase and fewer 500/502 errors when you're on the site, editing wikis.

    I had pushed for the hardware upgrade on Friday to try to get the performance boost ahead of the weekend, so I apologize for the disruption to your gaming sessions over the weekend.

    We'll be pushing our upgrades on Monday/Tuesday so that there'll be a lower chance of outages hitting in the later part of the week.
  • Kronos_DPC
    Kronos_DPC
    Posts: 2
    The site now works for be but since today no Update Emails are going out if I update a post or create a new one.
  • andreww38
    andreww38
    Posts: 239
    Thanks for the update, Kronos, I'll have the tech team check on this.
  • GamingMegaverse
    GamingMegaverse
    Posts: 2,998
    Same as Kronos- not a single email update being received by my active players or myself.

    Just trying to help out.

  • andreww38
    andreww38
    Posts: 239
    We exhausted our email quota sending out 400k+ emails to announce the server migration last weekend, we're in the process of replenishing it.
  • Kallak
    Kallak
    Posts: 1,090
    I assume notifications from posts on the forums counts toward this as well? I can turn mine off to help, I check daily anyway.
    All the best,
    - Kallak
  • andreww38
    andreww38
    Posts: 239
    @Kallak - it's ok, you don't have to turn anything off. We don't want any OP user to have a reduced user experience.
    We're working on a fix.
  • andreww38
    andreww38
    Posts: 239
    email notification update:

    We've replenished our email quota, though the emails are stuck in a "scheduled email queue".

    The last time Alex worked with our developers on this, the moment it was cleared, OP's email system sent out like 50k emails in a matter of minutes.

    Obviously, we don't want this to recur, so our techs are working on resolving this in the next couple of hours and are working on releasing the "scheduled emails" into the immediate send queue.

    This means that email notifications will return to normal once this is fixed.
    We're working towards getting this done by Tuesday morning.

    Will post an update once this is done.
  • andreww38
    andreww38
    Posts: 239
    Re: email notifications

    The techs and I have been testing campaign invites, wiki forum post notifications, update notifications, they seem to be working correctly.

    There may be some bugs in specific OP accounts.
    If you're still facing issues, post in the "support thread":http://help.obsidianportal.com/discussions/problems/19507-any-ideas-for-what-to-do-when-invites-dont-send and we'll have the tech look specifically at your account.

    Note that they may send invites from your account, so please don't be alarmed if it looks like someone's been eating your porridge, sleeping in your bed, sending invites from your account. Cos they'll need to do that to troubleshoot your account.
  • andreww38
    andreww38
    Posts: 239
    Server upgrade:

    OP functions on multiple servers, so we've upgraded our main server as we noticed it was getting close to 100% system resource utilization.

    We're on a new, bigger server (we're on cloud hosting, so this is a virtual server carved out of our section of the cloud, not a physical server).

    OP should be loading faster for most users, wikis editing/viewing should result in fewer 500/502 errors.

    If you notice anything awry, please post a "support ticket":http:/help.obsidianportal.com

    There was about 15minutes of downtime, at about 3am EST.

    Having done 2 upgrades in the span of 10 days, we're getting better at smoothing out any potential issues.
Sign In or Register to comment.

March 2024
Wrath of the Highborn

Read the feature post on the blog
Return to Obsidian Portal

Howdy, Stranger!

It looks like you're new here. If you want to get involved, click one of these buttons!

Discussions