Server maintenance / OP going offline...for a bit.

andreww38
andreww38
edited January 2016 in General Archive
Hi all,

Take note that we will be doing server maintenance and stabilization work on our server on Monday, 2am to 4am PST.
In a best case scenario, we might have the site up within 30 minutes, or it might take 2 hours.

If we discover any critical issues or catastrophic errors, we'll do a rollback to this code, which is fairly stable and re-schedule another maintenance session.

This server work will help ensure that as we apply the fixes, that the server will be more stable, especially if we might be rewriting fairly large chunks of OP's code in the coming weeks.

-
*What do you need to do in preparation for the brief outage?*

1) Continue using the site as you would normally do so, up until 15 minutes before Monday, 2am PST.
2) Exit your campaign, save all changes that you've made.
3) Once the server is back up again, you can continue working on your campaigns (not too many people working on the site on Monday, 4am, as far as we can tell...)
4) Any further updates will be posted in this thread.

-

While this site is down, we'll post updates on our Twitter account and Facebook page.

If you need to reach me, you can always email me at [email protected]
I will be watching email as I can while I'm working with the tech team during the server maintenance. (More focus on making sure the tech stuff flows smoothly).

*Questions?*
Post in this thread, I'll watch and answer as best I can till about half an hour before the maintenance.
Else, email me at [email protected]
«1

Comments

  • andreww38
    andreww38
    Posts: 239
    OP Facebook page:
    https://www.facebook.com/obsidianportal

    OP Twitter page:
    https://twitter.com/obsidianportal

    My email:
    [email protected]

    Please copy-paste the above info into a google doc, text file, email it to yourself, etc, cos the site will be down, so you won't be able to access this page to get the links...

    Apologies for stating the obvious, but I have received emails from users in other projects along the lines of "I couldn't access the web page during the server maintenance. What's up?"

    Me: ...
  • andreww38
    andreww38
    Posts: 239
    FYI, the site will go down about 6 hours from now.

    At about 145am PST (445am EST) please save your work.
    You can stay logged on to the site, though you might encounter a Error 404 or Error 500, while the site is taken off for upgrading.

    You are welcome to PM, email me on the FB page, Twitter account, email (see message above) if you have any questions.

    There is the potential I might be hit with a lot of PMs, emails at that time, while I am trying to manage the tech team too, so apologies for the delay in response.

    Watching for the wall updates on the FB group and the tweets might be the best way to go.

    Else, we'll see you once we have a shiny, more stable OP up and we'll be rolling out the fixes shortly after.
  • andreww38
    andreww38
    Posts: 239
    Welcome back! For those of you who might have been counting, we had a few seconds of downtime, and the tech team spent about an hour checking that everything was working correctly.

    The server is now prepped to take on a series of upgrades, features, etc.

    2 updates in place now:
    1) 308 campaign URLs which contain underscores have been rewritten, so they won't generate 404/500 errors. They will however have a new campaign URL, so you will need to get that out to your players, or potential players.

    Example: return_of_the_jedi.obsidianportal.com will not appear as return-of-the-jedi.obsidianportal.com in your interface.

    If you are facing any problems, please post a "customer support ticket":http://help.obsidianportal.com

    ---
    2) File Storage Locker: storage used / storage remaining is now in.
    It's accessible from your campaign page via "Settings", "Basic Settings" and appears below "Campaign URL"

    We're still in the process of rebuilding/improving the file storage locker, so we'll announce more improvements as we test and push them to the server.

    ---
    Questions?
    Post them below.
  • SkidAce
    SkidAce
    Posts: 807
    File Storage! Something we asked for!!

    Thank you.
  • andreww38
    andreww38
    Posts: 239
    For those who're wondering, the issue of problematic image uploading is due to some bugs in the jQuery code.

    There will be some effort and doing involved in debugging and getting it to work properly and this means almost all file types are affected.

    We are also aware about the problems with youtube embedding, so that is also being looked at.

    The image upload is an easier task, so the tech guys are working on that first. (YouTube embedding might be a different beast, so we'll take care of that, after we deal with the whitewalkers.) So I may have an update re: image uploading later this week.

    Note: I'll give an update, not necessarily a fix, depending on how much code they need to fix/re-write/re-code. So please, hold off on the walking naked through the streets of King's Landing with someone ringing a bell and going "shame, shame, shame" for now.

    ----
    PS: There are plans to detach file storage locker so it'll be it's own discrete component within your campaign panel, so you won't need to go into the wiki editor, then access it that way. I've also got feedback from Wolfhound that having a file listing will be useful, together with the ability to delete those files. This is also going to take some doing and advanced planning, so that is something we're planning, plotting and gathering the dragonglass for now...


    Thanks for listen, er, reading..
  • Maesenko
    Maesenko
    Posts: 325 edited January 2016
    Thank you for the beautiful update!

    EDIT: I agree with Wolfhound, a list of what items comprise the File Storage Locker and the ability to manage/delete those files would be an amazingly nice thing to have, but I imagine that will be a ways down the road (but hopefully something we will see this year?).
    Post edited by Maesenko on

    ~Mae

    CotM Selection Committee

  • killervp
    killervp
    Posts: 2,735
    Very impressed with the update, and shocked that I am less than 1 GB.

    Just trying to help out.

  • Keryth987
    Keryth987
    Posts: 786
    I am as well, which surprised me. However, I am wondering if the amount shown on each campaign is displaying how much is available for that campaign or for my overall account. Because if it shows only how much of my total is used by that campaign, it would be much more useful to get a total usage for my account then.


    Keelah Se'lai,
    Keryth
    "Shadows Over New York":http://www.obsidianportal.com/campaigns/shadows-over-new-york
    "2013 Campaign of The Year":http://blog.obsidianportal.com/2013-coty-shadows-over-new-york/
    "Campaign of the Month July 2013":http://blog.obsidianportal.com/shadows-over-new-york-julys-campaign-of-the-month/
  • killervp
    killervp
    Posts: 2,735 edited January 2016
    Crap, if it is showing what is available I am in trouble...
    killervp
    "A God...Rebuilt":https://a-god-rebuilt.obsidianportal.com
    Post edited by killervp on

    Just trying to help out.

  • andreww38
    andreww38
    Posts: 239
    A far as I'm aware, it shows a global total, cos segmenting your file storage by campaign, then revising that number each time a new campaign is added, deleted will mean that your stats will have a great deal of variation.

    To be honest, the code you are seeing now is what was there when Kaleidoscope acquired it. Why it wasn't included could be an interesting topic for philosophical debate, but my raison d'etre is to make sure stuff works.

    -
    Re: the file listing --- that part isn't terribly hard to code and put in.
    The issue is that if you have existing wikis that make reference to an object/media element that was deleted, then it's a broken/404 error, the wiki fails to load, then OP users will be hiring faceless men to come after me again.

    So we are trying to exercise a little initiative (roll d12...pass!), and plan a few steps ahead, ideally not putting in features which cause the site to crash (has been done before...but not on my watch).

    We also test every feature and bug fix on our test server before we put it here, though there is a difference in server resource load when 5 beta testers are trying it on the test server and when 5,000 users are accessing the same feature, at the same time, on this server.

    Eh, so that's what's happening behind the curtain.
  • Maesenko
    Maesenko
    Posts: 325 edited January 2016
    Andrew, what game do you play which uses a d12 for initiative? Seriously, I'm curious... What game uses a d12 for Initiative?

    I went back and reread the KS section regarding the File Storage Locker, it appears that the wording is stating the total storage granted is relative to the campaign, with Free users getting 250MB of space and Ascendant users getting a whopping 10GB, of which I currently see a total of 5GB. That's a lot of storage space for an entire account, let alone a single campaign. Now granted, most people only have one active campaign (at least as a GM) on the site, so that doesn't seem like much of an issue, and this is even with the rose-tinted glasses of nostalgia and wishful thinking that was going on during that KS. But realistically, I think it's more feasible for it to be per Account, rather than per Campaign, as there are those of us who could end up being data hogs, especially since there's no limit to the amount of campaigns an Ascendant can have.

    On to specifics:
    Now, as it stands I show a 5GB total on each of my campaign pages, and only the ones with images stored on site have a data used total (the others display 0 bytes used).
    What I don't know is if I am being allocated a separate 5GB of space for each campaign or if that 5GB is across all campaigns.

    Of the campaigns I have, only two of them are currently storing any data, one is only a couple KB, and the other a couple MB.
    What I don't know is if those totals are a combined portion of a total 5GB, or if they each tally into a separate 5GB. (Going along with the above question.)

    And none of this addresses the potential additional 5GB (10GB total) that we are supposed to have according to the KS.
    I'm not trying to gouge more out, if the conditions are unreasonable I understand, as it's already a lot of room without it doubling for each campaign of each Ascendant.
    But I would like to know where that stands as well. Are we getting 10GB as would fit with whatever context above, or will a line be drawn and a "no" be decreed?

    And lastly, what are the currently supported file formats, and how can we access that list and (eventually) request additional format support?

    I'm sure I'm forgetting something, but I'll worry about that later. For now, I anxiously await your reply.

    As Always,
    _~Mae_
    Post edited by Maesenko on

    ~Mae

    CotM Selection Committee

  • killervp
    killervp
    Posts: 2,735
    Awesome points and questions Mae!
    I would like to add one- since this is literally now almost a year ago as far as " the code you are seeing now is what was there when Kaleidoscope acquired it..." does that mean it updates in real time?
    killervp
    "A God...Rebuilt":https://a-god-rebuilt.obsidianportal.com

    Just trying to help out.

  • andreww38
    andreww38
    Posts: 239
    @ Mae -

    The d12 for intiative is for the game I made up in my head.

    -
    Looking at the file storage locker at the "OP reforged page":https://www.kickstarter.com/projects/252020431/obsidian-portal-reforged/description and looking at the block of text:
    "if we can hit this stretch goal, we will be able to increase the storage size limits for both free and Ascendant campaigns. The free limit will climb to 250MB and the Ascendant limit will rise to 10GB."

    I believe Micah originally meant the file limits apply to accounts, not campaigns, because it would be a nightmare in resource provisioning to add another 250mb per campaign created and another 10gb per campaign created in an ascendant account.

    At present, the file storage limits are coded on a per account basis. I think someone fraked up on the verbiage
  • andreww38
    andreww38
    Posts: 239
    @ killervp -

    Er, at the risk of sounding defensive, the current OP code that is on the server is now (bugs and all) are the state of tech development at the time kaleidoscope purchased it, so as some have pointed out in the KS updates, the stretch goals were not delivered on, or were not developed in a form that could be installed on the server.

    In some cases, the tech team is fixing buggy code, and in instances where the feature did not exist (like displaying the file storage space used/quota), we are building it up from scratch.

    ---
    Updates are written on the tech team's machines, they have a programming environment set up for OP code, once they write the code, it is put on the test site, and the testers (inner council and myself) try to break it or make it not work or report any weird behavior. The bugs are then fixed, and retested. Once it's ok, we put it on the live server (this one).

    With hundreds, or thousands (on a fri/sat heavy gaming session night), the feature may break again, due to scalability isuses, we then have the tech guys work on an improved fix, then we push it out again.

    er: TL;DR answer: updates are not live, though we try to develop, test them as swiftly as we can before putting them on here.
  • andreww38
    andreww38
    Posts: 239
    Re: The file storage quotas, and I think killervp said his resource bar said he was close to maxing his out.

    It shows "5gb" for my ascendant account too, so it looks like the 10gb quota isn't provisioned in.

    I'll bring this up in the next management meeting and see how we can deliver on this.

    I believe 5gb will probably be sufficient for most GMs, and 10% of power user GMs like killvervp might need the full 10gb.

    -

    I'm wondering if you guys will be ok if we have 5gb as the default file storage available, and we can do an free upsize to 10gb when needed?

    Else if we have 90% of ascendant accounts only using 500mb to 1gb, and the balance of the 5gb is not used, then part of your ascendant membership will be used for server resources that aren't used. (eg: we'll be provisioning/paying for 1000s of pentabytes, exabytes of storage that are lying dormant?)
  • andreww38
    andreww38
    Posts: 239
    @ keryth - that's a great suggestion.

    I believe the system counts each campaign individually now and counts/deducts it against the total storage available for your account.

    I'll see if we can refine it better to aggregate a total storage used figure and display that instead.
  • Lxcharon
    Lxcharon
    Posts: 189
    I have a question, if the problems seem to come from heavy server load (and I know this is an expensive solution) but wouldn't upgrading to a larger server fix that? I mean it would have to be upgraded at some point down the line if OP starts fixing its problems (which you have been doing) and people start coming back/start using OP then the problems we have with the handful of still using members will be exponentially increased when there are more users. Right?
  • andreww38
    andreww38
    Posts: 239
    @ Lxcharon - we do increase our server resources as we see the load increase, we upgraded our database a few weeks ago and improved some of OP's code so 500/502 errors would occur less frequently.

    We don't have an issue with increasing CPU cycles (we're on a cloud host) for when the barrage of fri/sat night gaming sessions (and traffic) hits, we like to keep it at a conservative 70-80% of system resources used.

    With the disk storage issue, we're buying/renting pentabytes of disk storage, which might not be used. In suggesting a 5gb standard and free upsize to 10gb, I'm suggesting we can more efficiently use resources to keep "disk storage" costs at the same level and use the difference to buy more CPU cycles instead.

    It's not an issue of cutting costs (I have not seen this happen), it's more of cutting down disk storage expenses (unused) if we can, and funnelling it towards CPU, Ram, which are more directly used for the Fri/Sat gaming sessions.

    We're on a cloud server, so we can play around with ram, cpu, disk storage settings/allocations easily, compared to a fixed physical server.

    Do ask more questions if my "data dump" was vague.....
  • Lxcharon
    Lxcharon
    Posts: 189
    No,no, I didn't know you ran on cloud storage. That makes sense then. I was trying my hand at trying to help as an outsider looking in. Thanks for the quick responses.
  • killervp
    killervp
    Posts: 2,735
    Sorry about the confusion on my statement. What I was referring to is the amount left/used as far as storage. Will that update in real time? I upload an image, go back to the settings screen, and see a change.
    Never meant it the way taken-my bad.

    Just trying to help out.

  • killervp
    killervp
    Posts: 2,735
    Sorry about the confusion on my statement. What I was referring to is the amount left/used as far as storage. Will that update in real time? I upload an image, go back to the settings screen, and see a change.
    Never meant it the way taken-my bad.

    Just trying to help out.

  • andreww38
    andreww38
    Posts: 239
    I believe the file storage updates when you do a physical page refresh (eg: hitting the F5 key).

    Real-time updates will be possible if we're using ajax or flash, but I don't think it's the case with RoR which may work more like html.

    bottomline: you need to hit F5 or surf away then surf back to look at the disk storage update.
  • twiggyleaf
    twiggyleaf
    Posts: 1,601 edited January 2016
    I'm happy with current 5 GB, if there is some provision to offer more when things start going above 4 GB. Presently my usage seems to be 73 MB (Gosh, is that all?) for Mysteria and 12.3 MB (Almost unbelievably low) for Shimring, and I thought they were both massive sites...I also have two campaigns run by fellow players operating as co-GMs, one at 11.7 MB and another at 193 KB; so if all those figures are correct then I can probably design away for ages before having to worry.

    twigs
    Post edited by twiggyleaf on

    "I met a traveller from an antique land....."

    CotM May 2016: Mysteria: set in Wolfgang Baur’s MIDGARD.

    Previous CotM Aug 2012: Shimring: High Level Multiplanar Campaign

    Inner Council Member

  • Krothos
    Krothos
    Posts: 184
    That is some SERIOUS compression on those drives! LOL
  • killervp
    killervp
    Posts: 2,735
    You are telling me Krothos. This is what I see on my page

    File Storage Locker
    Storage Used: 371 MB | Storage available 5 GB (100%)

    Andrew had mine actually checked, and it was actually 4.2 GB, so obviously not calculating correctly.
    killervp
    "A God...Rebuilt":https://a-god-rebuilt.obsidianportal.com
    "OP's COTM April 2012":http://blog.obsidianportal.com/a-god-rebuilt-aprils-cotm/
    Over 350 Fans, and Looking for More!

    Just trying to help out.

  • SkidAce
    SkidAce
    Posts: 807
    Weird. I added a 1 meg file and my Storage Used went up by 1 meg. So that part works.

    But like killervp I suspect my actual used is higher.
  • Basileus
    Basileus
    Posts: 589
    Speaking with absolutely no idea of how it was implemented, considering the tiny numbers I've seen so far, I wonder if its only aggregating storage *since* the featrue was applied (Monday)?
  • NikMak
    NikMak
    Posts: 379
    Im happy with the 5GB to start with a free 10GB upgrade when needed.
  • andreww38
    andreww38
    Posts: 239 edited January 2016
    FYI, there is some bugginess in the way the file storage locker quotas are being calculated.

    And these are pre-kickstarter quotas that are being displayed, eg: 2mb for free accounts, 5gb for ascendant accounts.
    We had the techs doing some digging in the source code.

    -
    The way it's implemented, what you're seeing is the amount of storage used for each campaign, rather than an aggregated amount for all campaigns.

    To have it working the way it should, and in a way that will be useful for you, the users, we'll need to recode chunks of the file storage locker code, so that you don't see something like "Storage used: 500kb | Storage remaining 10mb (0%)" because 500kb is for the current campaign, but 10mb is what's left of your 5gb account (eg: ascendant).

    In an ideal situation (that we're working towards), I am seeing your feedback that you would prefer (total storage used) | (total storage remaining), rather than
    (storage used for this compaign) | (total storage remaining)

    Yes?
    -----
    And.....we'll find some way to figure out how to do something like allocate 5gb storage for ascendant accounts, then have an unlock (auto or manually triggered) to 10gb when its needed for the 10% of power users like killvervp who might be able to use the entire 10gb.

    I'll be specing out a fix for the tech team to recode the file storage code once I figure out a way to make this work.
    Post edited by andreww38 on
  • Basileus
    Basileus
    Posts: 589 edited January 2016
    "total used / total available" makes the most sense intuitively when viewed without any context or explanation. When I look at it, that's what I assume is meant.

    If its going to be anything else, that's probably fine, just needs the notation changed to explain how the numbers are related to the individual campaigns instead.
    Post edited by Basileus on
Sign In or Register to comment.