Obsidian Portal API discussion

Micah
Micah
edited November 2010 in General Archive
First things first: "Read the blog post":http://blog.obsidianportal.com/?p=1249

This thread is a place to discuss the API that we are considering.

*NOTE* I don't want to quash participation, but I want to focus on serious developers and applications. I don't want this to become a "what if..." discussion about all the cool possibilities. So, let's try to keep scope creep and gold plating to a minimum. I'd rather have a single simple concrete application than a hundred theoretical awesome ones.
2

Comments

  • Micah
    Micah
    Posts: 894
    The first app that we know is coming is a simple initiative-tracking tool made by Vanir from Critical Hits. He asked me if there were a way that he could pull in all the PCs from a campaign. That means authentication, campaigns, and characters are currently at the top of my list.
  • giftedmunchkin
    Posts: 11
    I wonder if access to public campaigns is really needed? I suppose if it's easy for you guys to make, then I'm all for it, but no matter what the only uses I can think of for the API is as a reference at the table - for anything else you could just use the website - and therefore I'm not sure if any public stuff is needed, just access to your own campaigns. Others may disagree, though.
  • Curufea
    Curufea
    Posts: 161
    I'd like an API that I can query with a chron job to check for revisions of an OP campaign and be able to create an offsite mirror that includes revision history. A workaround for the currently lack or revision history. Which would also lead to having a customised interface to the OP content - enable the use of your own templates and page layouts/CSS
  • TALlama
    TALlama
    Posts: 3
    I'd strongly suggest you design any API to follow the Atom Publishing Protocol: http://www.ietf.org/rfc/rfc5023.txt It allows you to define arbitrary collections of entries (a wiki is a collection, an adventure log is another, the character list is yet another, etc), and then provides a lot of metadata (timestamps, authorship, etc) and wraps an arbitrary Content-Type. For most of what you're doing the content is gonna be the raw Textile, but for characters you could easily support dnd4e files (or whatever) in future revisions. When you start allowing write access you could let us tool-builders push the appropriate header and the body as raw HTML, preprocessed in whatever we want on our end (I have a Ruby script my adventure log entries pass through that highlights power names, links character and location names to the wiki, etc).

    The best part about this is that APP would allow lots of already-existing tools to talk to Obsidian Portal right out of the gate. My blogging tool could suddenly post Adventure Log posts, and so could a slew of others. Most languages have pre-exising bindings or libraries to talk to APP services, so those tools become easier to write and maintain. It would give you an easy way to test your implementation, too: just use some of the tools that are already out there.
  • Micah
    Micah
    Posts: 894
    I've looked at the APP and it honestly seems like a great spec that went nowhere. I love Atom as an extensible feed protocol, but the publishing side seems to have very little support. For example, I couldn't find a single desktop blog publishing tool that supported APP. They all seemed to use the proprietary (but simpler) protocols. I was looking for exactly the reason you mention: interoperability with existing tools.

    Most likely we'll do something very simple and RESTful. Right now I'm looking at the Twitter API as an example. I like their simple interface, and the ability to do everything with simple query string parameters makes it easy, although the OAuth stuff is a bit of a pain. Still haven't decided whether the additional security of OAuth is worth the additional complexity.
  • TALlama
    TALlama
    Posts: 3
    Windows Live Writer, MarsEdit, and Ecto all claim APP support. And it's kind of the poster child for RESTful interfaces; if that's the way you're going it seems like that'd be a natural fit. Now it's possible all those tools don't actually work
  • RobJustice
    RobJustice
    Posts: 178
    This is the kind of thing that I would never plan for as a developer but if the tools were presented to me I would work with. I currently have no designs to make an application but if the appropriate APIs were provided I would seriously start considering it. I'm not sure if that gives me liscense to throw in my two cents or not... but I'm going to anyway.

    The first thing I would do with the proper APIs is develop a lightweight app for Android that lets me pull wiki and character pages from my campaign. A faster alternative to using the browser to pull up the site. I'd also include functions to download and track updates via my Android and push notifications whenever a page\character is changed or created. Odds are good the only pages\characters I'd be interested in would be my personal ones and anything I've favorited. While I can see an advantage to being able to pull from the entire public listing I just don't think I would want to often enough.

    Something to keep in mind is Ascendants and general Memberships. I'd keep API usage restricted to Members only, no totally open calls. I'd also limit the majority of the API usage, particularly any future write ability, to Ascendants. Messing with APIs, especially if you have write access, can seriously mess up data and I'd limit my pool of possible complainers to the people who are willing to pay a couple bucks for it.

    I guess, in the end, the calls I'm most interested in are restricted to the campaigns and characters of myself and my players. That's my two cents.
  • Micah
    Micah
    Posts: 894
    Rob,

    The use cases you discuss are the ones we envision as well. I expect most people care about their campaigns and theirs alone. They want quicker access, like you say, or a better mobile experience. That's why we're going to focus on campaigns, wiki pages, and characters to start. With that data alone, someone could make a pretty nice application.

    The majority of the API will be restricted to logged-in access. In other words, the application will need to log in as you (the user) in order to access the majority of the data. Right now, we are looking at using a standard called "OAuth":http://oauth.net/ that makes it easy for users to authorize applications to access their data. It's very similar to how Twitter, Flickr, and Netflix applications work. The application would send you to Obsidian Portal, you'd click a button saying that you agree to give access, and then the app just magically works. At any time you can revoke access to any application and the others will continue working. It's more secure and flexible than giving your username/password to a bunch of different applications. On the downside, it's more complicated for developers, but there are a lot of libraries out there to do most of the heavy lifting.

    As for Ascendant vs Regular members, we're considering things here, but it's doubtful that we'll segregate much along those lines. We really don't want to discourage developers from integrating with Obsidian Portal, and if they know their tool or app will only work with a subset of our users, they might decide it's not worth the trouble.
  • Micah
    Micah
    Posts: 894 edited November 2010
    I've started putting together "a help/tutorial section for the API":http://help.obsidianportal.com/faqs/api There's not much there right now, but the "Users API page":http://help.obsidianportal.com/faqs/api/api-users has some meat to it.

    I would like some advice/suggestions about what should be public and private. Others are more keyed-in to privacy concerns than me, and I'm curious to know what people think should be public or private. Obviously, email address is private, private campaigns are private, and gm-only info as well. But what about stuff that's easily accessible via the website? For example, should anyone be able to retrieve a list of the campaigns you are in, including the private ones? Anyone can learn this now from your profile page pretty easily. Seems harmless to me, but I'd like to know others' thoughts.
    Post edited by Micah on
  • Micah
    Micah
    Posts: 894
    I've just deployed the pre-pre-pre-alpha version of the API. There is support for getting some basic user, campaign, and character info. You can read more here: "http://help.obsidianportal.com/faqs/api/api-overview":http://help.obsidianportal.com/faqs/api/api-overview

    We'll be making some changes to some of the routes and URLs in the near future, so consider this all very unstable at this point. However, if you have an idea you're itching to try out (and you'll actually do it, not just talk about it...), let me know what you need. This is moving much faster than I had anticipated, and I'm excited to get people on board.
  • shammond42
    shammond42
    Posts: 65
    OK, I've got a Rails app running on localhost using the OAuth gem. It seemed to go OK. I am very experience with Rails, but this was my first time with OAuth and it took me about 30 minutes to be able to pull over my user record.

    For my first usage of the API, I'm interested in visualizing some information about the popularity of different game systems, possibly showing how it has changed over time. Ideally there would be a game systems api, I could call for this. Alternatively I would be interested in getting at public campaigns and pulling the data out of there.

    I realize there is probably a limited use for this sort of app, but it should make a quick and easy example.
  • Micah
    Micah
    Posts: 894
    shammond42,

    If you're not already using it, I highly recommend checking out the "ruby oauth gem":https://rubygems.org/gems/oauth, and possibly the "Rails oauth-plugin gem":https://rubygems.org/gems/oauth-plugin That's what we used to set up our OAuth provider, and it really does a lot of the heavy lifting. I imagine you already know this, as it would probably take a lot longer than 30 minutes to do it without the OAuth gems to handle all the token generation and signing.

    Ultimately, my hope is to make a gem that sits on top of the oauth gem and provides a nice ruby interface, somewhat like a lot of the other api gems do. That's a ways off, though, unfortunately. Alternately, we may offer a prize or reward to people who make libraries in different languages. We'd need a stable API for that, though.

    Search campaigns by game system is a good idea. I'll put that down as a TODO. General searching is probably a ways off, but we can try and figure out something in the short term. It's these kinds of requests that help us figure out what's really required.

    Thanks for trying it out! You're the first!
  • shammond42
    shammond42
    Posts: 65
    I'm using the OAuth gem. I was thinking ab out a ruby wrapper like you were describing, we'll see what happens...

    Even simpler than search within a game system, would be something like this.

    http://api.obsidianportal.com/v1/game_systems?order_by=popularity

    or

    http://api.obsidianportal.com/v1/game_systems/show.json?system_name=pathfinder

    The second would return something like this, and the first would return an array of these...

    {id: 1, name: Pathfinder, publisher: Paizo, camapign_counts: [
    total: 100,
    in_planning: 25,
    playing: 25,
    on_hiatus: 25,
    completed: 25
    ]}

    I don't know exactly what data you have on particular game systems, but you get the idea. I'd like to build an app that pulled that data over daily and did some visualizations on it.
  • shammond42
    shammond42
    Posts: 65
    As far a privacy goes, I feel that anything I can find on the site, I should be able to get through the API. And anything I'm sharing publically, I am willing to have exposed through the API.
  • Micah
    Micah
    Posts: 894
    A quick update after some discussion:

    1) We're probably going to be changing some of the URLs to be a little more Rails-y resource routes.

    2) We're going to switch from internal database ids to UUIDs for all the objects. This will mega-guarantee that things are unique, plus it avoids any kind of Twitter-API-style "oh crap" moments where they run out of integer IDs for things. Also, it means that everything in the system will have a unique id, which can be helpful in certain cases. Note: This means ids for objects should be stored as >= 32 character strings.

    3) We're going to change all the lookups to use IDs instead of id-or-slug. If you want to look something up by slug, you'll have to pass a parameter ("lookup_by_slug=true" or something like that). This will make things much easier on our end, without adding much difficulty for API clients.

    These are the kinds of random shakeups we expect to happen in the next couple weeks. We want to play with things a bit and see what we like before we get things set in stone.
  • TALlama
    TALlama
    Posts: 3
    Instead of lookup_by_slug why not just pivot on if the client is passing an id param or a slug param?

    I agree with the public-on-the-site-is-public-to-the-api discussion.

    And I'm hoping to start hacking on some of this in the next few days. Excited!
  • Micah
    Micah
    Posts: 894
    We made the updates to the API, and updated the documentation. The URLs are more Rails-y now, and I really like the UUIDs over the raw database IDs.

    Now to see about adding some more functionality to it.
  • Micah
    Micah
    Posts: 894
    We updated the OAuth workflow URLs as well. They are now served via SSL and from the 'www' subdomain. More here: "http://help.obsidianportal.com/faqs/api/api-authentication-oauth":http://help.obsidianportal.com/faqs/api/api-authentication-oauth

    This was to avoid some SSL/cookie issues that occurred when someone tried to create their tokens.
  • shammond42
    shammond42
    Posts: 65
    I'm working on a ruby gem this morning. Its coming along, I hope to have code on github to share by the end of the weekend.

    How do we edit our application parameters on Obsidian Portal? I think I accidentally signed up for two, so I would like to delete one, and I would like to edit the redirect link of the other.

    Thanks,
    Steve
  • shammond42
    shammond42
    Posts: 65
    Never mind I found it. Go to /oauth_clients
  • shammond42
    shammond42
    Posts: 65
    A couple of minor tweaks...

    When I go to edit application the submit button is labeled "edit".

    On the page to authorize an app to access my account, it would be nice if the label for the checkbox was connected to the checkbox so that I have a bigger target to click. You might be able to simplify this further by making two submit buttons. "Authorize" and "Don't Authorize" then the user can get off that page with a single click.

    Thanks
  • Micah
    Micah
    Posts: 894
    Sorry, just saw these questions...

    We need to give some attention to all the OAuth pages, from client application signup to user authorization to client app editing. The ones you see here are pretty much the raw templates created by the oauth-plugin gem that we used to get up and running. They're functional, but just barely. I'll see if I can work on them soon.

    As to writing the gem, that's awesome! I have to say over and over that we're still juggling things and trying various ways of structuring the data, so please don't get too angry if we rename fields or move them around.

    That being said, we're going to offer a *lifetime Ascendant membership* to anyone who writes and agrees to maintain an open source API library in any language. Only the first such library in a language is eligible (ie. the first rubygem, PHP library, .NET library, etc), assuming the first one is any good. I'll probably sit down and write some guidelines, but the basic idea is that we really want to lower the barrier of entry to the API. If you all can help us do that, a lifetime membership is the least we can do.
  • shammond42
    shammond42
    Posts: 65
    mage-hand 0.1.1 is now available.

    mage-hand is a ghostly hand that reaches across the internet to access the Obsidian Portal API. At the moment, mage-had only supports the authentication part of the api and only works with Rails Apps. Support for all the rest is coming soon.

    gem install 'mage-hand'

    or check out the code here.

    https://github.com/shammond42/mage-hand

    Steve
  • shammond42
    shammond42
    Posts: 65
    I've got two big issues right now. I'm not sure if they are on my end or yours, any thoughts are appreciated.

    1. A new browser session requires a new authorization. I've had as many as 10 outstanding tokens issued to my app. I can revoke them manually. I may be doing something wrong in requesting them, but I think that what I really need is an option to "rember" authorizing the app and an expiration on the tokens for apps that aren't "remembered". There may already be an expiration and I just haven't waited long enough.

    2. The different url for authentication vs. the rest of the API is painful. I have to specify the full URL for all requests to the API. I want to go through the OAuth gem to see what options I have for managing that on the client, but it would be nice to be able to use api.obsidianportal.com for authentication or www.obsidianportal.com for the api.

    Thanks!
    Steve
  • Micah
    Micah
    Posts: 894 edited November 2010
    Steve,

    I love the name. A nice melding of the Ruby community "name it something crazy" philosophy, aligned with the subject matter. I love it.

    Let me answer your questions as best I can:

    1) I'm not sure I understand here, but it sounds like you're not persisting the access tokens that come back. For a single user, you should be able to go through the authorization workflow, ending up with an AccessToken and its secret (for a particular user). If you save this access token and secret to the database (or wherever), you should then be able to make as many calls as you want on behalf of that user. They only need to do the OAuth workflow once, then you have everything you need forever. The tokens never expire (for now), but the user can revoke them.

    I'm pretty sure the 4 things needed to make an API call are: consumer key, consumer secret, access token, access secret. At the end of the request_token call, you should have all these things.

    Did I miss the point of the question?

    2) I agree that this is a pain, but it will have to stay, for now at least. It comes down to 4 things:
    i) We really want the API on its own subdomain
    ii) We really want the OAuth access workflow to be SSL. This is "required" by the spec, and we try to be privacy conscious when we can
    iii) Without dropping $1000, we can't have an SSL certificate that covers all our subdomains, and we can't have 2 SSL certificates on the same machine (technically it's possible, but it's complicated)
    iv) User session cookies created on the api subdomain aren't valid for the www subdomain, meaning that even if we could sort out the SSL stuff, the authorize/login step would still be complicated.

    Thanks to SSL and cookies, our hands are tied on this one. However, before making this decision, I dug into the OAuth gem a little, and I found that it's not so bad. It's not well documented, but there's a way to specify the raw OAuth URLs separate from the main consumer host, like so:

    @consumer = OAuth::Consumer.new(key, secret, {
    :site => "http://api.obsidianportal.com",
    :scheme => :header,
    :http_method => :post,
    :request_token_url => "https://www.obsidianportal.com/oauth/request_token",
    :access_token_url => "https://www.obsidianportal.com/oauth/access_token",
    :authorize_url => "https://www.obsidianportal.com/oauth/authorize"
    })@

    You just specify "XXX_url" instead of "XXX_path" for the OAuth endpoints. I haven't tested this but scanning the code it looks like it should work. I just hope that OAuth libraries in other languages are similarly flexible. After this, all further API request can just be routed to "/v1/users/XXX" and such.
    Post edited by Micah on
  • shammond42
    shammond42
    Posts: 65
    Ahh, I found the documentation for :authorize_path, but I didn't find :authorize_url. I'll give that a try tonight. I'm going to have to think about the persistance issue. I was thinking that for simple apps, I might not even need a user record in my app's database, but maybe it won't work that way.
  • Micah
    Micah
    Posts: 894
    authorize_url isn't really documented anywhere. I had to dig through the code. Fair warning: in the code "authorize_url" is marked with a comment that says "TODO: This is ugly, fix it somehow" so maybe they don't plan to support it going forward. I don't know why not, though, as there's nothing in the OAuth spec that says the workflow specs have to be on the same domain/host as the rest of the API.

    As to persistence, there's really no way to get around it, I don't think, unless you want the user to reauthorize each time. Persistence makes sense for apps where users are using the app to modify their campaigns (like mobile apps). But, it makes less sense for some tools that are less user centric, like your idea for a stats visualizer for the various game systems.

    There's an authentication method in OAuth called "2 legged authentication" where API calls can be made only on behalf of a consumer (app), not necessarily a particular user (provider + consumer + user = 3 legged). Maybe someday we'll do 2 legged authentication, and split out some API methods for use without needing a logged in user (stuff like overall search, or basic campaign stats), but for now that's out of scope.
  • Micah
    Micah
    Posts: 894
    An update: We're changing how errors are returned, so it can support returning multiple errors at once. Essentially, we want to be able to handle cases where some inputs are invalid ('name is too short' or 'email address invalid') and return a list of validation errors instead of just saying "something went wrong"

    In addition, we're going to add specific error codes to some of the more common errors (like invalid parameters) to help app developers catch and handle these recoverable errors.

    Expect a fairly sizable update tonight, including wiki pages as a resource. I've also started diving into the rest of the CRUD functionality for wiki pages. I have get/update now. I hope to have create/delete by tonight.

    As an aside, I'm a little worried about exposing delete functionality. For now, we don't have an undelete ability, and I'm worried about buggy apps (or straight-up malicious ones) irrecoverably deleting people's wiki pages (or characters, campaigns, etc). My current line of thought is to add an intermediate "trash bin" stage where the deleted pages go before being eventually purged. Still, that's not trivial by any means. Anyone have thoughts on this?
  • Micah
    Micah
    Posts: 894
    Pushed the update live. I'll try to document the error responses sometime soon.

    For now, check out "the Wiki Page documentation":http://help.obsidianportal.com/faqs/api/api-wiki-pages It's pretty standard CRUD stuff, which I hope to stick to as much as possible.
  • Micah
    Micah
    Posts: 894
    I created a "test project":https://github.com/micahwedemeyer/obsidianportal_tag_renamer today on github. It actually might come in handy in the near future, as we're going to switch tags to being case-sensitive. People may want to rename a bunch of tags to use the casing they prefer.
Sign In or Register to comment.