Help porting Loomio to Sandstorm

Hm, yes we're doing semi-weekly releases, so we're actually up to 0.17.0 now (you can note the current version # on production - as of this writing 0.17.8). However, the changes aren't so massively severe as to cripple it, especially given that you're likely not dealing with the angular frontend that much.

Looking at that 2.1.2 tag is making my head hurt as well; I'm not sure why it's there.

I'm working on upgrading my port to 0.17.0. Unfortunately, it refuses to load any static assets with the following errors:

Started GET "/js/app.js?1447692365" for 127.0.0.1 at 2015-11-16 16:46:05 +0000

ActionController::RoutingError (No route matches [GET] "/js/app.js"):
  actionpack (4.2.3) lib/action_dispatch/middleware/debug_exceptions.rb:21:in `call'
  actionpack (4.2.3) lib/action_dispatch/middleware/show_exceptions.rb:30:in `call'
  railties (4.2.3) lib/rails/rack/logger.rb:38:in `call_app'
  railties (4.2.3) lib/rails/rack/logger.rb:20:in `block in call'
  activesupport (4.2.3) lib/active_support/tagged_logging.rb:68:in `block in tagged'
  activesupport (4.2.3) lib/active_support/tagged_logging.rb:26:in `tagged'
  activesupport (4.2.3) lib/active_support/tagged_logging.rb:68:in `tagged'
  railties (4.2.3) lib/rails/rack/logger.rb:20:in `call'
  ahoy_matey (1.1.1) lib/ahoy/engine.rb:20:in `call_with_quiet_ahoy'
  request_store (1.1.0) lib/request_store/middleware.rb:8:in `call'
  actionpack (4.2.3) lib/action_dispatch/middleware/request_id.rb:21:in `call'
  rack (1.6.4) lib/rack/methodoverride.rb:22:in `call'
  rack (1.6.4) lib/rack/runtime.rb:18:in `call'
  activesupport (4.2.3) lib/active_support/cache/strategy/local_cache_middleware.rb:28:in `call'
  actionpack (4.2.3) lib/action_dispatch/middleware/static.rb:116:in `call'
  heroku-deflater (0.5.3) lib/heroku-deflater/skip_binary.rb:19:in `call'
  rack (1.6.4) lib/rack/deflater.rb:35:in `call'
  heroku-deflater (0.5.3) lib/heroku-deflater/serve_zipped_assets.rb:50:in `call'
  rack (1.6.4) lib/rack/sendfile.rb:113:in `call'
  airbrake (4.1.0) lib/airbrake/user_informer.rb:16:in `_call'
  airbrake (4.1.0) lib/airbrake/user_informer.rb:12:in `call'
  railties (4.2.3) lib/rails/engine.rb:518:in `call'
  railties (4.2.3) lib/rails/application.rb:165:in `call'
  rack (1.6.4) lib/rack/content_length.rb:15:in `call'
  thin (1.6.3) lib/thin/connection.rb:86:in `block in pre_process'
  thin (1.6.3) lib/thin/connection.rb:84:in `catch'
  thin (1.6.3) lib/thin/connection.rb:84:in `pre_process'
  thin (1.6.3) lib/thin/connection.rb:53:in `process'
  thin (1.6.3) lib/thin/connection.rb:39:in `receive_data'
  eventmachine (1.0.7) lib/eventmachine.rb:187:in `run_machine'
  eventmachine (1.0.7) lib/eventmachine.rb:187:in `run'
  thin (1.6.3) lib/thin/backends/base.rb:73:in `start'
  thin (1.6.3) lib/thin/server.rb:162:in `start'
  rack (1.6.4) lib/rack/handler/thin.rb:19:in `run'
  rack (1.6.4) lib/rack/server.rb:286:in `start'
  railties (4.2.3) lib/rails/commands/server.rb:80:in `start'
  railties (4.2.3) lib/rails/commands/commands_tasks.rb:80:in `block in server'
  railties (4.2.3) lib/rails/commands/commands_tasks.rb:75:in `tap'
  railties (4.2.3) lib/rails/commands/commands_tasks.rb:75:in `server'
  railties (4.2.3) lib/rails/commands/commands_tasks.rb:39:in `run_command!'
  railties (4.2.3) lib/rails/commands.rb:17:in `<top (required)>'
  script/rails:6:in `require'
  script/rails:6:in `<main>'

This worked fine with 0.15.0 so I don't know what the cause is. Also, unfortunately I haven't done Rails in half a decade, so I have no idea where to begin looking. Was something changed WRT static assets between 0.15.0 and 0.17.0? I looked at the diff of 0.17.0 and my sandstorm branch, but nothing jumps out at me.

I'd appreciate any help, as I'm officially out of ideas. Check out my repository https://dev.thewordnerd.info/nolan/loomio-sandstorm. You'll need Vagrant and Vagrant-spk installed, but at that point you can spin up an instance inside a VM and see it in action.

git clone --recursive https://dev.thewordnerd.info/git/nolan/loomio-sandstorm

I think if you do:

cd loomio
git checkout 4b81bb6

you should be back to the working version, as that looks like my initial changes pre 0.17.0 merge.

Thanks.

Arrrrr... Our funky lineman build system stikes again. Sorry about this.

The secret is hidden in lib/tasks/deploy.task

You need to run lineman build and cp -R lineman/dist/* public/ so that the client files are able to be served by the rails app.

Aha, that more or less did it. Thanks.

Now I have another issue in that I was editing the haml files under app/views, not the Lineman UI. When building/running the app, I set a SANDSTORM=1 environment variable so I can tweak the UI. In Lineman, what is the most idiomatic way to check for this environment variable and export something I can query for in the haml templates during the build?

One example is the logo. If the Loomio install isn't official, you output the hostname instead. I expect this is usually OK, but in Sandstorm that appears as something like "eb4c7aa000d30f9f31a5477b2a9920d5.local.sandstorm.io" which is a bit ugly. :) I'd rather replace that with something like "Loomio" in cases where SANDSTORM=1 is set in the build environment where Lineman is run. I'd also like to disable the link to your pricing plans in the start group form for self-hosters, unless there's some reason you'd like that intact (maybe you can check for officialness there too?)

I'm sure I can find the answer, but I'd probably have to understand Lineman/Grunt/several other frameworks first, and I'd honestly just rather get this done so folks have a nice one-click Loomio install. :) Thanks a bunch.

Hi @nolandarilek,

Check out app/helpers/angular_helper.rb, there you'll see where we set up the app configuration which is passed into the angular interface as the AppConfig.

I reckon some sort of line there like
custom: { sandstorm: ENV['SANDSTORM'] }

which will allow you to use AppConfig.custom.sandstorm in the angular side of things.

Excellent, thanks, that's entirely not where I was looking. :) I was in lineman/config/application.coffee, setting:

  haml:
    options:
      context:
        isInSandstorm: process.env.SANDSTORM?

Then accessing this.isInSandstorm from the templates. Unfortunately something appears to be swallowing the SANDSTORM environment variable somewhere and I can't figure out what.

I'll give your solution a try.

OK, spoke too soon, I don't see app/helpers/angular_helper.rb in 0.17.0. Was that created after 0.17 was released?

Oh.. right, it's been recently added. If you update to the latest (18.0), it will be there, alternatively in 17.0 it'll be in app/controllers/base_controller.rb

Hey, sorry for the slowness on this, I'm poking at it in my free time, and no offense, but this project is a bit heavy on dependencies and technologies. :)

Trying to get 0.18 running in Loomio, I see lots of attempts to connect to http://localhost:9292. I assume this is some sort of real-time thing for the Angular interface? If so, how do I start it, and is it something that only the app needs to connect to? Or do I need to expose it externally somehow?

Also, I see the following in my logs:

Started GET "/default-cover-photo.png" for 127.0.0.1 at 2015-11-26 02:54:08 +0000

ActionController::RoutingError (No route matches [GET] "/default-cover-photo.png"):

Thoughts on how to fix that? I thought I was generating all my static assets correctly.

I also see that the API uses the PATCH method in places, in setting volumes in comments for instance. Any chance this might be changed? Looks like Sandstorm's HTTP server may block the PATCH method for non-external APIs. I'm asking them about this as well, so perhaps there's hope of changing this from Sandstorm's side.

Thanks.

I just joined this group to say THANK YOU @nolandarilek I for one think it is utterly wonderful that you are porting Loomio to Sandstorm! :)

Hi @nolandarilek hey do you have any quick updates on this? I know you were poking at it in your spare time. If there's anything even remotely resembling an ETA for Loomio in the Sandstorm App Market, some people would be really excited! :)

Logged in to say I'm excited by this. Thanks for your effort.

Haven't poked at this since my last update, but I'm still stumped by the PATCH issue. In short, Sandstorm's HTTP server for apps supports a mostly complete HTTP subset that just happens to not include support for PATCH, which some of the client-side code relies on. I've filed an issue with Sandstorm but thus far it hasn't been resolved. Maybe chiming in there might help:

https://github.com/sandstorm-io/sandstorm/issues/1218

Not sure what to do in lieu of that. I've made lots of patches to Loomio to make it run within Sandstorm, but changing the API is something I want to avoid. I'd also eventually like to merge the Sandstorm patches upstream, because this work definitely has value but I unfortunately don't have time to be the sole maintainer.

Thanks for your continued interest, I definitely still want to see this happen.

Thanks for your work on this, @nolandarilek ; really great stuff that's very high value; the exact sort of projects we love to see get started.

It seems to me like removing PATCH support is something which could happen, but is unlikely to land on the core dev roadmap in the immediate future. I wonder if someone like Connor Turland, who's been doing some interoperability work around Loomio, might be interested in this project, which seems to have a tonne of overlap with the Collaborative Technology Alliance stuff we've been discussing recently.

I'm also worried that our recent overhaul of the build system may
a) make a sandstorm port much easier, and / or
b) clash some with the existing work.

Do you have an up-to-date repository for this work? I'd love to have another peek at it sometime, and the link provided above appears to 521 at the moment.

@nolandarilek I'm going to add PUT support for those client-side PATCH endpoints today.

I've added PUT endpoints for our current PATCHy actions here: https://github.com/loomio/loomio/pull/2910

Let me know if there's additional work ya need here.

I think I'd rather push Sandstorm to add PATCH, than add more if(isInSandstorm()) { ... } conditionals. Honestly this whole thing is getting a bit overwhelming, and I see all of those conditionals as a place where things might break in the future. :)

I'm working on pushing my current code to Github. The commit history is a bit messy because lots of these changes don't add value on their own, but do when combined with 6-7 others. :) I still need to make a few more tweaks because Sandstorm's own development build processes have changed since I last worked on this. I'll post later today or tomorrow with the updated repo.

Sounds good to me. I'd love to see the code on github; some of our recent interoperability talks have got me thinking that Sandstorm might be a very strategic piece for Loomio going forward.

I think where this wants to end up eventually is a loomio_sandstorm plugin, the architecture for which is next up on the roadmap (you could check out out the README for creating plugins here: https://github.com/loomio/loomio/tree/plugins-effyeah/plugins)

I'm a big fan of the Sandstorm connection and of Loomio being enabled as a plug-in; thanks for sharing the github doc James! Just within the context of current Sandstorm apps such as Etherpad and Wekan, a Loomio plug-in could be a big game-changer.

I'd like to ask the meta-question: what's a good way to ensure that Sandstorm and Loomio don't become subtly incompatible over time as they change? Having both projects use strict SemVer for at least certain APIs?

How stable are the relevant Sandstorm APIs?

I don't understand the connection points between Loomio and Sandstorm, so I don't know what can change/break nor how likely those changes/breakages are, but I'm trying to think of a way that no one has to repeat some version of what @nolandarilek has been doing for every release!

Yes, it seems totally understandable to me that Nolan's at the end of his rope when it comes to doing Loomio customization; Loomio has been decidedly not easy to extend up until this point. That will change drastically this year.

From https://docs.sandstorm.io/en/latest/developing/handbook/ --

The platform is in alpha, so this document is likely to change. We will keep old versions of this document online, and we hope that you will watch this document over time to update your app as the platform updates. There will eventually be API deprecations and backwards-incompatible breakage, though we aim to keep that to a minimum.

Sorry, my gitbucket instance isn't happy so I'll just host this on Github. Here it is:

https://github.com/ndarilek/loomio-sandstorm

Note that Loomio is pulled in via a submodule so you'll need to use --recursive on clone or initialize submodules after.

I definitely want to see this happen but feel like I've bitten off more than I can chew. So any help with this would be greatly appreciated, and I can handle publication to the Sandstorm app market for the time being. This work is based on 0.18. Here are some general notes/challenges:

• Regarding the current PATCH issue, it's been nearly a decade since I used Rails. At the time I recall it having a hack where a method query parameter translated into the HTTP method, so https://my.app/api/endpoint?method=PATCH looked like a PATCH even if the real method was GET. Does this still exist, and if so, would anyone mind appending the magic query parameter to all PATCH requests until the issue is fixed on Sandstorm's end? I tried googling for this but couldn't find it, maybe the behavior was dropped or is packaged in a gem.
• Loomio knows it is running in Sandstorm because the SANDSTORM environment variable is set (value doesn't matter.) I've added a bunch of server-side checks for this variable, have bridged it into the Angular controllers, and have attempted to modify the interface accordingly.
• Sandstorm doesn't support sending emails--Loomio doesn't get a user's email address at all, if in fact one is even registered. Likewise, Sandstorm provides usernames, avatars, etc. I'm disabling portions of the interface in Sandstorm if they don't make sense. For instance, you can't set an email address or upload an avatar. You can't delete your account. Likewise, I disabled group/thread volume settings that imply you'll receive emails. We'll need to disable any portions of the interface that don't make sense, based on the presence of the SANDSTORM environment variable.
• It looks like some Loomio functionality is only available in the admin interface. We'll need to insert a link to /admin somewhere in the UI, because it's a bit difficult to create this link yourself based on the random Sandstorm subdomain for the instance (which is only opened in an iframe and not in the address bar.) I'm not sure where this link would look best. I added it to the older interface but not to the new Angular one.
• Likewise, it doesn't make sense to log out of a Sandstorm-hosted Loomio. Instead, you'd log out of Sandstorm itself. I think I disabled the logout link but who knows what may have changed since 0.18. :)
• Another random challenge I face is that I'm blind, so there are some aspects of this port that I can't verify. For instance, Sandstorm hosts its own user avatars so users can use the same ones across multiple apps, segment them into different identities, etc. I think I've tweaked Loomio to present Sandstorm's avatar URLs instead of its own but can't be sure. Likewise, I hope it is serving up stylesheets, group cover photos, etc. OK but I can't be sure. I think there were some issues uploading group photos that I never resolved, but again I can't test whether uploading group photos works even if I did have the functionality working so it hasn't been a priority for me. :) This is why I'm asking about placement of admin links, because your interface is complex enough that I'm not sure where they would look best.

When I left this I verified basic group create/edit/post functionality, but Loomio is a big app, I'm not too familiar with it, and I'm at a point where I don't want to base my co-op/business/whatever I'm building administrative infrastructure on something that can't run in Sandstorm so my first objective was to get Loomio running in it. Didn't realize how big this task would be. :) Most apps have a smaller scope and are fairly easy to assess, but Loomio is pretty large, and I'd appreciate help finishing this port off and testing functionality you'd expect to work. Happy to answer any Sandstorm-specific questions and accept PRs, or even grant write access to the repo. I don't need to own this, I just want to see it finished. :) I'd also appreciate help cleaning up the commit history if that is a requirement to merge the Sandstorm-specific Loomio changes upstream. This port has touched a lot of files, and I figured a few larger commits were easier to work with than a bunch of microcommits that couldn't individually add much value on their own. But admittedly toward the end I went a bit crazy and just wanted to

...save my progress. Apologies, last comment got cut off awkwardly. :)

This is a great help, thanks Nolan!

@nolandarilek thanks so much for the awesome handover. I wonder if your bullet points should be issues in Github, either on loomio/loomio, or on your new repo - what do you think?

And @jameskiesel - would you be up for becoming a maintainer on that repo?

Incidentally, we've been doing some work to make Loomio work well for people who are blind. If you're interested to share your experiences so we can continue to improve the interface, we'd love to have you in the Accessibility group.

Thanks again! I'm so excited about the positive social impact this work is going to have once it's ready for production :)

Yeah, I'd actually jumped on Loomio right about when your accessibility changes were landing and was/continue to be impressed. That's part of why I really want to see this Sandstorm port finished. :) Thanks for the accessibility link, I just now hit an issue attempting to skim this thread so I'll mention it there.

Cool. So I think this is very much a plugin-able type bit of functionality, which is cool because it means that if you want to run an instance on Sandstorm, you'll be able to do it by adding a single line in a configuration file, and if not, you don't have to worry about it.

I'm knee-and-wrist deep in plugin territory now, but give me a little time (a week, two, something like that) to mature it a little bit and we'll see what we can do from there.

Great to witness this work unfolding, and looking forward to eating the fruits :)

Thank you @nolandarilek @jameskiesel @richarddbartlett

Just an update:

PATCH support on Sandstorm has shipped, so that's no longer an issue. Thanks to folks for +1ing that. :)

We're now dealing with this:

https://github.com/sandstorm-io/sandstorm/issues/1489

which is a new development since I left off last November, and appears to be caused by issues with the Sandstorm Vagrant development box. Hoping that gets resolved soon.

Hello,

I was wondering if anyone had followed up on this and whether people think the port is possible now that sandstorm is more mature. Unfortunately I am not a developer so can't help on technical side but I use both sandstorm and loomio so more that happy to to testing.

@nolandarilek I´d like to talk to you in private. Can you get in touch through cavalcanti.fiocruz@gmail.com? tks

This is just a bump up to see if there has been any updates to the project. Loomio on Sandstorm would be just fantastic. Hope interest in this project has not waned.

We at Loomio have not heard anything about Loomio+Sandstorm since this thread. In the meantime our Docker based deploy system has been significantly improved though

edit: corrected the link