Background tasks (task module)

[gr2m]

This is a high-level issue for a complicated task, meant for discussion and to find a "champion" who wants to take ownership of bringing background tasks back to Hoodie. Comment below if you are interested in helping us :)

❓ The Motivation

Background tasks can be used to trigger server-side logic provided by plugins. Tasks are simply JSON documents just like data. And just as with data, we have offline sync, so users can start tasks even without an internet connectivity, which is more robust than relying on REST APIs.

🤔 What you will need to know to work on this issue

Node.js, hapi, PouchDB and you should have a general understanding of Hoodie’s sync architecture.

We had background tasks implemented in the "old" hoodie, tasks were simply stored in the users database and synchronized to the user’s database in the backend. We found out eventually that this was a bad design decision, as we want to use tasks even if the user did not yet sign up, and listening to all users databases has a very bad performance impact. So for the new hoodie, we decided to go with a dedicated "tasks" database instead.

We already have @hoodie/task-client which is very little code, it is using [@hoodie/store-client under the hood. And we have @hoodie/task-server which exposes only the CouchDB APIs relevant for sync. The idea is that the server is filtering /_changes so that a user can only see the tasks belonging to them, but it is not yet implemented. I would recommend to compare to hoodie-store-server’s implementation and start from there. Note that hoodie-store-server also abstracts away whether the app is using a CouchDB as its backend and if not falls back to PouchDB. We need the same for the task server module.

And lastly there is hoodie-standalone-task, a module we created to test the client & server together. I would recommend to use it for local development and testing. Once we got it all working in hoodie-standalone-task we can add the server & client code to hoodie itself (in @hoodie/client and @hoodie/server respectively).

hoodie-standalone-task is only a temporary repository so whoever is championing this, I’m happy to give you full admin rights so you can just merge your own pull requests yourself as you need it. For example, I think it would make sense to extend demo/index.html to have some UI that might help with debugging, like logging events into a table for example. Unfortunately there seems to be a bug in hoodie-standalone-task right now, I couldn’t get it work myself, but it worked in the past. So first task will be to get it working again :)

[inator]

I'm in!

[inator]

After working with Gregor today to gain a better understanding of the vision for tasks (thanks @gr2m!), here's what I propose are the low fruit todos to get started:

• hoodie-standalone-task: fix bug(s) to get it running again (see comment from Gregor) (see comment below)
• task-server
  • task-server: proxy queue endpoints ot tasks db
  • task-server: create central tasks database, listen for changes, emit events
  • task-server: create central tasks database
  • task-server: update READMEs
  • task-server: update package(s)
  • task-server: port store API to support tasks
  • task-server: PR working code
• task-client
  • task-client: instantiate store-client using server "queue" route
  • task-client: provide client API for adding, removing, restarting a task
  • task-client: provide client API for handling task events; on, one, off
  • task-client: handle purge/restore of local tasks store on login/logoff
  • task-client: update READMEs
  • task-client: update package(s)
  • task-client: PR working code
  • task-client: Add integration tests (see issue)

More to come, please comment on additions/changes to this list and I'll update accordingly. Thanks!

[gr2m]

👍 sounds good!

[inator]

@gr2m you wrote:

Unfortunately there seems to be a bug in hoodie-standalone-task right now, I couldn’t get it work myself, but it worked in the past. So first task will be to get it working again :)

Can you explain the buggy behavior? I'm running it now and it would be helpful to know what you were experiencing before I dive deeper. So far the PouchDB instance starts, client bundling occurs, and on the surface it seems to be running ok aside from the 404 errors noted as expected. Granted I haven't tried to do anything useful with it yet. :)

[gr2m]

When I run npm start in hoodie-standalone-task after installing dependencies, I see

$ npm start

> hoodie-standalone-task@ start /private/tmp/hoodie-standalone-task
> bin/server.js

161129/210509.099, [log,couchdb-task], data: Starting PouchDB Server...
161129/210510.078, [log,couchdb-task], data: PouchDB Server ready at localhost:1234/_utils
Server running at: http://gregor-2.local:3000
161129/210517.457, [response], http://gregor-2.local:3000: get / {} 200 (21ms) 
161129/210517.480, [response], http://gregor-2.local:3000: get /task.js {} 200 (28ms) 
161129/210517.694, [response], http://gregor-2.local:3000: get /api/queue/ {} 404 (6ms) 
161129/210517.833, [response], http://gregor-2.local:3000: put /api/queue/ {} 404 (7ms) 
161129/210517.857, [response], http://gregor-2.local:3000: get /favicon.ico {} 404 (2ms) 
161129/210518.570, [response], http://gregor-2.local:3000: get /api/queue/ {} 404 (2ms) 
161129/210518.575, [response], http://gregor-2.local:3000: put /api/queue/ {} 404 (3ms) 
161129/210519.444, [response], http://gregor-2.local:3000: get /api/queue/ {} 404 (2ms) 
161129/210519.450, [response], http://gregor-2.local:3000: put /api/queue/ {} 404 (1ms) 

the gets and puts continue, something’s not right there. I tested with a fresh install using node v6.9.1 and npm 3.10.9

[gr2m]

But note that the setup that spawns pouchdb-server in a child process is outdated, we no longer use it. Instead, @hoodie/task-server should either proxy to a CouchDB (maybe we can start with that) or use express-pouchdb in other cases. Here is how @hoodie/store-server does it

I’d spend a moment to try fix the setup with hoodie-standalone-task, but if it becomes a rabbit whole, I’ll try to do the same we do in@hoodie/store-server and compare to how https://github.com/hoodiehq/hoodie-store works, it is to the Hoodie Store module what hoodie-standalone-task is to the Task module

[inator]

An update:

After a thorough review of the hoodie-standalone-tasks repo, and an attempt to repair, Gregor and I determined that it's just going to take too much to bring it up to date and functional. Instead, I've started the process of creating a new standalone "hoodie-task" working example using hoodie-store as the basis.

So far, I've managed to get the REST portion of the tasks-server working with the endpoints as originally conceived - minus any queue id filtering, which will be necessary in the end product. Aside from filtering, next up will be the task server side API, which will surely include support for task events much like the store API does today. And since this is all based on the store-server module, I believe much of that will already be baked in through the Hoodie Store API. I also plan to leverage the other Store APIs that should allow for server-side manipulation of the lifecycle of tasks, including creation, updates, closing, etc.

All of this will likely make more sense when the code is pushed, but I wanted to get the basic thought process out there for comments sooner than later. More to come!

[inator]

@gr2m (and anyone else that's interested) - please have a look at this approach and let me know if I'm on the right track. Thanks much!

[gr2m]

left a comment at https://github.com/appback/task-server/commit/8e42dc4933e7839329bd7c0b5d8373131a4d54f6#commitcomment-20270126

[inator]

@gr2m: First task-server PR now ready for your review.

[inator]

@gr2m - as it relates to the start() method here:

queue.start(attributes/*, options */)

I'm wondering if we should go with something like this instead:

queue.add(attributes/*, options */)

For me personally, I would think of queue.start() as a method for starting the entire queue, rather than initiating a single task item.

your thoughts?