Node.js Job Queue

Reference:
Kue, https://github.com/Automattic/kue

Document/Example:
https://www.linkedin.com/pulse/job-queue-nodejs-adrien-desbiaux

1. Install Redis
2. Install Kue

Usage:
Producer
1. 建立queue instance
    var kue = require('kue'),
    var jobs = kue.createQueue();

2. 建立job
    job = jobs.create('bessel-filter-image', jobArgs).save();

3. 等待job 結束(callback)
     job
      .on('complete', function(result) { //whatever you want to do })
      .on('failed', function() { console.log(job.id); });
Consumer:
1. 建立queue instance?
2. 處理job
3. 處理完呼叫done

jobs.process('bessel-filter-image', function(job, done) {
   console.log(job.data.whatever); // stored in jobArgs
   var result = bessel(); // let's imagine applying the filter
   done(null, result); // forward the result of your job
});


更新Job Progress....
http://stackoverflow.com/questions/15375126/fetching-the-result-of-a-kue-job-and-pushing-this-to-the-client-over-open-connec

actually this is covered in the documentation - https://github.com/LearnBoost/kue

"Job Events

Job-specific events are fired on the Job instances via Redis pubsub. The following events are currently supported:

• failed the job has failed
• complete the job has completed
• promotion the job (when delayed) is now queued
• progress the job's progress ranging from 0-100 For example this may look something like the following:

  var job = jobs.create('video conversion', {
  
      title: 'converting loki\'s to avi'
    , user: 1
    , frames: 200
  
  });
  
  job.on('complete', function(){
      console.log("Job complete");
  }).on('failed', function(){
      console.log("Job failed");
  }).on('progress', function(progress){
      process.stdout.write('\r  job #' + job.id + ' ' + progress + '% complete');
  });

bare in mind that your job might not be processed immediatly (depends on your queue), so the client can wait some time for a result..

EDIT: as mentioned in the comments, a job doesn't return any results so you should store the result in the database along with the job id and query the database when the job is complete.

in order to keep the connection open, use res.write and res.end instead of res.json which ends the connection (You'll have to JSON.stringify the data yourself). also, remember that the browser can timeout if this takes too long..

Updating Progress

For a "real" example, let's say we need to compile a PDF from numerous slides with node-canvas. Our job may consist of the following data, note that in general you should not store large data in the job it-self, it's better to store references like ids, pulling them in while processing.

queue.create('slideshow pdf', {
    title: user.name + "'s slideshow"
  , slides: [...] // keys to data stored in redis, mongodb, or some other store
});

We can access this same arbitrary data within a separate process while processing, via the job.data property. In the example we render each slide one-by-one, updating the job's log and progress.

queue.process('slideshow pdf', 5, function(job, done){
  var slides = job.data.slides
    , len = slides.length;

  function next(i) {
    var slide = slides[i]; // pretend we did a query on this slide id ;)
    job.log('rendering %dx%d slide', slide.width, slide.height);
    renderSlide(slide, function(err){
      if (err) return done(err);
      job.progress(i, len, {nextSlide : i == len ? 'itsdone' : i + 1});
      if (i == len) done()
      else next(i + 1);
    });
  }

  next(0);
});