javascript - SQS with MongoDB for handling duplicates -


i have couple of ideas stopping duplicate handling of messages amazon's sqs queues. app have mongodb server, think can effective part of either strategy:

  1. store queue items in mongo, 'status' field - default pending. use sqs queue id of new message. 1 of worker processes id, findandmodify on actual item in mongo set status processing, unless it's being processed, when flag up.

  2. store queue items in queue. workers pick items queue, attempt insert mongo item id , other info. if item existed, don't insert or continue, since it's dupe.

the problems , questions have:

  • solution 1 seems counter-intuitive: why use sqs @ all? think it's because polling sqs more correct whole load of worker processes polling mongo work.

  • solution 2 don't know how implement. there atomic find-and-insert-if-doesn't-exist? simple get-or-insert-but-tell-me-which-occurred operation trick.

  • will of these work in large scale scenario, and/or there proven method haven't grasped?


....humm, wrote question above, had thought get-or-insert-but-tell-me-which-occurred operation (in js psuedocode):

var thingy = getrandomnumber();  findandmodify({     new: false,      upsert: true,     query: { $eq: { id: item_id } },     update: { thingy: thingy },     fields: { thingy: 1 } }); 

if item exists (and conflict), since new false, old document returned.

if item didn't exist, new false, empty document {} returned.

so either got {}, indicating resulted in insert, or actual document, indicating get, , id exists... atomic. thingy in there because don't know if mongodb needs data there, guess would? if used $inc on duplicates field instead, work upsert? stats on dupes later.

is right, maybe work?


Comments

Popular posts from this blog

google chrome - Developer tools - How to inspect the elements which are added momentarily (by JQuery)? -

angularjs - Showing an empty as first option in select tag -

php - Cloud9 cloud IDE and CakePHP -