javascript - SQS with MongoDB for handling duplicates -
i have couple of ideas stopping duplicate handling of messages amazon's sqs queues. app have mongodb server, think can effective part of either strategy:
store queue items in mongo, 'status' field - default pending. use sqs queue id of new message. 1 of worker processes id,
findandmodify
on actual item in mongo set status processing, unless it's being processed, when flag up.store queue items in queue. workers pick items queue, attempt insert mongo item id , other info. if item existed, don't insert or continue, since it's dupe.
the problems , questions have:
solution 1 seems counter-intuitive: why use sqs @ all? think it's because polling sqs more correct whole load of worker processes polling mongo work.
solution 2 don't know how implement. there atomic find-and-insert-if-doesn't-exist? simple get-or-insert-but-tell-me-which-occurred operation trick.
will of these work in large scale scenario, and/or there proven method haven't grasped?
....humm, wrote question above, had thought get-or-insert-but-tell-me-which-occurred operation (in js psuedocode):
var thingy = getrandomnumber(); findandmodify({ new: false, upsert: true, query: { $eq: { id: item_id } }, update: { thingy: thingy }, fields: { thingy: 1 } });
if item exists (and conflict), since new
false
, old document returned.
if item didn't exist, new
false
, empty document {}
returned.
so either got {}
, indicating resulted in insert, or actual document, indicating get, , id exists... atomic. thingy
in there because don't know if mongodb needs data there, guess would? if used $inc
on duplicates
field instead, work upsert? stats on dupes later.
is right, maybe work?
Comments
Post a Comment