Class: Falqon::Queue

Inherits:
Object
  • Object
show all
Extended by:
T::Sig
Includes:
Hooks
Defined in:
lib/falqon/queue.rb

Overview

Simple, efficient, and reliable messaging queue implementation

Defined Under Namespace

Classes: Metadata

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(name, retry_strategy: Falqon.configuration.retry_strategy, max_retries: Falqon.configuration.max_retries, retry_delay: Falqon.configuration.retry_delay, redis: Falqon.configuration.redis, logger: Falqon.configuration.logger, version: Falqon::PROTOCOL) ⇒ void

Create a new queue

Create a new queue in Redis with the given name. If a queue with the same name already exists, it is reused. When registering a new queue, the following Metadata is stored:

  • created_at: Timestamp of creation

  • updated_at: Timestamp of last update

  • version: Protocol version

Initializing a queue with a different protocol version than the existing queue will raise a VersionMismatchError. Currently queues are not compatible between different protocol versions, and must be deleted and recreated manually. In a future version, automatic migration between protocol versions may be supported.

Please note that retry strategy, maximum retries, and retry delay are configured per queue instance, and are not shared between queue instances.

Examples:

Create a new queue

queue = Falqon::Queue.new("my_queue")
queue.name # => "my_queue"
queue.id # => "falqon/my_queue"

Parameters:

  • name (String) —

    The name of the queue (without prefix)

  • retry_strategy (Symbol) (defaults to: Falqon.configuration.retry_strategy) —

    The retry strategy to use for failed messages

  • max_retries (Integer) (defaults to: Falqon.configuration.max_retries) —

    The maximum number of retries before a message is considered failed

  • retry_delay (Integer) (defaults to: Falqon.configuration.retry_delay) —

    The delay in seconds before a message is eligible for a retry

  • redis (ConnectionPool) (defaults to: Falqon.configuration.redis) —

    The Redis connection pool to use

  • logger (Logger) (defaults to: Falqon.configuration.logger) —

    The logger to use

  • version (Integer) (defaults to: Falqon::PROTOCOL) —

    The protocol version to use

Raises:



71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
 
# File 'lib/falqon/queue.rb', line 71

def initialize(
  name,
  retry_strategy: Falqon.configuration.retry_strategy,
  max_retries: Falqon.configuration.max_retries,
  retry_delay: Falqon.configuration.retry_delay,
  redis: Falqon.configuration.redis,
  logger: Falqon.configuration.logger,
  version: Falqon::PROTOCOL
)
  @name = name
  @id = [Falqon.configuration.prefix, name].compact.join("/")
  @retry_strategy = Strategies.const_get(retry_strategy.to_s.capitalize).new(self)
  @max_retries = max_retries
  @retry_delay = retry_delay
  @redis = redis
  @logger = logger
  @version = version

  redis.with do |r|
    queue_version = r.hget("#{id}:metadata", :version)

    raise Falqon::VersionMismatchError, "Queue #{name} is using protocol version #{queue_version}, but this client is using protocol version #{version}" if queue_version && queue_version.to_i != @version

    r.multi do |t|
      # Register the queue
      t.sadd([Falqon.configuration.prefix, "queues"].compact.join(":"), name)

      # Set creation and update timestamp (if not set)
      t.hsetnx("#{id}:metadata", :created_at, Time.now.to_i)
      t.hsetnx("#{id}:metadata", :updated_at, Time.now.to_i)

      # Set protocol version
      t.hsetnx("#{id}:metadata", :version, @version)
    end
  end

  run_hook :initialize, :after
end

Instance Attribute Details

#id ⇒ String (readonly)

The identifier of the queue (with prefix)

Returns:

  • (String) 


19
20
21
 
# File 'lib/falqon/queue.rb', line 19

def id
  @id
end

#max_retries ⇒ Integer (readonly)

The maximum number of retries before a message is considered failed

Returns:

  • (Integer) 


27
28
29
 
# File 'lib/falqon/queue.rb', line 27

def max_retries
  @max_retries
end

#name ⇒ String (readonly)

The name of the queue (without prefix)

Returns:

  • (String) 


15
16
17
 
# File 'lib/falqon/queue.rb', line 15

def name
  @name
end

#retry_delay ⇒ Integer (readonly)

The delay in seconds before a message is eligible for a retry

Returns:

  • (Integer) 


31
32
33
 
# File 'lib/falqon/queue.rb', line 31

def retry_delay
  @retry_delay
end

#retry_strategy ⇒ Strategy (readonly)

The configured retry strategy of the queue

Returns:

  • (Strategy) 


23
24
25
 
# File 'lib/falqon/queue.rb', line 23

def retry_strategy
  @retry_strategy
end

Class Method Details

.all ⇒ Array<Queue>

Get a list of all registered queues

Returns:

  • (Array<Queue>) —

    The queues



585
586
587
588
589
590
591
592
 
# File 'lib/falqon/queue.rb', line 585

def all
  Falqon.configuration.redis.with do |r|
    r
      .smembers([Falqon.configuration.prefix, "queues"].compact.join(":"))
      .sort
      .map { |id| new(id) }
  end
end

.size ⇒ Integer

Get the number of active (registered) queues

Returns:

  • (Integer) —

    The number of active (registered) queues



599
600
601
602
603
604
 
# File 'lib/falqon/queue.rb', line 599

def size
  Falqon.configuration.redis.with do |r|
    r
      .scard([Falqon.configuration.prefix, "queues"].compact.join(":"))
  end
end

Instance Method Details

#clear ⇒ Array<Identifier>

Clear the queue, removing all messages

This method clears all messages from the queue, including pending, processing, scheduled, and dead messages. It also resets the metadata counters for processed, failed, and retried messages, but does not deregister the queue.

Examples:

Clear the queue

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!")
queue.clear # => ["1"]

Returns:

  • (Array<Identifier>) —

    The identifiers of the cleared messages



319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
 
# File 'lib/falqon/queue.rb', line 319

def clear
  logger.debug "Clearing queue #{name}"

  run_hook :clear, :before

  # Clear all sub-queues
  message_ids = pending.clear + processing.clear + scheduled.clear + dead.clear

  redis.with do |r|
    r.multi do |t|
      # Clear metadata
      t.hdel("#{id}:metadata", :processed, :failed, :retried)

      # Set update timestamp
      t.hset("#{id}:metadata", :updated_at, Time.now.to_i)
    end
  end

  run_hook :clear, :after

  # Return identifiers
  message_ids
end

#delete ⇒ void

This method returns an undefined value.

Delete the queue, removing all messages and deregistering the queue

This method deletes the queue, removing all messages, metadata, and deregisters the queue.

Examples:

Delete the queue

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!")
queue.clear # => nil


353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
 
# File 'lib/falqon/queue.rb', line 353

def delete
  logger.debug "Deleting queue #{name}"

  run_hook :delete, :before

  # Delete all sub-queues
  [pending, processing, scheduled, dead]
    .each(&:clear)

  redis.with do |r|
    r.multi do |t|
      # Delete metadata
      t.del("#{id}:metadata")

      # Deregister the queue
      t.srem([Falqon.configuration.prefix, "queues"].compact.join(":"), name)
    end
  end

  run_hook :delete, :after
end

#empty? ⇒ Boolean

Check if the queue is empty

Only the pending queue is checked for emptiness.

Returns:

  • (Boolean) —

    Whether the queue is empty



515
516
517
 
# File 'lib/falqon/queue.rb', line 515

def empty?
  size.zero?
end

#metadata ⇒ Metadata

Metadata of the queue

Returns:

  • (Metadata) —

    The metadata of the queue

See Also:



525
526
527
528
529
530
 
# File 'lib/falqon/queue.rb', line 525

def metadata
  redis.with do |r|
    Metadata
      .parse(r.hgetall("#{id}:metadata"))
  end
end

#peek(index: 0) ⇒ Data?

Peek at the next message in the queue

Use #range to peek at a range of messages. This method does not block.

Examples:

Peek at the next message

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!")
queue.peek # => "Hello, world!"
queue.pop # => "Hello, world!"

Peek at the next message with an offset

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!", "Goodbye, world!")
queue.peek(1) # => "Goodbye, world!"
queue.pop # => "Hello, world!"

Parameters:

  • index (Integer) (defaults to: 0) —

    The index of the message to peek at

Returns:

  • (Data, nil) —

    The data of the peeked message



257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
 
# File 'lib/falqon/queue.rb', line 257

def peek(index: 0)
  logger.debug "Peeking at next message in queue #{name}"

  run_hook :peek, :before

  # Get identifier from pending queue
  message_id = pending.peek(index:)

  return unless message_id

  run_hook :peek, :after

  # Retrieve data
  Message.new(self, id: message_id).data
end

#pop(&block) ⇒ Data?

Pop data from the head of the queue

This method blocks until a message is available.

Acknowledgement

If a block is given, the popped data is passed to the block. If the block raises a Error exception, the message is retried according to the configured retry strategy. If no exception is raised, the message is ackwnowledged and removed from the queue.

If no block is given, the popped data is returned. The message is immediately acknowledged and removed from the queue.

Examples:

Pop a message (return-style)

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!")
queue.pop # => "Hello, world!"

Pop a message (block-style)

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!")
queue.pop do |data|
  puts data # => "Hello, world!"
end

Parameters:

  • block (T.proc.params(data: Data).void, nil) —

    A block to execute with the popped data (block-style)

Returns:

  • (Data, nil) —

    The popped data (return-style)



181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
 
# File 'lib/falqon/queue.rb', line 181

def pop(&block)
  logger.debug "Popping message from queue #{name}"

  run_hook :pop, :before

  message = redis.with do |r|
    # Move identifier from pending queue to processing queue
    message_id = r.blmove(pending.id, processing.id, :left, :right).to_i

    # Get retry count
    retries = r.hget("#{id}:metadata:#{message_id}", :retries).to_i

    r.multi do |t|
      # Set message status
      t.hset("#{id}:metadata:#{message_id}", :status, "processing")

      # Set update timestamp
      t.hset("#{id}:metadata", :updated_at, Time.now.to_i)
      t.hset("#{id}:metadata:#{message_id}", :updated_at, Time.now.to_i)

      # Increment processing counter
      t.hincrby("#{id}:metadata", :processed, 1)

      # Increment retry counter if message is retried
      t.hincrby("#{id}:metadata", :retried, 1) if retries.positive?
    end

    Message.new(self, id: message_id)
  end

  data = message.data

  yield data if block

  run_hook :pop, :after

  # Remove identifier from processing queue
  processing.remove(message.id)

  # Delete message
  message.delete

  data
rescue Error => e
  logger.debug "Error processing message #{message.id}: #{e.message}"

  # Increment failure counter
  redis.with { |r| r.hincrby("#{id}:metadata", :failed, 1) }

  # Retry message according to configured strategy
  retry_strategy.retry(message, e)

  nil
end

#push(*data) ⇒ Identifier+

Push data onto the tail of the queue

Examples:

Push a single message

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!") # => "1"

Push multiple messages

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!", "Goodbye, world!") # => ["1", "2"]

Parameters:

  • data (Data) —

    The data to push onto the queue (one or more strings)

Returns:



124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
 
# File 'lib/falqon/queue.rb', line 124

def push(*data)
  logger.debug "Pushing #{data.size} messages onto queue #{name}"

  run_hook :push, :before

  # Set update timestamp
  redis.with { |r| r.hset("#{id}:metadata", :updated_at, Time.now.to_i) }

  ids = data.map do |d|
    message = Message
      .new(self, data: d)
      .create

    # Push identifier to queue
    pending.add(message.id)

    # Set message status
    redis.with { |r| r.hset("#{id}:metadata:#{message.id}", :status, "pending") }

    # Return identifier(s)
    data.size == 1 ? (return message.id) : (next message.id)
  end

  run_hook :push, :after

  # Return identifier(s)
  ids
end

#range(start: 0, stop: -1)) ⇒ Array<Data>

Peek at the next messages in the queue

Use #peek to peek at a single message. This method does not block.

Examples:

Peek at the next messages

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!", "Goodbye, world!", "Hello again, world!")
queue.range(start: 1, stop: 2) # => ["Goodbye, world!", "Hello again, world!"]
queue.range(start: 1) # => ["Goodbye, world!", "Hello again, world!"]
queue.pop # => "Hello, world!"

Parameters:

  • start (Integer) (defaults to: 0) —

    The start index of the range to peek at

  • stop (Integer) (defaults to: -1)) —

    The stop index of the range to peek at (set to -1 to peek at all messages)

Returns:

  • (Array<Data>) —

    The data of the peeked messages



290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
 
# File 'lib/falqon/queue.rb', line 290

def range(start: 0, stop: -1)
  logger.debug "Peeking at next messages in queue #{name}"

  run_hook :range, :before

  # Get identifiers from pending queue
  message_ids = pending.range(start:, stop:)

  return [] unless message_ids.any?

  run_hook :range, :after

  # Retrieve data
  message_ids.map { |id| Message.new(self, id:).data }
end

#refill ⇒ Array<Identifier>

Refill the queue with messages from the processing queue

This method moves all messages from the processing queue back to the pending queue (in order). It is useful when a worker crashes or is stopped, and messages are left in the processing queue.

Examples:

Refill the queue

queue = Falqon::Queue.new("my_queue")
queue.push("Hello, world!")
queue.pop { Kernel.exit! }
...
queue.refill # => ["1"]

Returns:

  • (Array<Identifier>) —

    The identifiers of the refilled messages



390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
 
# File 'lib/falqon/queue.rb', line 390

def refill
  logger.debug "Refilling queue #{name}"

  run_hook :refill, :before

  message_ids = []

  # Move all identifiers from tail of processing queue to head of pending queue
  redis.with do |r|
    while (message_id = r.lmove(processing.id, id, :right, :left))
      # Set message status
      r.hset("#{id}:metadata:#{message_id}", :status, "pending")

      message_ids << message_id
    end
  end

  run_hook :refill, :after

  message_ids
end

#revive ⇒ Array<Identifier>

Revive the queue with messages from the dead queue

This method moves all messages from the dead queue back to the pending queue (in order). It is useful when messages are moved to the dead queue due to repeated failures, and need to be retried.

Examples:

Revive the queue

queue = Falqon::Queue.new("my_queue", max_retries: 0)
queue.push("Hello, world!")
queue.pop { raise Falqon::Error }
queue.revive # => ["1"]

Returns:

  • (Array<Identifier>) —

    The identifiers of the revived messages



426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
 
# File 'lib/falqon/queue.rb', line 426

def revive
  logger.debug "Reviving queue #{name}"

  run_hook :revive, :before

  message_ids = []

  # Move all identifiers from tail of dead queue to head of pending queue
  redis.with do |r|
    while (message_id = r.lmove(dead.id, id, :right, :left))
      # Set message status
      r.hset("#{id}:metadata:#{message_id}", :status, "pending")

      message_ids << message_id
    end
  end

  run_hook :revive, :after

  message_ids
end

#schedule ⇒ Array<Identifier>

Schedule failed messages for retry

This method moves all eligible messages from the scheduled queue back to the head of the pending queue (in order). Messages are eligible for a retry according to the configured retry strategy.

Examples:

Schedule failed messages

queue = Falqon::Queue.new("my_queue", max_retries: 0, retry_delay: 5, retry_strategy: :linear)
queue.push("Hello, world!")
queue.pop { raise Falqon::Error }
queue.schedule # => []
sleep 5
queue.schedule # => ["1"]

Returns:

  • (Array<Identifier>) —

    The identifiers of the scheduled messages



464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
 
# File 'lib/falqon/queue.rb', line 464

def schedule
  logger.debug "Scheduling failed messages on queue #{name}"

  run_hook :schedule, :before

  message_ids = T.let([], T::Array[Identifier])

  # Move all due identifiers from scheduled queue to head of pending queue
  redis.with do |r|
    # Select all identifiers that are due (score <= current timestamp)
    # FIXME: replace with zrange(by_score: true) when https://github.com/sds/mock_redis/issues/307 is resolved
    # TODO: work in batches
    message_ids = r.zrangebyscore(scheduled.id, 0, Time.now.to_i).map(&:to_i)

    logger.debug "Scheduling messages #{message_ids.join(', ')} on queue #{name}"

    r.multi do |t|
      message_ids.each do |message_id|
        # Set message status
        t.hset("#{id}:metadata:#{message_id}", :status, "pending")

        # Add identifier to pending queue
        pending.add(message_id)

        # Remove identifier from scheduled queue
        scheduled.remove(message_id)
      end
    end
  end

  run_hook :schedule, :after

  message_ids
end

#size ⇒ Integer

Size of the queue

Returns:

  • (Integer) —

    The number of messages in the queue



504
505
506
 
# File 'lib/falqon/queue.rb', line 504

def size
  pending.size
end