jobqueue
Differences
This shows you the differences between two versions of the page.
|
jobqueue [2011/07/26 15:23] gonium |
jobqueue [2012/10/30 10:27] (current) |
||
|---|---|---|---|
| Line 36: | Line 36: | ||
| https://github.com/mysmartgrid/msg-qworker | https://github.com/mysmartgrid/msg-qworker | ||
| - | The gem can be installed using `gem install msg-qworker`. | + | The gem can be installed using |
| + | |||
| + | $ gem install msg-qworker | ||
| + | |||
| + | The library consists of a producer, a qworker and the drunken sailor. | ||
| + | |||
| + | * The producer puts job in the default tube. This should be used as a template on how to put jobs (JSON-formatted) in the tube. | ||
| + | * The qworker deals with the execution of jobs in a reliable way. It is capable of forking jobs, monitoring them and reacting on failure conditions. | ||
| + | * The drunken sailor is a test program that simulates all kinds of misbehaving jobs, i.e. non-terminating jobs. | ||
| + | |||
| + | Smaller tasks such as sending emails etc. can be implemented directly in the ruby library. TODO: Provide a framework of Jobs for this. | ||
| + | |||
| + | An example run looks like this: | ||
| + | |||
| + | $ beanstalkd | ||
| + | |||
| + | First, start beanstalkd. By default it listens on localhost:11300. After the queue runs, we can put a job into the default tube: | ||
| + | |||
| + | $ ruby bin/producer.rb -v -c etc/qproducerrc -a drunken_sailor | ||
| + | |||
| + | This tells the producer to use the config file packaged within the gem (adjust your path) and execute the drunken_sailor algorithm. The output of this command looks like this: | ||
| + | |||
| + | Using this configuration: | ||
| + | --- | ||
| + | BEANSTALK_PORT: "11300" | ||
| + | BEANSTALK_SERVER: localhost | ||
| + | --- | ||
| + | Starting producer, connecting to localhost:11300 | ||
| + | generated work package: [{"uuid":"bc3a925099b9012e0f06549a20077664","type":"drunken_sailor"},{"23":"42","foo":"bar"}] | ||
| + | |||
| + | Then, you can start the qworker in order to run the algorithm: | ||
| + | |||
| + | $ ruby qworker.rb -c ../etc/qworkerrc -v | ||
| + | Using this configuration: | ||
| + | --- | ||
| + | GRACE_TIME: 2 | ||
| + | RESULT_BASEDIR: /var/tmp/qworker | ||
| + | BEANSTALK_PORT: "11300" | ||
| + | BEANSTALK_SERVER: localhost | ||
| + | --- | ||
| + | Starting producer, connecting to localhost:11300 | ||
| + | |||
| + | ### Dequeued job bc3a925099b9012e0f06549a20077664, algorithm drunken_sailor | ||
| + | + result directory: /var/tmp/qworker/bc/bc3a925099b9012e0f06549a20077664 | ||
| + | + input data: 2342foobar | ||
| + | + time left for processing: 2 | ||
| + | + job queue priority: 1337 | ||
| + | Drunken sailor PID: 2403 | ||
| + | Drunken Sailor: Infinite loop. | ||
| + | Algorithm did not terminate within expected timeframe. | ||
| + | Attempting to kill algorithm PID 2403 | ||
| + | Job not found, cannot release it. Ignoring. | ||
| + | |||
| + | ### Dequeued job bc3a925099b9012e0f06549a20077664, algorithm drunken_sailor | ||
| + | + result directory: /var/tmp/qworker/bc/bc3a925099b9012e0f06549a20077664 | ||
| + | + input data: 2342foobar | ||
| + | + time left for processing: 2 | ||
| + | + job queue priority: 1337 | ||
| + | Drunken sailor PID: 2408 | ||
| + | Drunken Sailor: Faulty execution. | ||
| + | Algorithm terminated within expected time constraints. | ||
| + | Algorithm terminated with exitstatus 10 | ||
| + | Job failed, leaving job in queue. | ||
| + | |||
| + | ### Dequeued job bc3a925099b9012e0f06549a20077664, algorithm drunken_sailor | ||
| + | + result directory: /var/tmp/qworker/bc/bc3a925099b9012e0f06549a20077664 | ||
| + | + input data: 2342foobar | ||
| + | + time left for processing: 2 | ||
| + | + job queue priority: 1337 | ||
| + | Drunken sailor PID: 2409 | ||
| + | Drunken Sailor: Normal execution. | ||
| + | Algorithm terminated within expected time constraints. | ||
| + | Algorithm terminated with exitstatus 0 | ||
| + | Job completed, removing from queue. | ||
| + | |||
| + | As you can see, the job was executed three times -- only the last time was successful. Stray processes are getting killed etc, so the behavior of an algorithm does not affect the stability of the system. You might notice that there is a queue priority and expected runtime encoded on the qworker side. Actually, these values are defined by the producer, the code looks like this: | ||
| + | |||
| + | beanstalk.put( | ||
| + | json_payload, # Content of the job. | ||
| + | pri=1337, # Priority of this job. <1024 is considered urgent. | ||
| + | delay=0, # Should the job be delayed for X seconds? | ||
| + | ttr=3 # Seconds until the job will be re-queued. | ||
| + | ) | ||
| + | |||
| + | This is not exposed to the user, but a real producer application should of course implement adequate settings. The result directory ''/var/tmp/qworker/bc/bc3a925099b9012e0f06549a20077664'' is created for each job. | ||
| ====== References ====== | ====== References ====== | ||
jobqueue.1311686588.txt.gz · Last modified: 2012/10/30 10:42 (external edit)
Page Tools
Except where otherwise noted, content on this wiki is licensed under the following license: CC Attribution-Noncommercial-Share Alike 3.0 Unported
