How to upload an older queue_log file

2009-02-20
Imagine you have a queue_log file that uploads to partition P003, and is missing from timestamp 100 to 150; and imagine that old data is available in a file called queue_log.old
  1. Make sure there is no data in the table for partition P003 between timestamps 100 and 150.
    e.g SELECT * FROM queue_log WHERE partition='P003' AND time_id BETWEEN 100 and 150
  2. Stop importing on partition P003
  3. Move queue_log data newer than timestamp 150 to a new partition (eg P003B)
    e.g. UPDATE queue_log SET partition = 'P003B' WHERE partition='P003' AND time_id > 150
  4. Run the qloaderd on your queue_log.old file, so that it will load all data.
    You will have to watch its log file to se when it ends adding lines.
  5. Check that all data has been loaded
    e.g. SELECT count(*) FROM queue_log WHERE partition='P003' AND time_id BETWEEN 100 and 150
  6. Stop the qloader process
  7. Move back newer data from P003B to P003
    e.g UPDATE queue_log SET partition='P003' WHERE partition='P003B'
  8. Point qloaderd to your normal queue_log file, and restart normal importing
After you are done, try running a report to see if old data is actually seen by QueueMetrics.