Default TIF, not PDF, for Reliability. #58
Conversation
1. Previously, we used PDFs to store the data, which `zbarimg` can't
interpret reliably. Now, we store the data in multi-page TIF files
which are significantly smaller files that `zbarimg` can reliably
read. The critical change is on line 1281, where I specified
`format="tiff"` and a compression scheme.
2. Additionally, I changed the default output. Now, instead of printing
to stdout by default, we save to a file. This saves the user from
needing to pick the file extension and unintentionally write to
unreliable PDFs. Now, you must use "-" to manually specify outputting
to stdout. The critical change is on line 1110.
With these two changes, I'm able to write and restore 750KB
backups (bz2 compressed to 200KB) in 40 pages. The rest of the
changes are related to renaming variables to be more generic and
fixing the relevant help-text.
I recommend testing these modes with this backup command:
rm qr-backup.log
time python3 ./qr-backup -v --instructions cover ~/test.tar.bz2 \
2>&1 | tee qr-backup.log
And this restore command:
rm qr-restore.log
time python3 ./qr-backup -v --restore ~/test.tar.bz2.qr.tif -o \
~/test.tar.bz2 2>&1 | tee qr-restore.log
NickDaly
changed the title
|
Thanks for the PR. I have a few concerns here.
|
| tmp_pdf = io.BytesIO() | ||
| pages[0].save(tmp_pdf, format="pdf", save_all=True, append_images=pages[1:], resolution=dpi, producer="qr-backup", title=f"qr-backup paper backup of {restore_file} with sha256 {sha256sum} and length {original_len}", creationDate=backup_date, modDate=backup_date) | ||
| tmp_file = io.BytesIO() | ||
| pages[0].save(tmp_file, format="tiff", compression="tiff_lzw", save_all=True, append_images=pages[1:], resolution=dpi, producer="qr-backup", title=f"qr-backup paper backup of {restore_file} with sha256 {sha256sum} and length {original_len}", creationDate=backup_date, modDate=backup_date) |
There was a problem hiding this comment.
Confirmed that the LZW patent has expired, so this should be okay for open-source
|
Thanks for taking a look. I agree that fixing zbarimg is the best fix and I'll look into that. I'll also test with your zbarimg fix to verify whether that addresses the issue (I was testing with Debian Stable's zbarimg v0.23.92). I'll plan on providing an update before November. |
|
If you can show that 100% of QRs can be read from TIFF, and I find there are no big printer issues, I'll accept this PR. I realized there's another major advantage, which is that running on debian currently requires a workaround due to the situation with ghostscript+imagemagick on debian. |
|
I'll upload the logs when I'm home tonight, but I had interesting results after testing with the updated (higher-resolution) zbar:
Overall, the process works fine with the updated zbar, even though it doesn't appear to during the backup process. |
|
The restore process manually converts the pdfs to images before feeding them to zbar to work around the previous bug. An hour is really long! I should fix that, but it should be a separate bug to keep discussion clear. I plan to be very busy this month, so this is really bad timing for me to code on qr-backup, apologies |
|
I think, at this point, we can assume no further changes to qr-backup are needed. The smaller TIF filesize and improved restore speed are nice, but it's probably not worth the complexity for the end-user of switching away from PDF. The logs I got with my file are attached below, we had 1,300 codes to restore and it took 1,360 codes to do it, so some images didn't read correctly, even with the updated 900px resolution. However, that was within the margin for error. I also added some timestamps: the restore-test took a half-hour, while the restore only took a minute. Please let me know if you have any further questions. |
Previously, we used PDFs to store the data, which
zbarimgcan't interpret reliably. Now, we store the data in multi-page TIF files which are significantly smaller files thatzbarimgcan reliably read. The critical change is on line 1281, where I specifiedformat="tiff"and a compression scheme.Additionally, I changed the default output. Now, instead of printing to stdout by default, we save to a file. This saves the user from needing to pick the file extension and unintentionally write to unreliable PDFs. Now, you must use "-" to manually specify outputting to stdout. The critical change is on line 1110.
With these two changes, I'm able to write and restore 750KB backups (bz2 compressed to 200KB) in 40 pages. The rest of the changes are related to renaming variables to be more generic and fixing the relevant help-text.
I recommend testing these modes with this backup command:
And this restore command: