Recent changes to this wiki:
poll vote (I don't know)
diff --git a/survey.mdwn b/survey.mdwn index 9fe6473..70a8b35 100644 --- a/survey.mdwn +++ b/survey.mdwn @@ -2,4 +2,4 @@ Do you make backups? -[[!poll 0 "yes" 0 "no" 0 "I don't know"]] +[[!poll 0 "yes" 0 "no" 1 "I don't know"]]
diff --git a/survey.mdwn b/survey.mdwn new file mode 100644 index 0000000..9fe6473 --- /dev/null +++ b/survey.mdwn @@ -0,0 +1,5 @@ +# Question 1 + +Do you make backups? + +[[!poll 0 "yes" 0 "no" 0 "I don't know"]]
diff --git a/index.mdwn b/index.mdwn index db38f66..3e2f1b8 100644 --- a/index.mdwn +++ b/index.mdwn @@ -3,6 +3,8 @@ [[!img sticker.png alt="Obnam sticker" link=no class=sticker]] +[[survey]] + Obnam is an **easy, secure backup program.** Backups can be stored on local hard disks, or online via the
Update NEWS for 1.19
diff --git a/NEWS.mdwn b/NEWS.mdwn index 3973813..2061b69 100644 --- a/NEWS.mdwn +++ b/NEWS.mdwn @@ -8,6 +8,35 @@ development, called `green-albatross`. It is **NOT** meant for real use. It is likely to change in incompatible ways without warning. Do not use it unless you're willing to lose your backup. +Version 1.19, released 2016-01-15 +--------------------------------- + +Bug fixes: + +* Backup no longer ignores a closed SSH connection. This means it + won't keep trying to use it, forever. Instead, it crashes and + terminates the backup. + +* The Paramiko SSH implementation, which Obnam uses, changed the + interface to the `prefetch` method in its 1.16 version. Obnam can + now deal with either variant of the method. Found and reported by + Kyle Manna, who provided a patch that Lars Wirzenius rewrote to be + backwards compatible to older versions of Paramiko. + +Improvements to the manual: + +* The manual now has an appendix listing all Obnam errors, with codes + and explanations. This will need to be updated manually from time to + time. + +* The manual now has sections on turning on full debug logging and + reporting problems. + +Improvements to functionality: + +* The output of `obnam generations` now show time zone. Lars Wirzenius + implemented based on suggestion by Limdi. + Version 1.18.2, released 2015-11-15 ---------------------------------
Update NEWS for 1.18.2
diff --git a/NEWS.mdwn b/NEWS.mdwn index 444dac5..3973813 100644 --- a/NEWS.mdwn +++ b/NEWS.mdwn @@ -8,6 +8,20 @@ development, called `green-albatross`. It is **NOT** meant for real use. It is likely to change in incompatible ways without warning. Do not use it unless you're willing to lose your backup. +Version 1.18.2, released 2015-11-15 +--------------------------------- + +Bug fixes: + +* The `--exclude-caches` option now works correctly again. Prevoiusly + it would exclude a cache directory, but would scan through and back + up the contents of the cache directory. As a result, the backup + generation would be much bigger, but have hidden files, not visible + in the output of `obnam ls`. To fix that, remove any generations + made with Obnam 1.18 or 1.18.1. + + This will affect other exclusions as well. + Version 1.18.1, released 2015-11-06 ---------------------------------
Update files for Obnam 1.18.1
diff --git a/NEWS.mdwn b/NEWS.mdwn
index 82c7472..444dac5 100644
--- a/NEWS.mdwn
+++ b/NEWS.mdwn
@@ -8,6 +8,44 @@ development, called `green-albatross`. It is **NOT** meant for real
use. It is likely to change in incompatible ways without warning. Do
not use it unless you're willing to lose your backup.
+Version 1.18.1, released 2015-11-06
+---------------------------------
+
+Bug fixes:
+
+* The `--quiet` option now disables the statistics reports at the end
+ of a backup run.
+
+Version 1.18, released 2015-11-04
+---------------------------------
+
+Bug fixes:
+
+* William Boughton fixed parsing for sftp URLs with IPv6 addresses.
+ Previously, `sftp://[::1]` would be interpreted by Obnam as an
+ address `[` followed by the port `:1]`, but now it is correctly
+ interpreted as the adddress `::1` and no explicit port.
+
+* Ian Campbell fixed a bug in the kdirstat plugin, improving the
+ handling of unknown file types.
+
+* Lars Wirzenius changed the `scan_tree` code to not be recursive, to
+ avoid problems with directory trees that are deeper than Python's
+ call stack limit allows.
+
+Minor changes:
+
+* Lars Wirzenius added support for a multiline progress message during
+ backup. Version 0.24 or newer of `ttystatus` is needed for this, but
+ Obnam will work with an older version by displaying the same
+ single-line progress message as before.
+
+* Ben Boeckel added the `--gnupghome` setting so that Obnam can be
+ configured to use a separate GnuPG (gpg) configuration directory.
+
+* Henri Sivonen improved the compression code to not compress if the
+ result would be larger.
+
Version 1.17, released 2015-09-12
---------------------------------
diff --git a/README.mdwn b/README.mdwn
index 6203274..f06dcd9 100644
--- a/README.mdwn
+++ b/README.mdwn
@@ -144,6 +144,13 @@ requests, and other feedback. I prefer e-mail the mailing list: see
It would be helpful if you can run `make clean check` before submitting
a patch, but it is not strictly required.
+Note that to run the test suite you will need the following installed:
+
+* pep8
+* coverage
+* CoverageTestRunner ( http://liw.fi/coverage-test-runner/ )
+ (python-coverage-test-runner on Debian)
+
Legal stuff
-----------
diff --git a/obnam.1.txt b/obnam.1.txt
index 1361059..a948c45 100644
--- a/obnam.1.txt
+++ b/obnam.1.txt
@@ -14,37 +14,38 @@ SYNOPSIS
[--dump-setting-names] [--encrypt-with=ENCRYPT-WITH]
[--exclude=EXCLUDE] [--exclude-caches] [--exclude-from=FILE]
[--fsck-fix] [--fsck-ignore-chunks] [--fsck-ignore-client=NAME]
- [--fsck-last-generation-only] [--fsck-rm-unused] [--fsck-skip-dirs]
- [--fsck-skip-files] [--fsck-skip-generations]
- [--fsck-skip-per-client-b-trees] [--fsck-skip-shared-b-trees]
- [--fuse-opt=FUSE] [--generate-manpage=TEMPLATE]
- [--generation=GENERATION] [-h] [--help] [--help-all]
+ [--fsck-last-generation-only] [--fsck-rm-unused]
+ [--fsck-skip-checksums] [--fsck-skip-dirs] [--fsck-skip-files]
+ [--fsck-skip-generations] [--fsck-skip-per-client-b-trees]
+ [--fsck-skip-shared-b-trees] [--fuse-opt=FUSE]
+ [--generate-manpage=TEMPLATE] [--generation=GENERATION]
+ [--gnupghome=HOMEDIR] [-h] [--help] [--help-all]
[--idpath-bits=IDPATH-BITS] [--idpath-depth=IDPATH-DEPTH]
[--idpath-skip=IDPATH-SKIP] [--include=INCLUDE] [--keep=KEEP]
[--key-details] [--keyid=KEYID] [--leave-checkpoints]
[--list-config-files] [--lock-timeout=TIMEOUT] [--log=FILE]
- [--log-keep=N] [--log-level=LEVEL] [--log-max=SIZE] [--log-mode=MODE]
+ [--log-keep=N] [--log-level=LEVEL] [--log-max=SIZE] [--log-mode=MODE]
[--lru-size=SIZE] [--memory-dump-interval=SECONDS]
[--no-always-restore-setuid] [--no-default-configs]
- [--no-dump-repo-file-metadata] [--no-exclude-caches] [--no-fsck-fix]
+ [--no-dump-repo-file-metadata] [--no-exclude-caches] [--no-fsck-fix]
[--no-fsck-ignore-chunks] [--no-fsck-last-generation-only]
- [--no-fsck-rm-unused] [--no-fsck-skip-dirs] [--no-fsck-skip-files]
- [--no-fsck-skip-generations] [--no-fsck-skip-per-client-b-trees]
- [--no-fsck-skip-shared-b-trees] [--no-key-details]
- [--no-leave-checkpoints] [--no-one-file-system] [--no-pretend]
- [--no-dry-run] [--no-no-act] [--no-pure-paramiko] [--no-quiet]
- [--no-silent] [--no-small-files-in-btree] [--no-strict-ssh-host-keys]
- [--no-verbose] [--no-weak-random] [--node-size=SIZE]
- [--one-file-system] [--output=FILE] [--pretend] [--dry-run] [--no-act]
- [--pretend-time=TIMESTAMP] [--pure-paramiko] [--quiet] [--silent]
- [-r=URL] [--repository=URL] [--repository-format=FORMAT] [--root=URL]
- [--sftp-delay=SFTP-DELAY] [--small-files-in-btree]
- [--ssh-command=EXECUTABLE] [--ssh-host-keys-check=VALUE]
- [--ssh-key=FILENAME] [--ssh-known-hosts=FILENAME]
- [--strict-ssh-host-keys] [--symmetric-key-bits=BITS]
- [--testing-fail-matching=REGEXP] [--to=TO] [--trace=TRACE]
- [--upload-queue-size=SIZE] [--verbose] [--verify-randomly=N]
- [--version] [--warn-age=AGE] [--weak-random]
+ [--no-fsck-rm-unused] [--no-fsck-skip-checksums] [--no-fsck-skip-dirs]
+ [--no-fsck-skip-files] [--no-fsck-skip-generations]
+ [--no-fsck-skip-per-client-b-trees] [--no-fsck-skip-shared-b-trees]
+ [--no-key-details] [--no-leave-checkpoints] [--no-one-file-system]
+ [--no-pretend] [--no-dry-run] [--no-no-act] [--no-pure-paramiko]
+ [--no-quiet] [--no-silent] [--no-small-files-in-btree]
+ [--no-strict-ssh-host-keys] [--no-verbose] [--no-weak-random]
+ [--node-size=SIZE] [--one-file-system] [--output=FILE] [--pretend]
+ [--dry-run] [--no-act] [--pretend-time=TIMESTAMP] [--pure-paramiko]
+ [--quiet] [--silent] [-r=URL] [--repository=URL]
+ [--repository-format=FORMAT] [--root=URL] [--sftp-delay=SFTP-DELAY]
+ [--small-files-in-btree] [--ssh-command=EXECUTABLE]
+ [--ssh-host-keys-check=VALUE] [--ssh-key=FILENAME]
+ [--ssh-known-hosts=FILENAME] [--strict-ssh-host-keys]
+ [--symmetric-key-bits=BITS] [--testing-fail-matching=REGEXP] [--to=TO]
+ [--trace=TRACE] [--upload-queue-size=SIZE] [--verbose]
+ [--verify-randomly=N] [--version] [--warn-age=AGE] [--weak-random]
obnam [options] _lock
obnam [options] add-key [CLIENT-NAME]...
@@ -73,273 +74,275 @@ SYNOPSIS
obnam [options] verify [DIRECTORY]...
DESCRIPTION
- obnam makes, restores, manipulates, and otherwise deals with backups.
- It can store backups on a local disk or to a server via sftp. Every
- backup generation looks like a fresh snapshot, but is really incremen‐
+ obnam makes, restores, manipulates, and otherwise deals with backups.
+ It can store backups on a local disk or to a server via sftp. Every
+ backup generation looks like a fresh snapshot, but is really incremen‐
tal: the user does not need to worry whether it's a full backup or not.
- Only changed data is backed up, and if a chunk of data is already
+ Only changed data is backed up, and if a chunk of data is already
backed up in another file, that data is re-used.
The place where backed up data is placed is called the backup reposito‐
- ry. A repository may be, for example, a directory on an sftp server,
- or a directory on a USB hard disk. A single repository may contain
- backups from several clients. Their data will intermingle as if they
- were using separate repositories, but if one client backs up a file,
+ ry. A repository may be, for example, a directory on an sftp server,
+ or a directory on a USB hard disk. A single repository may contain
+ backups from several clients. Their data will intermingle as if they
+ were using separate repositories, but if one client backs up a file,
the others may re-use the data.
- obnam command line syntax consists of a command possibly followed by
+ obnam command line syntax consists of a command possibly followed by
arguments. The commands are list below.
· backup makes a new backup. The first time it is run, it makes a
full backup, after that an incremental one.
- · restore is the opposite of a backup. It copies backed up data
- from the backup repository to a target directory. You can re‐
+ · restore is the opposite of a backup. It copies backed up data
+ from the backup repository to a target directory. You can re‐
store everything in a generation, or just selected files.
· clients lists the clients that are backed up to the repository.
- · generations lists every backup generation for a given client,
+ · generations lists every backup generation for a given client,
plus some metadata about the generation.
- · genids lists the identifier for every backup generation for a
- given client. No other information is shown. This can be use‐
+ · genids lists the identifier for every backup generation for a
+ given client. No other information is shown. This can be use‐
ful for scripting.
· ls lists the contents of a given generation, similar to ls -lAR.
- · kdirstat lists the contents of a given generation, in a format
- which is compatible with the kdirstat cache file format, which
+ · kdirstat lists the contents of a given generation, in a format
+ which is compatible with the kdirstat cache file format, which
can then be used to visualise the contents of a backup.
- · verify compares data in the backup with actual user data, and
+ · verify compares data in the backup with actual user data, and
makes sure they are identical. It is most useful to run immedi‐
- ately after a backup, to check that it actually worked. It can
- be run at any time, but if the user data has changed, verifica‐
+ ately after a backup, to check that it actually worked. It can
+ be run at any time, but if the user data has changed, verifica‐
(Diff truncated)
Add link to manual CI version
diff --git a/index.mdwn b/index.mdwn
index 675fae8..db38f66 100644
--- a/index.mdwn
+++ b/index.mdwn
@@ -45,6 +45,8 @@ Documentation
* The full manual (currently a work in progress): available as
[a web page](http://code.liw.fi/obnam/manual/obnam-manual.en.html) and
[PDF](http://code.liw.fi/obnam/manual/obnam-manual.en.pdf).
+ - [CI versions](http://code.liw.fi/obnam/manual/ci/), built from
+ git, but not yet released, are also available.
- The Obnam test suite is also meant to be useful for the more
technical users of Obnam to read:
[web page](http://code.liw.fi/obnam/yarns.html),
Changed pages:
bugs/sftp-get-put-for-transfer-for-speed
Changed by: liw
Commit type: git
Date: 09:30:39 10/25/15
Close bug report
diff --git a/bugs/sftp-get-put-for-transfer-for-speed.mdwn b/bugs/sftp-get-put-for-transfer-for-speed.mdwn index afe0c44..efa37a7 100644 --- a/bugs/sftp-get-put-for-transfer-for-speed.mdwn +++ b/bugs/sftp-get-put-for-transfer-for-speed.mdwn @@ -3,3 +3,5 @@ Would it be faster to use the sftp put and get methods instead of the current open/read/write/close code for transferring files to and from the repository? + +[[done]] probably not worth it --liw
Close bug report
diff --git a/bugs/non-linux-file-types.mdwn b/bugs/non-linux-file-types.mdwn index 32349d7..55ff9fc 100644 --- a/bugs/non-linux-file-types.mdwn +++ b/bugs/non-linux-file-types.mdwn @@ -7,3 +7,6 @@ Obnam should support non-linux file types: --liw + +[[done]] There's no real way I can do this, but I'll be happy to accept good +patches. However, this wishlist has been open for years, without any action. --liw
Changed pages:
bugs/encryption-improvement-suggestion
Changed by: liw
Commit type: git
Date: 09:25:46 10/25/15
Close bug report
diff --git a/bugs/encryption-improvement-suggestion.mdwn b/bugs/encryption-improvement-suggestion.mdwn
index 7a49354..1d8dc9f 100644
--- a/bugs/encryption-improvement-suggestion.mdwn
+++ b/bugs/encryption-improvement-suggestion.mdwn
@@ -35,3 +35,5 @@
<zeri> "64 bit hex number encrypted" << that should have been 64
digits :)
+
+[[done]] I'm not going to override a user's gpg.conf. --liw
Changed pages:
bugs/avoid-stat-for-unchanged-directory
Changed by: liw
Commit type: git
Date: 09:20:26 10/25/15
Close bug report
diff --git a/bugs/avoid-stat-for-unchanged-directory.mdwn b/bugs/avoid-stat-for-unchanged-directory.mdwn index e7900ec..d4a95ba 100644 --- a/bugs/avoid-stat-for-unchanged-directory.mdwn +++ b/bugs/avoid-stat-for-unchanged-directory.mdwn @@ -13,3 +13,5 @@ at grandparent level. Thus, recursing is always necessary. --liw + +[[done]] wishlist has been open for years, no change. --liw
Add FUUG foundation to sponsors list
diff --git a/index.mdwn b/index.mdwn index 48ae0e2..675fae8 100644 --- a/index.mdwn +++ b/index.mdwn @@ -58,18 +58,26 @@ Documentation * [[FAQ]] * [[Development]] stuff -Sponsored by ------------- +Sponsored by Bytemark +--------------------- [[!img bytemark-ff3270c4.svg alt="Bytemark logo" link=no class=bytemark]] -Obnam development is partly sponsored by [Bytemark][] by providing a -[BigV][] virtual machine for benchmarking. Benchmark results are at -<http://benchmark.obnam.org>. +Obnam development is partly sponsored by [Bytemark][] by providing +[BigV][] virtual machines for development and benchmarking. [Bytemark]: https://www.bytemark.co.uk/r/liw [BigV]: http://www.bigv.io/ +Sponsored by the FUUG foundation +-------------------------------- + +The [FUUG foundation] gave a grant in September, 2015, for buying a +fast machine with lots of disk space for development and benchmarking +of Obnam. + +[FUUG foundation]: http://fuug.fi/saatio/ + Links -----
Changed pages:
bugs/commit-progress-reporting
Changed by: liw
Commit type: git
Date: 16:01:27 10/10/15
Mark bug as fixed
diff --git a/bugs/commit-progress-reporting.mdwn b/bugs/commit-progress-reporting.mdwn index 2123a4c..5db52e9 100644 --- a/bugs/commit-progress-reporting.mdwn +++ b/bugs/commit-progress-reporting.mdwn @@ -5,3 +5,9 @@ Improve Obnam's progress reporting during committing. - how much work will there be to do the commit? - how many files to upload? - how many files to move in journal? + + +--- + +[[done]] as this requires changes to larch and I want to not touch it anymore. +A new repository format will drop larch anyway. --liw
Changed pages:
bugs/backs-parent-of-root-if-relative
Changed by: liw
Commit type: git
Date: 15:57:27 10/10/15
Mark bug as fixed
diff --git a/bugs/backs-parent-of-root-if-relative.mdwn b/bugs/backs-parent-of-root-if-relative.mdwn index ff3d412..b771059 100644 --- a/bugs/backs-parent-of-root-if-relative.mdwn +++ b/bugs/backs-parent-of-root-if-relative.mdwn @@ -1,3 +1,5 @@ [[!meta title="Backs up parent of live data root, if relative path used"]] If the live data root directory is given with a relative pathname, Obnam will back up its parent instead. --liw + +[[done]] --liw
Add faq on comparisons with others
diff --git a/faq/compared-to.mdwn b/faq/compared-to.mdwn new file mode 100644 index 0000000..94df452 --- /dev/null +++ b/faq/compared-to.mdwn @@ -0,0 +1,7 @@ +[[!meta title="How does Obnam compare to ...?"]] + +Speaking as Lars, the main Obnam developer: If someone wants to write +(and, preferably, maintain) a comparison between various backup +programs, please do. I'll add a link to it here. However, I do not +want to do that, as it's a lot of work to do well, and it's unfair to +do it badly.
removed
diff --git a/jusdustprodic1986.mdwn b/jusdustprodic1986.mdwn deleted file mode 100644 index 45b983b..0000000 --- a/jusdustprodic1986.mdwn +++ /dev/null @@ -1 +0,0 @@ -hi
removed
diff --git a/wq.mdwn b/wq.mdwn deleted file mode 100644 index 1a4baf5..0000000 --- a/wq.mdwn +++ /dev/null @@ -1 +0,0 @@ -
Changed pages:
jusdustprodic1986
Changed by: jusdustprodic1986
Commit type: web
Date: 22:51:18 09/21/15
diff --git a/jusdustprodic1986.mdwn b/jusdustprodic1986.mdwn new file mode 100644 index 0000000..45b983b --- /dev/null +++ b/jusdustprodic1986.mdwn @@ -0,0 +1 @@ +hi
diff --git a/wq.mdwn b/wq.mdwn new file mode 100644 index 0000000..1a4baf5 --- /dev/null +++ b/wq.mdwn @@ -0,0 +1 @@ +
Add FAQ on checking a repo is encrypted
diff --git a/faq/is-encrypted.mdwn b/faq/is-encrypted.mdwn new file mode 100644 index 0000000..4568168 --- /dev/null +++ b/faq/is-encrypted.mdwn @@ -0,0 +1,7 @@ +[[!meta title="How can I verify that my repository is encrypted?"]] + +If the repository contains the files `clientlist/key` and +`clientlist/userkeys`, it's encrypted. + +The files `key` and `clientlist` are in every toplevel directory in +the repository, except the one called `metadata`.
removed
diff --git a/sad.mdwn b/sad.mdwn deleted file mode 100644 index 5a28ebb..0000000 --- a/sad.mdwn +++ /dev/null @@ -1 +0,0 @@ -[hi](http://gogole.com)
diff --git a/sad.mdwn b/sad.mdwn new file mode 100644 index 0000000..5a28ebb --- /dev/null +++ b/sad.mdwn @@ -0,0 +1 @@ +[hi](http://gogole.com)
Drop mentions of now-removed formats dummy, simple
diff --git a/ondisk.mdwn b/ondisk.mdwn index c99cb92..1e1afca 100644 --- a/ondisk.mdwn +++ b/ondisk.mdwn @@ -208,12 +208,5 @@ better or worse. * [[Format **6**|format-6]] was introduced prior to version 1.0, and is currently the main format. It is intended for real use. -* Format **dummy** provides memory-only storage, and is only used for - testing and development of the repository interface itself. Its code - is its description. -* Format **simple** is an on-disk format that is meant to be as simple - as possible. It was developed to ensure Obnam can support multiple - formats. It may be dropped at any time if it turns out not to be - useful anymore. Its code is its description. * Format [[Green Albatross|format-green-albatross]] is an experimental format, intended to eventually replace format 6.
Fix a few typos.
diff --git a/ondisk.mdwn b/ondisk.mdwn index 424ce36..c99cb92 100644 --- a/ondisk.mdwn +++ b/ondisk.mdwn @@ -71,7 +71,7 @@ Security and privacy The repository design must assume an attacker has at least read-only access to the repository. This means the design should avoid leaking -information via filenames, or other such things. Some data leak in +information via filenames, or other such things. Some data leak is unavoidable: it is, for example, unavoidable that an attacker can keep track of which files were changed when. @@ -160,11 +160,11 @@ collisions. Encryption ---------- -Storage is dump, and so it doesn't encrypt itself, or at least Obnam +Storage is dumb, and so it doesn't encrypt itself, or at least Obnam can't assume it does. Further, storage may be controlled by someone else, such as an online storage provider, and while they may be trusted to store data, they can't be trusted to not leak data. Thus, -Obnam encrypts data before putting it in storage (assuming user +Obnam encrypts data before putting it in storage (assuming the user enables encryption). Encryption is done by combining symmetric and asymmetric (public key) @@ -214,6 +214,6 @@ better or worse. * Format **simple** is an on-disk format that is meant to be as simple as possible. It was developed to ensure Obnam can support multiple formats. It may be dropped at any time if it turns out not to be - useful anymore. Its code is its descrpition. + useful anymore. Its code is its description. * Format [[Green Albatross|format-green-albatross]] is an experimental format, intended to eventually replace format 6.
Update NEWS for 1.17
diff --git a/NEWS.mdwn b/NEWS.mdwn index f8e38fe..82c7472 100644 --- a/NEWS.mdwn +++ b/NEWS.mdwn @@ -8,6 +8,18 @@ development, called `green-albatross`. It is **NOT** meant for real use. It is likely to change in incompatible ways without warning. Do not use it unless you're willing to lose your backup. +Version 1.17, released 2015-09-12 +--------------------------------- + +* Lukáš Poláček added the `--fsck-skip-checksums` setting to + greatly speed up `obnam fsck`. + +* Lars Wirzenius fixed a bug that caused Obnam to sometimes back up + the parent of the backup live data root. In other words, if running + `obnam backup $HOME/important`, then Obnam might backup the whole + of the home directory, instead of just the important subdirectory. + + Version 1.16, released 2015-09-06 ---------------------------------
Add FAQ about missing chunk
diff --git a/faq/forget-missing-chunk.mdwn b/faq/forget-missing-chunk.mdwn new file mode 100644 index 0000000..eef0510 --- /dev/null +++ b/faq/forget-missing-chunk.mdwn @@ -0,0 +1,16 @@ +[[!meta title="Missing chunk error with 'obnam forget'"]] + +Obnam had a bug in which "obnam forget" would remove a chunk before +removing references to the chunk. If the "obnam forget" was +interrupted, the repository would have references to chunks that had +already been removed. This would cause a later run of "obnam forget" +to try to remove a chunk that no longer exists, and that would cause a +crash. + +The crashing was fixed in Obnam version 1.13. As of that version, +"obnam forget" will ignore that a chunk is missing if it's removing +that chunk anyway. + +There remains a problem, unfixed as of Obnam 1.16, where the chunk +still exists in lookup indexes, and "obnam backup" may try to use the +chunk.
Changed pages:
bugs/backs-parent-of-root-if-relative
Changed by: liw
Commit type: git
Date: 16:20:19 09/07/15
Add bug report
diff --git a/bugs/backs-parent-of-root-if-relative.mdwn b/bugs/backs-parent-of-root-if-relative.mdwn new file mode 100644 index 0000000..ff3d412 --- /dev/null +++ b/bugs/backs-parent-of-root-if-relative.mdwn @@ -0,0 +1,3 @@ +[[!meta title="Backs up parent of live data root, if relative path used"]] + +If the live data root directory is given with a relative pathname, Obnam will back up its parent instead. --liw
Update NEWS for 1.16
diff --git a/NEWS.mdwn b/NEWS.mdwn
index 470628b..f8e38fe 100644
--- a/NEWS.mdwn
+++ b/NEWS.mdwn
@@ -8,6 +8,23 @@ development, called `green-albatross`. It is **NOT** meant for real
use. It is likely to change in incompatible ways without warning. Do
not use it unless you're willing to lose your backup.
+Version 1.16, released 2015-09-06
+---------------------------------
+
+* Fixed another typo in a variable name ("netloc"), found by Benedikt
+ Neuffer.
+
+* Fixed a lot of missing module imports, unnecessary module imports,
+ and other minor bugs and style issues found by pylint. Pylint now
+ gets run automatically by the test suite.
+
+ This includes a fix in `exclude_pathnames_plugin.py` to add a missing
+ import and fix variable namaes, by Diane Trout. A similar fix was
+ also contributed by Mesar Hameed.
+
+* Lukáš Poláček fixed an unlocking problem when GnuPG fails during an
+ Obnam run. The lock should now be removed rather than left behind.
+
Version 1.15, released 2015-08-19
---------------------------------
Add bug about sftp error messages
diff --git a/bugs/sftp-errors.mdwn b/bugs/sftp-errors.mdwn new file mode 100644 index 0000000..b79365e --- /dev/null +++ b/bugs/sftp-errors.mdwn @@ -0,0 +1,16 @@ +[[!meta title="Sftp gives no useful error when remote disk is full"]] + +If server's disk is full, Obnam seems to give no useful error message: + + 2015-08-29 16:31:42 ERROR Can't back up + /home/liw/ick/liw.state/extrautils/builds/106/build.log: RD5FA4X: + System error: chunks/1052/1065/3340/212da18852839223: None: + Failure + + 2015-08-29 16:31:42 ERROR OSError(None, 'Failure') + +It looks like Obnam is getting too little info from paramiko, but +maybe that can be fixed. + +Reported by Bazyli Brzóska, +<http://listmaster.pepperfish.net/pipermail/obnam-dev-obnam.org/2015-April/000144.html>
Add bug about FUSE and hardlinks
diff --git a/bugs/fuse-hardlinks.mdwn b/bugs/fuse-hardlinks.mdwn new file mode 100644 index 0000000..8d83aed --- /dev/null +++ b/bugs/fuse-hardlinks.mdwn @@ -0,0 +1,11 @@ +[[!meta title="FUSE doesn't expose hardlinks correctly"]] + +Backing up two hardlinks to the same file, then mounting the backup +repository with the FUSE plugin, results in the hardlinks being shown +as having different inode numbers. They do thus not expose the +hardlinks correctly. + +Reported by Ilya Zonov +(<http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2015-April/003516.html>, +the list archive shows the message wrong, but can be decoded with +base64).
Make NEWS be NEWS again
diff --git a/NEWS.mdwn b/NEWS.mdwn
index 6203274..470628b 100644
--- a/NEWS.mdwn
+++ b/NEWS.mdwn
@@ -1,181 +1,1224 @@
-Obnam, a backup program
-=======================
+Obnam NEWS
+==========
-Obnam is a backup program.
+This file summarizes changes between releases of Obnam.
+NOTE: Obnam has an **EXPERIMENTAL** repository format under
+development, called `green-albatross`. It is **NOT** meant for real
+use. It is likely to change in incompatible ways without warning. Do
+not use it unless you're willing to lose your backup.
-Home page
----------
+Version 1.15, released 2015-08-19
+---------------------------------
-The Obnam home page is at <http://obnam.org/>, see there
-for more information.
+* Fixed a typo in a variable name ("netloc"), found by Dirk.
+Version 1.14, released 2015-08-14
+---------------------------------
-Installation
-------------
+Bug fixes:
-The source tree contains packaging for Debian. Run `debuild -us -uc -i.git` to
-build an installation package.
+* Since 1.9, Obnam has had trouble with sftp URLs for backup roots,
+ particularly for URLs specifying the server's root directory. Dennis
+ Jacobfeuerborn found the reason: the backup plugin was treating URLs
+ as filenames. This should now be fixed.
-On other systems, using the `setup.py` file should work: run
-"python setup.py --help" for advice. If not, please report a bug.
-(I've only tested `setup.py` enough for to build the Debian package.)
+Version 1.13, released 2015-08-01
+---------------------------------
-You need Python 2.6 or 2.7 (Python 3 is not yet supported). You also
-need to install my Python B-tree library, and some of my other
-libraries and tools, which you can get from:
+Bug fixes:
-* <http://liw.fi/larch/>
-* <http://liw.fi/ttystatus/>
-* <http://liw.fi/coverage-test-runner/> (for automatic tests)
-* <http://liw.fi/tracing/>
-* <http://liw.fi/cliapp/>
-* <http://liw.fi/genbackupdata/>
-* <http://liw.fi/summain/>
-* <http://liw.fi/cmdtest/>
-* <http://liw.fi/seivot/> (for benchmarks)
+* Lukáš Poláček found and fixed a repository corruption problem: if
+ `obnam forget` was interrupted at the wrong moment, it might remove
+ a chunk, but not the reference to it. This would case a future run
+ of `obnam forget` to crash due to a missing chunk (error code
+ R43272X). `obnam forget` will now ignore such a missing chunk, since
+ it would've deleted it anyway.
-You also need third party libraries:
+ Lars Wirzenius then changed things so that chunk files are only
+ removed once references to the chunks have been committed.
-* paramiko: <http://www.lag.net/paramiko/>
+Improvements:
-See debian/control for the full set of build dependencies and runtime
-dependencies on a Debian system. (That set actually gets tested. The
-above list is maintained manually and may get out of date from time
-to time.)
+* `obnam forget` now commits changes after each generation it has
+ removed. This means that if the operation is committed, less work is
+ lost. Suggested by Lukáš Poláček, re-implemented by Lars Wirzenius.
-If you want to run obnam from the repository directory (rather than installing
-it), you need to do some setup. Run `./check --unit-tests` for setup and
-to verify with unit tests or `./check --help` to setup without any tests.
-You'll need dev files for python and the Coverage Test Runner python module (on
-Debian, those are the python-dev and python-coverage-test-runner packages).
+Version 1.12, released 2015-07-08
+---------------------------------
-Use
----
+Bug fixes:
-To get a quick help summary of options:
+* Steven Monai reported that using `--one-file-system` would crash,
+ and it turned out to be a missing import.
- ./obnam --help
+* Jan Niggemann reported that `--exclude-caches` no longer worked.
+ This was due to a bug introduced when the option was moved to its
+ own plugin (for cleaner code). The bug was masked by another bug, in
+ the Yarn test suite. Both bugs have now been fixed.
-To make a backup:
+Improvements:
- ./obnam backup --repository /tmp/mybackup $HOME
+* Jan Niggemann translated the Obnam manpage to German. Due to cliapp
+ not supporting other languages than English yet, the manual page
+ lacks option descriptions.
-For more information, see the manual page:
+Version 1.11, released 2015-07-02
+---------------------------------
- man -l obnam.1
+* The 1.10 release failed to correctly include the Green Albatross
+ code, due to a missing line in `setup.py`. This has been fixed.
+Version 1.10, released 2015-07-01
+---------------------------------
-Hacking
--------
+Major bug fixes:
-Obnam source code is stored in git for version control purposes;
-you can get a copy as follows:
+* Lars Wirzenius fixed the `obnam backup` command to lock the whole
+ repository, the same way as `obnam forget` does, when it removes
+ checkpoint generations. This means that during checkpoint removal,
+ no other client can make a backup, which is unfortunate. To avoid
+ that, set `leave-checkpoints = yes` in the configuration. That will
+ prevent `obnam backup` from removing checkpoints.
- git clone git://git.liw.fi/obnam
+Minor new features:
-The 'master' branch is the main development one. Any bug fixes and
-features should be developed in a dedicated branch, which gets merged
-to master when the changes are done and considered good.
+* Lars Wirzenius added the `obnam list-formats` command to list all
+ repository formats.
-To build and run automatic tests:
+* The default value for the `upload-queue-size` setting is now 1024,
+ chosen based on some benchmarking made by Lars Wirzenius to balance
+ speed and memory use.
- ./check
- ./check --unit-tests # unit tests only, no black box tests
- ./check --network # requires ssh access to localhost
+* An EXPERIMENTAL new repository format, `green-albatross`, as been
+ introduced. It is not ready for actual use, and is only added so
+ that its code doesn't diverge far from the main line of development.
-`check` is a wrapper around `python setup.py`, but since using that
-takes several steps, the script makes things easier.
+* Teemu Hukkanen reported that the Synology NAS device returns EACCES
+ instead of ENOENT when user tries to remove a non-existent file.
+ Obnam now copes with either error code.
-You need my CoverageTestRunner to run tests, see above for where to get it. A
-couple of scripts exist to run benchmarks and profiles:
+Minor fixes:
- ./metadata-speed 10000
- ./obnam-benchmark --size=1m/100k --results /tmp/benchmark-results
- viewprof /tmp/benchmark-results/*/*backup-0.prof
- seivots-summary /tmp/benchmark-results/*/*.seivot | less -S
+* `python setup.py build` no longer formats the manual page into plain
+ text. This is now done in `python setup.py docs` instead. The latter
+ is an optional build step, and probably only works on Debian.
-There are two kinds of results: Python profiling output, and `.seivot`
-files.
+* `obnam restore --to=DIR` now requires that the directory `DIR`
+ either doesn't exist, or it is empty when the restore starts. This
+ is to prevent users from restore on top of a running system.
-For the former, `viewprof` is a little helper script I wrote,
-around the Python pstats module.
-You can use your own, or get mine from extrautils
-(<http://liw.fi/extrautils/>). Running the benchmarks under profiling
-makes them a little slower (typically around 10% for me, when I've
-compared), but that's OK: the absolute numbers of the benchmarks are
-less important than the relative ones. It's nice to be able to look at
-the profiler output, if a benchmark is surprisingly slow, without
-having to re-run it.
+Version 1.9, released 2015-03-22
+--------------------------------
-`seivots-summary` is a tool to display summaries of the measurements
-made during a benchmark run. `seivot` is the tool that makes the
-measurements. I typically save a number of benchmark results, so that
-I can see how my changes affect performance over time.
+New features:
(Diff truncated)
Update for Obnam 1.15
diff --git a/NEWS.mdwn b/NEWS.mdwn index 1c2c317..6203274 100644 --- a/NEWS.mdwn +++ b/NEWS.mdwn @@ -1,1219 +1,181 @@ -Obnam NEWS -========== +Obnam, a backup program +======================= -This file summarizes changes between releases of Obnam. +Obnam is a backup program. -NOTE: Obnam has an **EXPERIMENTAL** repository format under -development, called `green-albatross`. It is **NOT** meant for real -use. It is likely to change in incompatible ways without warning. Do -not use it unless you're willing to lose your backup. -Version 1.14, released 2015-08-14 ---------------------------------- +Home page +--------- -Bug fixes: +The Obnam home page is at <http://obnam.org/>, see there +for more information. -* Since 1.9, Obnam has had trouble with sftp URLs for backup roots, - particularly for URLs specifying the server's root directory. Dennis - Jacobfeuerborn found the reason: the backup plugin was treating URLs - as filenames. This should now be fixed. -Version 1.13, released 2015-08-01 ---------------------------------- +Installation +------------ -Bug fixes: +The source tree contains packaging for Debian. Run `debuild -us -uc -i.git` to +build an installation package. -* Lukáš Poláček found and fixed a repository corruption problem: if - `obnam forget` was interrupted at the wrong moment, it might remove - a chunk, but not the reference to it. This would case a future run - of `obnam forget` to crash due to a missing chunk (error code - R43272X). `obnam forget` will now ignore such a missing chunk, since - it would've deleted it anyway. +On other systems, using the `setup.py` file should work: run +"python setup.py --help" for advice. If not, please report a bug. +(I've only tested `setup.py` enough for to build the Debian package.) - Lars Wirzenius then changed things so that chunk files are only - removed once references to the chunks have been committed. +You need Python 2.6 or 2.7 (Python 3 is not yet supported). You also +need to install my Python B-tree library, and some of my other +libraries and tools, which you can get from: -Improvements: +* <http://liw.fi/larch/> +* <http://liw.fi/ttystatus/> +* <http://liw.fi/coverage-test-runner/> (for automatic tests) +* <http://liw.fi/tracing/> +* <http://liw.fi/cliapp/> +* <http://liw.fi/genbackupdata/> +* <http://liw.fi/summain/> +* <http://liw.fi/cmdtest/> +* <http://liw.fi/seivot/> (for benchmarks) -* `obnam forget` now commits changes after each generation it has - removed. This means that if the operation is committed, less work is - lost. Suggested by Lukáš Poláček, re-implemented by Lars Wirzenius. +You also need third party libraries: -Version 1.12, released 2015-07-08 ---------------------------------- +* paramiko: <http://www.lag.net/paramiko/> -Bug fixes: +See debian/control for the full set of build dependencies and runtime +dependencies on a Debian system. (That set actually gets tested. The +above list is maintained manually and may get out of date from time +to time.) -* Steven Monai reported that using `--one-file-system` would crash, - and it turned out to be a missing import. +If you want to run obnam from the repository directory (rather than installing +it), you need to do some setup. Run `./check --unit-tests` for setup and +to verify with unit tests or `./check --help` to setup without any tests. +You'll need dev files for python and the Coverage Test Runner python module (on +Debian, those are the python-dev and python-coverage-test-runner packages). -* Jan Niggemann reported that `--exclude-caches` no longer worked. - This was due to a bug introduced when the option was moved to its - own plugin (for cleaner code). The bug was masked by another bug, in - the Yarn test suite. Both bugs have now been fixed. +Use +--- -Improvements: +To get a quick help summary of options: -* Jan Niggemann translated the Obnam manpage to German. Due to cliapp - not supporting other languages than English yet, the manual page - lacks option descriptions. + ./obnam --help -Version 1.11, released 2015-07-02 ---------------------------------- +To make a backup: -* The 1.10 release failed to correctly include the Green Albatross - code, due to a missing line in `setup.py`. This has been fixed. + ./obnam backup --repository /tmp/mybackup $HOME -Version 1.10, released 2015-07-01 ---------------------------------- +For more information, see the manual page: -Major bug fixes: + man -l obnam.1 -* Lars Wirzenius fixed the `obnam backup` command to lock the whole - repository, the same way as `obnam forget` does, when it removes - checkpoint generations. This means that during checkpoint removal, - no other client can make a backup, which is unfortunate. To avoid - that, set `leave-checkpoints = yes` in the configuration. That will - prevent `obnam backup` from removing checkpoints. -Minor new features: +Hacking +------- -* Lars Wirzenius added the `obnam list-formats` command to list all - repository formats. +Obnam source code is stored in git for version control purposes; +you can get a copy as follows: -* The default value for the `upload-queue-size` setting is now 1024, - chosen based on some benchmarking made by Lars Wirzenius to balance - speed and memory use. + git clone git://git.liw.fi/obnam -* An EXPERIMENTAL new repository format, `green-albatross`, as been - introduced. It is not ready for actual use, and is only added so - that its code doesn't diverge far from the main line of development. +The 'master' branch is the main development one. Any bug fixes and +features should be developed in a dedicated branch, which gets merged +to master when the changes are done and considered good. -* Teemu Hukkanen reported that the Synology NAS device returns EACCES - instead of ENOENT when user tries to remove a non-existent file. - Obnam now copes with either error code. +To build and run automatic tests: -Minor fixes: + ./check + ./check --unit-tests # unit tests only, no black box tests + ./check --network # requires ssh access to localhost -* `python setup.py build` no longer formats the manual page into plain - text. This is now done in `python setup.py docs` instead. The latter - is an optional build step, and probably only works on Debian. +`check` is a wrapper around `python setup.py`, but since using that +takes several steps, the script makes things easier. -* `obnam restore --to=DIR` now requires that the directory `DIR` - either doesn't exist, or it is empty when the restore starts. This - is to prevent users from restore on top of a running system. +You need my CoverageTestRunner to run tests, see above for where to get it. A +couple of scripts exist to run benchmarks and profiles: -Version 1.9, released 2015-03-22 --------------------------------- + ./metadata-speed 10000 + ./obnam-benchmark --size=1m/100k --results /tmp/benchmark-results + viewprof /tmp/benchmark-results/*/*backup-0.prof + seivots-summary /tmp/benchmark-results/*/*.seivot | less -S -New features: +There are two kinds of results: Python profiling output, and `.seivot` +files. -* James Vasile changed Obnam so it can backup an individual file, - instead of an entire directory. +For the former, `viewprof` is a little helper script I wrote, +around the Python pstats module. +You can use your own, or get mine from extrautils +(<http://liw.fi/extrautils/>). Running the benchmarks under profiling +makes them a little slower (typically around 10% for me, when I've +compared), but that's OK: the absolute numbers of the benchmarks are +less important than the relative ones. It's nice to be able to look at +the profiler output, if a benchmark is surprisingly slow, without +having to re-run it. -* James Vasile added the `--include` option to Obnam, allowing one to - include files that would otherwise be excluded (see `--exclude`). +`seivots-summary` is a tool to display summaries of the measurements +made during a benchmark run. `seivot` is the tool that makes the +measurements. I typically save a number of benchmark results, so that +I can see how my changes affect performance over time. (Diff truncated)
two minor orthography/spelling fixes
diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn
index 1a9a03a..de9463f 100644
--- a/format-green-albatross.mdwn
+++ b/format-green-albatross.mdwn
@@ -3,8 +3,8 @@
This page describes the repository format called 'Green Albatross'. It
is a preliminary format, intended to improve Obnam performance over
[[format-6]]. Current development status is PONDERING;
-implementatation has started bug everything on this
-page may and probably will change, and until declared stable, the on-disk
+implementatation has started, but everything on this
+page may and probably will change. Until declared stable, the on-disk
data structured WILL change without warning. Only use this format to
help with its development.
@@ -251,5 +251,5 @@ subdirectory.
}
Again, nothing is updated in-place, everything is updated
-copy-on-write. When a node is updated, it's parent all the way to the
+copy-on-write. When a node is updated, its parent all the way to the
root directory also get updated.
Update development status of green-albatross
diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn index ac02f3f..1a9a03a 100644 --- a/format-green-albatross.mdwn +++ b/format-green-albatross.mdwn @@ -3,8 +3,10 @@ This page describes the repository format called 'Green Albatross'. It is a preliminary format, intended to improve Obnam performance over [[format-6]]. Current development status is PONDERING; -implementatation has not started yet and in fact everything on this -page may and probably will change. +implementatation has started bug everything on this +page may and probably will change, and until declared stable, the on-disk +data structured WILL change without warning. Only use this format to +help with its development. Introduction
Update for Obnam 1.14
diff --git a/NEWS.mdwn b/NEWS.mdwn index 84b654e..1c2c317 100644 --- a/NEWS.mdwn +++ b/NEWS.mdwn @@ -8,6 +8,16 @@ development, called `green-albatross`. It is **NOT** meant for real use. It is likely to change in incompatible ways without warning. Do not use it unless you're willing to lose your backup. +Version 1.14, released 2015-08-14 +--------------------------------- + +Bug fixes: + +* Since 1.9, Obnam has had trouble with sftp URLs for backup roots, + particularly for URLs specifying the server's root directory. Dennis + Jacobfeuerborn found the reason: the backup plugin was treating URLs + as filenames. This should now be fixed. + Version 1.13, released 2015-08-01 ---------------------------------
Add title
diff --git a/faq/website-changes.mdwn b/faq/website-changes.mdwn index 7d38f72..0f9f1ad 100644 --- a/faq/website-changes.mdwn +++ b/faq/website-changes.mdwn @@ -1,3 +1,5 @@ +[[!meta title="How do I suggest changes to the website?"]] + If you want to suggest changes to the website: * E-mail `obnam-dev@lists.pepperfish.net` with the suggestion.
diff --git a/download.mdwn b/download.mdwn
index 29fd355..648f679 100644
--- a/download.mdwn
+++ b/download.mdwn
@@ -17,9 +17,12 @@ Download
<http://download.opensuse.org/repositories/home:/Vayun/>
- spec files:
<https://build.opensuse.org/project/show?project=home%3AVayun>
+
* Centos
-: yum install epel-release
-: yum install obnam fuse python-fuse
+
+ yum install epel-release
+
+ yum install obnam fuse python-fuse
* from version control:
diff --git a/download.mdwn b/download.mdwn
index 10f93e6..29fd355 100644
--- a/download.mdwn
+++ b/download.mdwn
@@ -17,6 +17,10 @@ Download
<http://download.opensuse.org/repositories/home:/Vayun/>
- spec files:
<https://build.opensuse.org/project/show?project=home%3AVayun>
+* Centos
+: yum install epel-release
+: yum install obnam fuse python-fuse
+
* from version control:
`git clone git://git.liw.fi/obnam`
Update files for 1.13 release
diff --git a/NEWS.mdwn b/NEWS.mdwn
index f39d214..84b654e 100644
--- a/NEWS.mdwn
+++ b/NEWS.mdwn
@@ -3,6 +3,32 @@ Obnam NEWS
This file summarizes changes between releases of Obnam.
+NOTE: Obnam has an **EXPERIMENTAL** repository format under
+development, called `green-albatross`. It is **NOT** meant for real
+use. It is likely to change in incompatible ways without warning. Do
+not use it unless you're willing to lose your backup.
+
+Version 1.13, released 2015-08-01
+---------------------------------
+
+Bug fixes:
+
+* Lukáš Poláček found and fixed a repository corruption problem: if
+ `obnam forget` was interrupted at the wrong moment, it might remove
+ a chunk, but not the reference to it. This would case a future run
+ of `obnam forget` to crash due to a missing chunk (error code
+ R43272X). `obnam forget` will now ignore such a missing chunk, since
+ it would've deleted it anyway.
+
+ Lars Wirzenius then changed things so that chunk files are only
+ removed once references to the chunks have been committed.
+
+Improvements:
+
+* `obnam forget` now commits changes after each generation it has
+ removed. This means that if the operation is committed, less work is
+ lost. Suggested by Lukáš Poláček, re-implemented by Lars Wirzenius.
+
Version 1.12, released 2015-07-08
---------------------------------
diff --git a/obnam.1.txt b/obnam.1.txt
index fdf3aca..1361059 100644
--- a/obnam.1.txt
+++ b/obnam.1.txt
@@ -457,7 +457,7 @@ OPTIONS
name of backup repository (can be pathname or supported URL)
--repository-format=FORMAT
- what format to use for new repositories? one of "6", "simple"
+ use FORMAT for new repositories; one of "6", "green-albatross"
--to=TO
where to restore or FUSE mount; for restores, must be empty or
Close bug
diff --git a/bugs/inefficient-metadata.mdwn b/bugs/inefficient-metadata.mdwn index af3af18..9b7d0ae 100644 --- a/bugs/inefficient-metadata.mdwn +++ b/bugs/inefficient-metadata.mdwn @@ -12,4 +12,4 @@ do, does that have an impact on runtime? --liw Workaround: use compression. -Proper fixes are going to require FORMAT GREEN ALBATROSS. [[done] --liw +Proper fixes are going to require FORMAT GREEN ALBATROSS. [[done]] --liw
Close bug
diff --git a/bugs/inefficient-metadata.mdwn b/bugs/inefficient-metadata.mdwn index 0a953bf..af3af18 100644 --- a/bugs/inefficient-metadata.mdwn +++ b/bugs/inefficient-metadata.mdwn @@ -10,3 +10,6 @@ Where is the space used? Can we store it more efficiently and if we do, does that have an impact on runtime? --liw + +Workaround: use compression. +Proper fixes are going to require FORMAT GREEN ALBATROSS. [[done] --liw
Changed pages:
bugs/obnam.org-does-not-have-git-clone-and-patching-instructions
Changed by: liw
Commit type: git
Date: 16:29:43 08/01/15
Close bug
diff --git a/bugs/obnam.org-does-not-have-git-clone-and-patching-instructions.mdwn b/bugs/obnam.org-does-not-have-git-clone-and-patching-instructions.mdwn index beff775..57e9577 100644 --- a/bugs/obnam.org-does-not-have-git-clone-and-patching-instructions.mdwn +++ b/bugs/obnam.org-does-not-have-git-clone-and-patching-instructions.mdwn @@ -1,2 +1,4 @@ The obnam.org website should have instructions for how to clone it, and how to send patches to it. --liw + +[[done]] --liw
Changed pages:
bugs/backup-downloads-too-much-data
Changed by: liw
Commit type: git
Date: 16:28:52 08/01/15
Close bug
diff --git a/bugs/backup-downloads-too-much-data.mdwn b/bugs/backup-downloads-too-much-data.mdwn index 2487feb..612b41c 100644 --- a/bugs/backup-downloads-too-much-data.mdwn +++ b/bugs/backup-downloads-too-much-data.mdwn @@ -19,3 +19,7 @@ This is way too much downloaded data. --liw + +There's been some fixes to this, and Obnam is now much clearer about +the overhead. Also, most fixes are going into FORMAT GREEN ALBATROSS. +[[done]] --liw
Close bug
diff --git a/bugs/salsa-tins.mdwn b/bugs/salsa-tins.mdwn index 840e32f..4525dd4 100644 --- a/bugs/salsa-tins.mdwn +++ b/bugs/salsa-tins.mdwn @@ -36,4 +36,4 @@ modification. --liw -This is implemented in git for FORMAT GREEN ALBATROSS. --liw +This is implemented in git for FORMAT GREEN ALBATROSS. [[done]] --liw
Close bug
diff --git a/bugs/lock-key-in-ram.mdwn b/bugs/lock-key-in-ram.mdwn index 969b0c7..a65396a 100644 --- a/bugs/lock-key-in-ram.mdwn +++ b/bugs/lock-key-in-ram.mdwn @@ -15,3 +15,9 @@ via a file descriptor. --liw + +I'm going to be switching from running gpg for symmetric encryption +in the future, anyway. I'll be doing symmetric encryption in-process +using python-crypt, and that means a lot of the sensitive data is going +to be in Python strings anyway. Locking anything in memory doesn't seem +feasible. [[done]] --liw
Changed pages:
bugs/Flexibilize path handling when doing backups
Changed by: liw
Commit type: git
Date: 16:21:08 08/01/15
Close bug
diff --git a/bugs/Flexibilize_path_handling_when_doing_backups.mdwn b/bugs/Flexibilize_path_handling_when_doing_backups.mdwn index 678f8ee..a363d40 100644 --- a/bugs/Flexibilize_path_handling_when_doing_backups.mdwn +++ b/bugs/Flexibilize_path_handling_when_doing_backups.mdwn @@ -34,3 +34,11 @@ So either the mangling hook is allowed to drop paths entirely. But this feels ve I propose not to change Repository and think of Repository just getting virtual paths from its callers. So instead the functions calling into Repository should be changed. In this case the backup command. I have stopped here. -- Elrond + +If this ever gets implemented, the paths returned by a filename mangling +hook will have to still be absolute, I think. --liw + +However, I don't particularly want this feature. It has the promise of +hours upon hours of debugging over email, when people use it. If someone +makes a clean patch (with tests and everyting), I'll consider it, but +keeping this bug open for years hasn't resulted in that. [[done]] --liw
Changed pages:
bugs/fd-leak
bugs/file-id-and-client-id-collision-corruption
bugs/force-lock currently doesn't work
bugs/forget progress reporting broken
bugs/gpg-passphrase
bugs/larch-journal-processing-progress-reporting
bugs/multirepository
bugs/obnam fsck should do something about unused chunks
bugs/restore-overwrites
bugs/salsa-tins
...
Changed by: liw
Commit type: git
Date: 16:18:46 08/01/15
Update/close bugs
diff --git a/bugs/fd-leak.mdwn b/bugs/fd-leak.mdwn index 1abfbd0..8808b18 100644 --- a/bugs/fd-leak.mdwn +++ b/bugs/fd-leak.mdwn @@ -2,3 +2,5 @@ Ben Kelly reported on August 31, 2012, that he's seeing crashes due to file descriptor leaks. See list mail archive for logs and suggested patches. I have not been able to reproduce this, however. --liw + +There have been no new reports of this for years. [[done]] --liw diff --git a/bugs/file-id-and-client-id-collision-corruption.mdwn b/bugs/file-id-and-client-id-collision-corruption.mdwn index fffe049..b074bf9 100644 --- a/bugs/file-id-and-client-id-collision-corruption.mdwn +++ b/bugs/file-id-and-client-id-collision-corruption.mdwn @@ -2,3 +2,10 @@ Martin Hostettler reported that Obnam doens't properly choose file-ids and client ids: there can be collisions, resulting in badness. See the e-mail at: <http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-May/003018.html> + +--- + +This may still happen, but it's not severe enough that I want to fix it. +Format 6 is in maintenance mode where only serious bugs get attention, and the +rest of my time goes into producing a bette format (which doesn't have this +bug at all). [[done]] --liw diff --git a/bugs/force-lock_currently_doesn__39__t_work.mdwn b/bugs/force-lock_currently_doesn__39__t_work.mdwn index 6c0bf7a..ba30f3f 100644 --- a/bugs/force-lock_currently_doesn__39__t_work.mdwn +++ b/bugs/force-lock_currently_doesn__39__t_work.mdwn @@ -45,3 +45,8 @@ but it's good enough for 1.0, I think, so removing the blocker tag. --liw Making the lock breaking more benign and intelligent is a wishlist. Adding tag. --liw + +--- + +Turns out that this will either mean unencrypted locks or not putting +any useful info in lock files. [[done]] --liw diff --git a/bugs/forget_progress_reporting_broken.mdwn b/bugs/forget_progress_reporting_broken.mdwn index 39b2392..f3ecc26 100644 --- a/bugs/forget_progress_reporting_broken.mdwn +++ b/bugs/forget_progress_reporting_broken.mdwn @@ -8,3 +8,6 @@ forgetting more than two generations, the progress display is stuck at until the very end. It's updated to 64/64 right before finishing. -- weinzwang + + +This should be fixed in git now. [[done]] --liw diff --git a/bugs/gpg-passphrase.mdwn b/bugs/gpg-passphrase.mdwn index e749ae8..308fa82 100644 --- a/bugs/gpg-passphrase.mdwn +++ b/bugs/gpg-passphrase.mdwn @@ -38,3 +38,10 @@ Then call obnam as normal. This will only work on a desktop system where there is someone to notice that a pinentry window has popped up. However it looks like there may be a way to forward the gpg-agent socket over ssh, and thus run obnam with encryption from cron on a headless remote machine (<a href="http://superuser.com/questions/161973/how-can-i-forward-a-gpg-key-via-ssh-agent">See here</a>). You'd probably have to store the private key on the remote machine though.. so not sure how useful that would be. --Scott + + +--- + +I continue to be of the opinion that a setting for the passphrase for +the GPG is pointless. The symmetric key is encrypted by GPG public key +only. [[done]] --liw diff --git a/bugs/larch-journal-processing-progress-reporting.mdwn b/bugs/larch-journal-processing-progress-reporting.mdwn index 8b494ce..dcaea07 100644 --- a/bugs/larch-journal-processing-progress-reporting.mdwn +++ b/bugs/larch-journal-processing-progress-reporting.mdwn @@ -3,3 +3,8 @@ When larch is processing a journal (committing or deleting at startup, committing at end), obnam should be showing useful progress reporting for that. + +--- + +Larch and format 6 in Obnam are in maint mode and won't get this kind +of change anymore, I'm afraid. [[done]] --liw diff --git a/bugs/multirepository.mdwn b/bugs/multirepository.mdwn index 84593e5..72695cf 100644 --- a/bugs/multirepository.mdwn +++ b/bugs/multirepository.mdwn @@ -51,3 +51,8 @@ suggestion: --liw + +--- + +this is handled easily by having a set of config files and giving the +"profile" one to --config. [[done]] --liw diff --git a/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn b/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn index 29464d1..458c5e9 100644 --- a/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn +++ b/bugs/obnam_fsck_should_do_something_about_unused_chunks.mdwn @@ -6,3 +6,5 @@ Currently, obnam fsck reports chunks that are unused: but doesn't do anything about it. There should be an option to remove those unused chunks from the repository. --weinzwang + +fsck can delete unused chunks now. [[done]] --liw diff --git a/bugs/restore-overwrites.mdwn b/bugs/restore-overwrites.mdwn index 5515b39..d367df8 100644 --- a/bugs/restore-overwrites.mdwn +++ b/bugs/restore-overwrites.mdwn @@ -2,3 +2,6 @@ exists. Obnam should require the target directory to not exist, so that restores do not overwrite valuable data. --liw + +Obnam restore now requires a new or non-empty directory to restore to. +[[done]] --liw diff --git a/bugs/salsa-tins.mdwn b/bugs/salsa-tins.mdwn index 450174e..840e32f 100644 --- a/bugs/salsa-tins.mdwn +++ b/bugs/salsa-tins.mdwn @@ -34,3 +34,6 @@ removal of those files is easy: it is the current code without modification. --liw + + +This is implemented in git for FORMAT GREEN ALBATROSS. --liw diff --git a/bugs/should-detect-encryption-automatically-for-read-operations.mdwn b/bugs/should-detect-encryption-automatically-for-read-operations.mdwn index 5404a68..361f8a3 100644 --- a/bugs/should-detect-encryption-automatically-for-read-operations.mdwn +++ b/bugs/should-detect-encryption-automatically-for-read-operations.mdwn @@ -10,3 +10,5 @@ has been set to. Suggested by Damien Couroussé. --liw + +Should be done since a while. [[done]] --liw diff --git a/bugs/use-fsync.mdwn b/bugs/use-fsync.mdwn index faa3dbf..f24f86b 100644 --- a/bugs/use-fsync.mdwn +++ b/bugs/use-fsync.mdwn @@ -7,3 +7,5 @@ backup run. --liw I want this to not have a huge performance impact, though. Learning from the lessons of dpkg, sqlite/liferea/firefox, etc, and using fsync/fdatasync and `sync_file_range` in the right ways is going to be necessary. --liw + +Not doable over sftp, of course. --liw diff --git a/bugs/verify-stricter.mdwn b/bugs/verify-stricter.mdwn index 5d8a0df..3aa4a64 100644 --- a/bugs/verify-stricter.mdwn +++ b/bugs/verify-stricter.mdwn @@ -27,3 +27,7 @@ the list of bugs in http://liw.fi/obnam/bugs/ and would be happy to review and merge a patch for this. --liw + +This is best done by using diff(1) on a FUSE mounted backup-repository, +I think. Don't want to duplicate all basic command line tools in Obnam. +[[done]] --liw diff --git a/bugs/whole-file-checksums.mdwn b/bugs/whole-file-checksums.mdwn index 9a2fe26..e72139b 100644 --- a/bugs/whole-file-checksums.mdwn +++ b/bugs/whole-file-checksums.mdwn @@ -3,3 +3,6 @@ It would be good for Obnam to do the whole-file checksum with a different checksum algorithm, or by using a suitable salt, to catch problems with single-chunk files, e.g., when there is a hash collision. --liw + +This is essentially a duplicate of the many-checksum-algos wishlist +bug. It's coming. [[done]] --liw
Add FAQ on changing website
diff --git a/faq/website-changes.mdwn b/faq/website-changes.mdwn new file mode 100644 index 0000000..7d38f72 --- /dev/null +++ b/faq/website-changes.mdwn @@ -0,0 +1,7 @@ +If you want to suggest changes to the website: + +* E-mail `obnam-dev@lists.pepperfish.net` with the suggestion. +* If you can, see <http://obnam.org/ikiwiki.cgi?do=branchable> and + git clone the site, then send a patch. + +Thanks!
Tweak size, position
diff --git a/index.mdwn b/index.mdwn
index 9c5d916..48ae0e2 100644
--- a/index.mdwn
+++ b/index.mdwn
@@ -61,7 +61,7 @@ Documentation
Sponsored by
------------
-[[!img bytemark-ff3270c4.svg alt="Bytemark logo" link=no size=x50]]
+[[!img bytemark-ff3270c4.svg alt="Bytemark logo" link=no class=bytemark]]
Obnam development is partly sponsored by [Bytemark][] by providing a
[BigV][] virtual machine for benchmarking. Benchmark results are at
diff --git a/local.css b/local.css
index 525ab55..838348e 100644
--- a/local.css
+++ b/local.css
@@ -1,3 +1,7 @@
img.sticker {
float: right;
}
+img.bytemark {
+ float: right;
+ height: 40px;
+}
Tweak size
diff --git a/index.mdwn b/index.mdwn index 89a3710..9c5d916 100644 --- a/index.mdwn +++ b/index.mdwn @@ -61,8 +61,7 @@ Documentation Sponsored by ------------ -[[!img bytemark-ff3270c4.svg - alt="Bytemark logo" link=no class=bytemark size=500x64]] +[[!img bytemark-ff3270c4.svg alt="Bytemark logo" link=no size=x50]] Obnam development is partly sponsored by [Bytemark][] by providing a [BigV][] virtual machine for benchmarking. Benchmark results are at
Use SVG logo
diff --git a/bytemark-ff3270c4.svg b/bytemark-ff3270c4.svg
new file mode 100644
index 0000000..2685cf8
--- /dev/null
+++ b/bytemark-ff3270c4.svg
@@ -0,0 +1,32 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.0//EN" "http://www.w3.org/TR/2001/REC-SVG-20010904/DTD/svg10.dtd">
+<svg version="1.0" id="Layer_1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" x="0px" y="0px"
+ width="512.254px" height="66.05px" viewBox="0 0 512.254 66.05" enable-background="new 0 0 512.254 66.05" xml:space="preserve">
+<g id="Layer_6">
+ <g>
+ <path fill="#004EA4" d="M51.898,50.314h11.93c4.254,0,7.958-1.193,7.958-6.281c0-3.883-2.314-6.014-7.127-6.014H51.894v12.295
+ M51.898,25.715h10.73c4.25,0,6.935-1.199,6.935-5.452c0-3.326-2.78-4.529-6.935-4.529h-10.73V25.715z M31.551,0h36.162
+ c17.392,0,21.09,9.815,21.09,16.56c0,6.667-3.238,10.265-8.144,12.954c5.923,2.035,11.472,6.747,11.472,16.466
+ c0,13.217-11.472,20.066-23.12,20.066h-37.46V0z"/>
+ <polygon fill="#004EA4" points="104.822,41.719 81.601,0 104.078,0 115.086,24.331 126.557,0 148.846,0 125.171,41.719
+ 125.171,66.047 104.822,66.047 "/>
+ <polygon fill="#004EA4" points="157.541,16.938 139.043,16.938 139.043,0 196.388,0 196.388,16.938 177.891,16.938
+ 177.891,66.047 157.541,66.047 "/>
+ <polygon fill="#004EA4" points="194.626,0 249.293,0 249.293,16.938 214.973,16.938 214.973,25.163 246.146,25.163
+ 246.146,40.887 214.973,40.887 214.973,49.121 250.307,49.121 250.307,66.047 194.626,66.047 "/>
+ <polygon fill="#004EA4" points="249.109,0 278.061,0 287.497,38.848 287.682,38.848 297.115,0 326.064,0 326.064,66.047
+ 306.832,66.047 306.832,23.683 306.641,23.683 295.171,66.047 280.001,66.047 268.533,23.683 268.352,23.683 268.352,66.047
+ 249.109,66.047 "/>
+ <path fill="#004EA4" d="M361.58,42.457l-5.912-20.339h-0.189l-6.384,20.339H361.58z M345.576,0h19.89l24.05,66.047H368.43
+ l-2.781-9.434H344.66l-2.959,9.434h-20.444L345.576,0z"/>
+ <path fill="#004EA4" d="M405.146,28.865h10.634c3.792,0,8.979-0.646,8.979-6.565c0-4.163-2.314-6.566-10.088-6.566h-9.524
+ L405.146,28.865 M384.794,0h38.759c11.562,0,21.55,6.389,21.55,18.88c0,6.835-3.15,14.052-9.895,16.554
+ c5.544,2.125,8.965,8.229,9.708,16.463c0.278,3.229,0.372,11.096,2.221,14.15h-20.353c-1.018-3.328-1.38-6.754-1.662-10.175
+ c-0.559-6.29-1.111-12.856-9.156-12.856h-10.82V66.05h-20.352V0z"/>
+ <polygon fill="#004EA4" points="444.636,0 464.988,0 464.988,22.755 465.176,22.755 483.292,0 508.363,0 484.41,25.812
+ 512.254,66.047 486.906,66.047 470.628,40.334 464.988,46.537 464.988,66.047 444.636,66.047 "/>
+ <rect y="8.652" fill="#004EA4" width="20.23" height="19.252"/>
+ <rect y="39.874" fill="#004EA4" width="20.23" height="19.247"/>
+ </g>
+</g>
+</svg>
diff --git a/index.mdwn b/index.mdwn
index 90ecc18..89a3710 100644
--- a/index.mdwn
+++ b/index.mdwn
@@ -61,7 +61,7 @@ Documentation
Sponsored by
------------
-[[!img bytemark-4aa4ed38.png
+[[!img bytemark-ff3270c4.svg
alt="Bytemark logo" link=no class=bytemark size=500x64]]
Obnam development is partly sponsored by [Bytemark][] by providing a
Add link to benchmark results
diff --git a/index.mdwn b/index.mdwn
index c2c4f5c..90ecc18 100644
--- a/index.mdwn
+++ b/index.mdwn
@@ -65,7 +65,8 @@ Sponsored by
alt="Bytemark logo" link=no class=bytemark size=500x64]]
Obnam development is partly sponsored by [Bytemark][] by providing a
-[BigV][] virtual machine for benchmarking.
+[BigV][] virtual machine for benchmarking. Benchmark results are at
+<http://benchmark.obnam.org>.
[Bytemark]: https://www.bytemark.co.uk/r/liw
[BigV]: http://www.bigv.io/
Drop right-floating for Bytemark logo
diff --git a/local.css b/local.css
index e175fec..525ab55 100644
--- a/local.css
+++ b/local.css
@@ -1,6 +1,3 @@
img.sticker {
float: right;
}
-img.bytemark {
- float: right;
-}
Tweak logo
diff --git a/index.mdwn b/index.mdwn index 7b4dfe7..c2c4f5c 100644 --- a/index.mdwn +++ b/index.mdwn @@ -61,7 +61,8 @@ Documentation Sponsored by ------------ -[[!img bytemark-4aa4ed38.png alt="Bytemark logo" link=no class=bytemark]] +[[!img bytemark-4aa4ed38.png + alt="Bytemark logo" link=no class=bytemark size=500x64]] Obnam development is partly sponsored by [Bytemark][] by providing a [BigV][] virtual machine for benchmarking.
Changed pages:
bytemark-4aa4ed38.png
index
local.css
Changed by: liw
Commit type: git
Date: 12:48:23 07/17/15
Add thank you to Bytemark
diff --git a/bytemark-4aa4ed38.png b/bytemark-4aa4ed38.png
new file mode 100644
index 0000000..3b9d4dc
Binary files /dev/null and b/bytemark-4aa4ed38.png differ
diff --git a/index.mdwn b/index.mdwn
index f8fd44a..7b4dfe7 100644
--- a/index.mdwn
+++ b/index.mdwn
@@ -58,6 +58,17 @@ Documentation
* [[FAQ]]
* [[Development]] stuff
+Sponsored by
+------------
+
+[[!img bytemark-4aa4ed38.png alt="Bytemark logo" link=no class=bytemark]]
+
+Obnam development is partly sponsored by [Bytemark][] by providing a
+[BigV][] virtual machine for benchmarking.
+
+[Bytemark]: https://www.bytemark.co.uk/r/liw
+[BigV]: http://www.bigv.io/
+
Links
-----
diff --git a/local.css b/local.css
index 7d61432..e175fec 100644
--- a/local.css
+++ b/local.css
@@ -1,3 +1,6 @@
img.sticker {
float: right;
-}
\ No newline at end of file
+}
+img.bytemark {
+ float: right;
+}
Resize sticker
diff --git a/sticker.png b/sticker.png index 0682c9c..8b343c8 100644 Binary files a/sticker.png and b/sticker.png differ
Float sticker to the right
diff --git a/index.mdwn b/index.mdwn
index 4a05472..f8fd44a 100644
--- a/index.mdwn
+++ b/index.mdwn
@@ -1,7 +1,7 @@
[[!meta title="Obnam - backup program"]]
[[!tag program]]
-[[!img sticker.png alt="Obnam sticker" link=no]]
+[[!img sticker.png alt="Obnam sticker" link=no class=sticker]]
Obnam is an **easy, secure backup program.**
Backups can be stored on local hard disks,
diff --git a/local.css b/local.css
new file mode 100644
index 0000000..7d61432
--- /dev/null
+++ b/local.css
@@ -0,0 +1,3 @@
+img.sticker {
+ float: right;
+}
\ No newline at end of file
Add sticker image to front page
diff --git a/index.mdwn b/index.mdwn index c497793..4a05472 100644 --- a/index.mdwn +++ b/index.mdwn @@ -1,6 +1,8 @@ [[!meta title="Obnam - backup program"]] [[!tag program]] +[[!img sticker.png alt="Obnam sticker" link=no]] + Obnam is an **easy, secure backup program.** Backups can be stored on local hard disks, or online via the diff --git a/sticker.png b/sticker.png new file mode 100644 index 0000000..0682c9c Binary files /dev/null and b/sticker.png differ
Update NEWS, manpage for 1.12
diff --git a/NEWS.mdwn b/NEWS.mdwn
index fc9fa53..f39d214 100644
--- a/NEWS.mdwn
+++ b/NEWS.mdwn
@@ -3,6 +3,25 @@ Obnam NEWS
This file summarizes changes between releases of Obnam.
+Version 1.12, released 2015-07-08
+---------------------------------
+
+Bug fixes:
+
+* Steven Monai reported that using `--one-file-system` would crash,
+ and it turned out to be a missing import.
+
+* Jan Niggemann reported that `--exclude-caches` no longer worked.
+ This was due to a bug introduced when the option was moved to its
+ own plugin (for cleaner code). The bug was masked by another bug, in
+ the Yarn test suite. Both bugs have now been fixed.
+
+Improvements:
+
+* Jan Niggemann translated the Obnam manpage to German. Due to cliapp
+ not supporting other languages than English yet, the manual page
+ lacks option descriptions.
+
Version 1.11, released 2015-07-02
---------------------------------
diff --git a/obnam.1.txt b/obnam.1.txt
index ad26309..fdf3aca 100644
--- a/obnam.1.txt
+++ b/obnam.1.txt
@@ -1,4 +1,4 @@
-OBNAM(1) General Commands Manual OBNAM(1)
+OBNAM(1) General Commands Manual OBNAM(1)
@@ -7,43 +7,43 @@ NAME
SYNOPSIS
obnam [--always-restore-setuid] [--checkpoint=SIZE] [--chunk-size=SIZE]
- [--chunkids-per-group=NUM] [--client-name=CLIENT-NAME]
+ [--chunkids-per-group=NUM] [--client-name=CLIENT-NAME]
[--compress-with=PROGRAM] [--config=FILE] [--crash-limit=COUNTER]
- [--critical-age=AGE] [--deduplicate=MODE] [--dump-config]
- [--dump-memory-profile=METHOD] [--dump-repo-file-metadata]
- [--dump-setting-names] [--encrypt-with=ENCRYPT-WITH]
- [--exclude=EXCLUDE] [--exclude-caches] [--exclude-from=FILE]
- [--fsck-fix] [--fsck-ignore-chunks] [--fsck-ignore-client=NAME]
+ [--critical-age=AGE] [--deduplicate=MODE] [--dump-config]
+ [--dump-memory-profile=METHOD] [--dump-repo-file-metadata]
+ [--dump-setting-names] [--encrypt-with=ENCRYPT-WITH]
+ [--exclude=EXCLUDE] [--exclude-caches] [--exclude-from=FILE]
+ [--fsck-fix] [--fsck-ignore-chunks] [--fsck-ignore-client=NAME]
[--fsck-last-generation-only] [--fsck-rm-unused] [--fsck-skip-dirs]
- [--fsck-skip-files] [--fsck-skip-generations]
- [--fsck-skip-per-client-b-trees] [--fsck-skip-shared-b-trees]
- [--fuse-opt=FUSE] [--generate-manpage=TEMPLATE]
- [--generation=GENERATION] [-h] [--help] [--help-all]
- [--idpath-bits=IDPATH-BITS] [--idpath-depth=IDPATH-DEPTH]
- [--idpath-skip=IDPATH-SKIP] [--include=INCLUDE] [--keep=KEEP]
- [--key-details] [--keyid=KEYID] [--leave-checkpoints]
- [--list-config-files] [--lock-timeout=TIMEOUT] [--log=FILE]
+ [--fsck-skip-files] [--fsck-skip-generations]
+ [--fsck-skip-per-client-b-trees] [--fsck-skip-shared-b-trees]
+ [--fuse-opt=FUSE] [--generate-manpage=TEMPLATE]
+ [--generation=GENERATION] [-h] [--help] [--help-all]
+ [--idpath-bits=IDPATH-BITS] [--idpath-depth=IDPATH-DEPTH]
+ [--idpath-skip=IDPATH-SKIP] [--include=INCLUDE] [--keep=KEEP]
+ [--key-details] [--keyid=KEYID] [--leave-checkpoints]
+ [--list-config-files] [--lock-timeout=TIMEOUT] [--log=FILE]
[--log-keep=N] [--log-level=LEVEL] [--log-max=SIZE] [--log-mode=MODE]
- [--lru-size=SIZE] [--memory-dump-interval=SECONDS]
- [--no-always-restore-setuid] [--no-default-configs]
- [--no-dump-repo-file-metadata] [--no-exclude-caches] [--no-fsck-fix]
- [--no-fsck-ignore-chunks] [--no-fsck-last-generation-only]
- [--no-fsck-rm-unused] [--no-fsck-skip-dirs] [--no-fsck-skip-files]
- [--no-fsck-skip-generations] [--no-fsck-skip-per-client-b-trees]
- [--no-fsck-skip-shared-b-trees] [--no-key-details]
- [--no-leave-checkpoints] [--no-one-file-system] [--no-pretend]
- [--no-dry-run] [--no-no-act] [--no-pure-paramiko] [--no-quiet]
+ [--lru-size=SIZE] [--memory-dump-interval=SECONDS]
+ [--no-always-restore-setuid] [--no-default-configs]
+ [--no-dump-repo-file-metadata] [--no-exclude-caches] [--no-fsck-fix]
+ [--no-fsck-ignore-chunks] [--no-fsck-last-generation-only]
+ [--no-fsck-rm-unused] [--no-fsck-skip-dirs] [--no-fsck-skip-files]
+ [--no-fsck-skip-generations] [--no-fsck-skip-per-client-b-trees]
+ [--no-fsck-skip-shared-b-trees] [--no-key-details]
+ [--no-leave-checkpoints] [--no-one-file-system] [--no-pretend]
+ [--no-dry-run] [--no-no-act] [--no-pure-paramiko] [--no-quiet]
[--no-silent] [--no-small-files-in-btree] [--no-strict-ssh-host-keys]
- [--no-verbose] [--no-weak-random] [--node-size=SIZE]
+ [--no-verbose] [--no-weak-random] [--node-size=SIZE]
[--one-file-system] [--output=FILE] [--pretend] [--dry-run] [--no-act]
- [--pretend-time=TIMESTAMP] [--pure-paramiko] [--quiet] [--silent]
+ [--pretend-time=TIMESTAMP] [--pure-paramiko] [--quiet] [--silent]
[-r=URL] [--repository=URL] [--repository-format=FORMAT] [--root=URL]
- [--sftp-delay=SFTP-DELAY] [--small-files-in-btree]
- [--ssh-command=EXECUTABLE] [--ssh-host-keys-check=VALUE]
- [--ssh-key=FILENAME] [--ssh-known-hosts=FILENAME]
- [--strict-ssh-host-keys] [--symmetric-key-bits=BITS]
+ [--sftp-delay=SFTP-DELAY] [--small-files-in-btree]
+ [--ssh-command=EXECUTABLE] [--ssh-host-keys-check=VALUE]
+ [--ssh-key=FILENAME] [--ssh-known-hosts=FILENAME]
+ [--strict-ssh-host-keys] [--symmetric-key-bits=BITS]
[--testing-fail-matching=REGEXP] [--to=TO] [--trace=TRACE]
- [--upload-queue-size=SIZE] [--verbose] [--verify-randomly=N]
+ [--upload-queue-size=SIZE] [--verbose] [--verify-randomly=N]
[--version] [--warn-age=AGE] [--weak-random]
obnam [options] _lock
@@ -74,16 +74,16 @@ SYNOPSIS
DESCRIPTION
obnam makes, restores, manipulates, and otherwise deals with backups.
- It can store backups on a local disk or to a server via sftp. Every
+ It can store backups on a local disk or to a server via sftp. Every
backup generation looks like a fresh snapshot, but is really incremen‐
tal: the user does not need to worry whether it's a full backup or not.
- Only changed data is backed up, and if a chunk of data is already
+ Only changed data is backed up, and if a chunk of data is already
backed up in another file, that data is re-used.
The place where backed up data is placed is called the backup reposito‐
- ry. A repository may be, for example, a directory on an sftp server,
+ ry. A repository may be, for example, a directory on an sftp server,
or a directory on a USB hard disk. A single repository may contain
- backups from several clients. Their data will intermingle as if they
+ backups from several clients. Their data will intermingle as if they
were using separate repositories, but if one client backs up a file,
the others may re-use the data.
@@ -91,111 +91,111 @@ DESCRIPTION
arguments. The commands are list below.
· backup makes a new backup. The first time it is run, it makes a
- full backup, after that an incremental one.
+ full backup, after that an incremental one.
· restore is the opposite of a backup. It copies backed up data
- from the backup repository to a target directory. You can re‐
- store everything in a generation, or just selected files.
+ from the backup repository to a target directory. You can re‐
+ store everything in a generation, or just selected files.
· clients lists the clients that are backed up to the repository.
· generations lists every backup generation for a given client,
- plus some metadata about the generation.
+ plus some metadata about the generation.
- · genids lists the identifier for every backup generation for a
- given client. No other information is shown. This can be use‐
- ful for scripting.
+ · genids lists the identifier for every backup generation for a
+ given client. No other information is shown. This can be use‐
+ ful for scripting.
· ls lists the contents of a given generation, similar to ls -lAR.
- · kdirstat lists the contents of a given generation, in a format
- which is compatible with the kdirstat cache file format, which
- can then be used to visualise the contents of a backup.
+ · kdirstat lists the contents of a given generation, in a format
+ which is compatible with the kdirstat cache file format, which
+ can then be used to visualise the contents of a backup.
· verify compares data in the backup with actual user data, and
- makes sure they are identical. It is most useful to run immedi‐
- ately after a backup, to check that it actually worked. It can
- be run at any time, but if the user data has changed, verifica‐
- tion fails even though the backup is OK.
+ makes sure they are identical. It is most useful to run immedi‐
+ ately after a backup, to check that it actually worked. It can
+ be run at any time, but if the user data has changed, verifica‐
+ tion fails even though the backup is OK.
· forget removes backup generations that are no longer wanted, so
- that they don't use disk space. Note that after a backup gener‐
- ation is removed the data can't be restored anymore. You can
- either specify the generations to remove by listing them on the
- command line, or use the --keep option to specify a policy for
- what to keep (everything else will be removed).
+ that they don't use disk space. Note that after a backup gener‐
+ ation is removed the data can't be restored anymore. You can
+ either specify the generations to remove by listing them on the
+ command line, or use the --keep option to specify a policy for
+ what to keep (everything else will be removed).
· fsck checks the internal consistency of the backup repository.
- It verifies that all clients, generations, directories, files,
- and all file contents still exists in the backup repository. It
- may take quite a long time to run.
+ It verifies that all clients, generations, directories, files,
+ and all file contents still exists in the backup repository. It
+ may take quite a long time to run.
(Diff truncated)
Update NEWS and README for 1.11
diff --git a/NEWS.mdwn b/NEWS.mdwn
index 79152fb..fc9fa53 100644
--- a/NEWS.mdwn
+++ b/NEWS.mdwn
@@ -3,6 +3,51 @@ Obnam NEWS
This file summarizes changes between releases of Obnam.
+Version 1.11, released 2015-07-02
+---------------------------------
+
+* The 1.10 release failed to correctly include the Green Albatross
+ code, due to a missing line in `setup.py`. This has been fixed.
+
+Version 1.10, released 2015-07-01
+---------------------------------
+
+Major bug fixes:
+
+* Lars Wirzenius fixed the `obnam backup` command to lock the whole
+ repository, the same way as `obnam forget` does, when it removes
+ checkpoint generations. This means that during checkpoint removal,
+ no other client can make a backup, which is unfortunate. To avoid
+ that, set `leave-checkpoints = yes` in the configuration. That will
+ prevent `obnam backup` from removing checkpoints.
+
+Minor new features:
+
+* Lars Wirzenius added the `obnam list-formats` command to list all
+ repository formats.
+
+* The default value for the `upload-queue-size` setting is now 1024,
+ chosen based on some benchmarking made by Lars Wirzenius to balance
+ speed and memory use.
+
+* An EXPERIMENTAL new repository format, `green-albatross`, as been
+ introduced. It is not ready for actual use, and is only added so
+ that its code doesn't diverge far from the main line of development.
+
+* Teemu Hukkanen reported that the Synology NAS device returns EACCES
+ instead of ENOENT when user tries to remove a non-existent file.
+ Obnam now copes with either error code.
+
+Minor fixes:
+
+* `python setup.py build` no longer formats the manual page into plain
+ text. This is now done in `python setup.py docs` instead. The latter
+ is an optional build step, and probably only works on Debian.
+
+* `obnam restore --to=DIR` now requires that the directory `DIR`
+ either doesn't exist, or it is empty when the restore starts. This
+ is to prevent users from restore on top of a running system.
+
Version 1.9, released 2015-03-22
--------------------------------
diff --git a/obnam.1.txt b/obnam.1.txt
index 1dc9fe9..ad26309 100644
--- a/obnam.1.txt
+++ b/obnam.1.txt
@@ -61,6 +61,7 @@ SYNOPSIS
obnam [options] help
obnam [options] help-all
obnam [options] kdirstat [FILE]...
+ obnam [options] list-formats
obnam [options] list-keys
obnam [options] list-toplevels
obnam [options] ls [FILE]...
@@ -74,12 +75,12 @@ SYNOPSIS
DESCRIPTION
obnam makes, restores, manipulates, and otherwise deals with backups.
It can store backups on a local disk or to a server via sftp. Every
- backup generation looks like a fresh snapshot, but is really incremen-
+ backup generation looks like a fresh snapshot, but is really incremen‐
tal: the user does not need to worry whether it's a full backup or not.
Only changed data is backed up, and if a chunk of data is already
backed up in another file, that data is re-used.
- The place where backed up data is placed is called the backup reposito-
+ The place where backed up data is placed is called the backup reposito‐
ry. A repository may be, for example, a directory on an sftp server,
or a directory on a USB hard disk. A single repository may contain
backups from several clients. Their data will intermingle as if they
@@ -89,64 +90,64 @@ DESCRIPTION
obnam command line syntax consists of a command possibly followed by
arguments. The commands are list below.
- o backup makes a new backup. The first time it is run, it makes a
+ · backup makes a new backup. The first time it is run, it makes a
full backup, after that an incremental one.
- o restore is the opposite of a backup. It copies backed up data
- from the backup repository to a target directory. You can re-
+ · restore is the opposite of a backup. It copies backed up data
+ from the backup repository to a target directory. You can re‐
store everything in a generation, or just selected files.
- o clients lists the clients that are backed up to the repository.
+ · clients lists the clients that are backed up to the repository.
- o generations lists every backup generation for a given client,
+ · generations lists every backup generation for a given client,
plus some metadata about the generation.
- o genids lists the identifier for every backup generation for a
- given client. No other information is shown. This can be use-
+ · genids lists the identifier for every backup generation for a
+ given client. No other information is shown. This can be use‐
ful for scripting.
- o ls lists the contents of a given generation, similar to ls -lAR.
+ · ls lists the contents of a given generation, similar to ls -lAR.
- o kdirstat lists the contents of a given generation, in a format
+ · kdirstat lists the contents of a given generation, in a format
which is compatible with the kdirstat cache file format, which
can then be used to visualise the contents of a backup.
- o verify compares data in the backup with actual user data, and
- makes sure they are identical. It is most useful to run immedi-
+ · verify compares data in the backup with actual user data, and
+ makes sure they are identical. It is most useful to run immedi‐
ately after a backup, to check that it actually worked. It can
- be run at any time, but if the user data has changed, verifica-
+ be run at any time, but if the user data has changed, verifica‐
tion fails even though the backup is OK.
- o forget removes backup generations that are no longer wanted, so
- that they don't use disk space. Note that after a backup gener-
+ · forget removes backup generations that are no longer wanted, so
+ that they don't use disk space. Note that after a backup gener‐
ation is removed the data can't be restored anymore. You can
either specify the generations to remove by listing them on the
command line, or use the --keep option to specify a policy for
what to keep (everything else will be removed).
- o fsck checks the internal consistency of the backup repository.
+ · fsck checks the internal consistency of the backup repository.
It verifies that all clients, generations, directories, files,
and all file contents still exists in the backup repository. It
may take quite a long time to run.
- o force-lock removes a lock file for a client in the repository.
+ · force-lock removes a lock file for a client in the repository.
You should only force a lock if you are sure no-one is accessing
that client's data in the repository. A dangling lock might
happen, for example, if obnam loses its network connection to
the backup repository.
- o client-keys lists the encryption key associated with each
+ · client-keys lists the encryption key associated with each
client.
- o list-keys lists the keys that can access the repository, and
+ · list-keys lists the keys that can access the repository, and
which toplevel directories each key can access. Some of the
- toplevel directories are shared between clients, others are spe-
+ toplevel directories are shared between clients, others are spe‐
cific to a client.
- o list-toplevels is like list-keys, but lists toplevels and which
+ · list-toplevels is like list-keys, but lists toplevels and which
keys can access them.
- o add-key adds an encryption key to the repository. By default,
+ · add-key adds an encryption key to the repository. By default,
the key is added only to the shared toplevel directories, but it
can also be added to specific clients: list the names of the
clients on the command line. They key is given with the --keyid
@@ -154,27 +155,27 @@ DESCRIPTION
the key id can access the backup repository (the shared
toplevels plus specified clients).
- o remove-key removes a key from the shared toplevel directories,
+ · remove-key removes a key from the shared toplevel directories,
plus any clients specified on the command line.
- o nagios-last-backup-age is a check that exits with non-zero re-
- turn if a backup age exceeds a certain threshold. It is suit-
+ · nagios-last-backup-age is a check that exits with non-zero re‐
+ turn if a backup age exceeds a certain threshold. It is suit‐
able for use as a check plugin for nagios. Thresholds can be
given the --warn-age and --critical-age options.
- o diff compares two generations and lists files differing between
+ · diff compares two generations and lists files differing between
them. Every output line will be prefixed either by a plus sign
(+) for files that were added, a minus sign (-) for files that
have been removed or an asterisk (*) for files that have
changed. If only one generation ID is specified on the command
- line that generation will be compared with its direct predeces-
+ line that generation will be compared with its direct predeces‐
sor. If two IDs have been specified, all changes between those
two generations will be listed.
- o mount makes the backup repository available via a read-only FUSE
- filesystem. Each backup generation is visible as a subdirecto-
+ · mount makes the backup repository available via a read-only FUSE
+ filesystem. Each backup generation is visible as a subdirecto‐
ry, named after the generation id. This means you can look at
(Diff truncated)
Add note about immutable bags
diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn index 29a2d0c..ac02f3f 100644 --- a/format-green-albatross.mdwn +++ b/format-green-albatross.mdwn @@ -84,6 +84,13 @@ For example, the first and third objects stored in the bag with id Note the use of hexadecimal for the bag id (so all bag identifiers are of the same length), and indexing in decimal, starting from zero. +We will keep bags effectively immutable so that an object id does not +need to change. This means that a bag may contain unused objects. If +it turns out that that's wasting too much data, we can "pack" bags by +replacing the unused blobs with empty values (Python's None) to save +space. This mutates a bag, but only in ways that (correct) users won't +notice. + Client list ===========
Tweak Green Albatross plans
Mainly drop YAML and use custom binary encoding instead.
Mainly drop YAML and use custom binary encoding instead.
diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn
index aab574c..29a2d0c 100644
--- a/format-green-albatross.mdwn
+++ b/format-green-albatross.mdwn
@@ -18,8 +18,8 @@ This repository format takes the following approach:
* Don't use the [Larch](http://liw.fi/larch/) B-tree implementation.
It's intricate code and insufficiently fast. Implement any data
structures directly.
-* With few specific exceptions, repository files are never updated.
- Everything is done copy-on-write. This enables caching. The
+* With few, specific exceptions, repository files are never
+ updated. Everything is done copy-on-write. This enables caching. The
exceptions are root nodes of DAGs, so that it's easy to know where
the DAG starts.
* Data is stored in objects of various types. Objects may be small,
@@ -27,35 +27,41 @@ This repository format takes the following approach:
bags. This includes chunks. Each bag is stored in its own file.
Objects are identified by the bag identifier, plus an index into the
bag.
-* Objects are stored in YAML.
- - Note that this may change if YAML turns out to be too slow, but
- using YAML gets us running quickly.
+* Objects are stored in a suitable serialisation encoding.
+ - This is not Python pickles, since Obnam can't assume those are
+ stable over the lifetime of a backup repository.
- JSON is not used, since JSON is not suitable for storing binary
data, such as filenames, without adding an encoding layer on top
of JSON.
+ - Previously this was specified as YAML, but YAML is not fast to
+ parse. So it's not YAML.
+ - Instead, a simple, custom binary encoding will be used. This
+ will encode ints, booleans, binary strings, or lists or dicts
+ (dict keys being lists). A quick prototype shows this to be easy
+ (worked the first time), and fairly fast even without
+ optimisation.
* Encryption is done exactly like in [[format-6]].
Bags
====
-A bag is also a YAML file, with the following structure:
+A bag is used to store a number of blobs. Bags are identified by a
+random 64-bit integer. This is used as the filename of the bag. The
+bags are stored in a 3-level directory structure, using the top three
+octets of the bag id as the directory names. Thus, a bag whose id in
+hexadecimal is 0x1234567890abcdef would be stored as
+`12/34/56/1234567890abcdef.bag`.
- - content: !!binary |
- ...
+A bag is implemented as a Python `dict` object:
-In other words, a bag is a list of YAML mappings, whose only key is
-"content", and the corresponding value is a blob. If the bag stores
-YAML objects, those are encoded and then stored as binary.
+ {
+ 'bag-id': 'cafef00d',
+ 'blobs': [...],
+ }
-Note that this bag implementation is chosen for implementation
-simplicity. It may change to something more efficient, if need be.
-
-Bags are identified by a random 64-bit integer. This is used as the
-filename of the bag. The bags are stored in a 3-level directory
-structure, using the top three octets of the bag id as the directory
-names. Thus, a bag whose id in hexadecimal is 0x1234567890abcdef would
-be stored as `12/34/56/1234567890abcdef.bag`.
+The `items` field contains the blobs. Each blob may be an arbitrary
+byte string (for chunks), or an encoded Python object.
Object identifiers
@@ -78,26 +84,24 @@ For example, the first and third objects stored in the bag with id
Note the use of hexadecimal for the bag id (so all bag identifiers are
of the same length), and indexing in decimal, starting from zero.
+
Client list
===========
-The client list is stored as `client-list/yaml` in the repository,
-using the following structure:
-
- "client-foo":
- client-id: 123
- encryption-key: "..."
+The client list is stored as `client-list/data.bag` in the repository,
+and each item in the bag has the following structure:
-The client list is not stored in a bag. If the number of clients
-sharing a repository grows sufficiently large, the single file will
-need to be be split up. However this doesn't look like a short-term
-problem.
+ {
+ 'client-name': 'foo',
+ 'client-id': 123,
+ 'encryption-key': None,
+ }
Chunks
======
-Chunks are also stored in bags. The chunk data is just a binary blob.
+Chunks are stored in bags. The chunk data is just a binary blob.
Chunk indexes
@@ -107,10 +111,13 @@ Chunk indexes map a checksum (using the user's chosen algorithm) to a
list of chunk ids, and a chunk id to a list of client ids. The root
object of the indexes is:
- checksums:
- algorithm: SHA-1
- root-id: "1234567890abcdef.1"
- used-by-tree: "1234567890abcdef.0"
+ {
+ 'checksums': {
+ 'algorithm': 'SHA-1',
+ 'root-id': '1234567890abcdef.1',
+ },
+ 'used-by-tree': '1234567890abcdef.0,,
+ }
Checksum to chunk ids
---------------------
@@ -119,32 +126,36 @@ The mapping from a checksum value to a list of chunk ids is done using
a lookup tree that is vaguely similar to a B-tree. The tree contains
index nodes and leaf nodes. Leaf nodes store the actual mappings:
- - checksum: e242ed3bffccdf271b7fbaf34ed72d089537b42f
- chunk-ids:
- - "1234567890abcdef.3"
- - "1234567890abcdef.4"
- - checksum: f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
- chunk-ids:
- - 1234567890abcdef.5
+ {
+ 'e242ed3bffccdf271b7fbaf34ed72d089537b42f': [
+ '1234567890abcdef.3',
+ '1234567890abcdef.4',
+ ],
+ 'f1d2d2f924e986ac86fdf7b36c94bcdf32beec15': [
+ '1234567890abcdef.5',
+ ],
+ ]
-In other words, a leaf node is a list of mappings with the following
-keys:
-
-* `checksum` --- the checksum of the data in each chunk
-* `chunk-ids` --- a list of chunk ids whose contents have the given
- checksum; note that there may be more than one chunk id
+In other words, a leaf node is a `dict` mapping a checksum to a list
+of chunk ids whose content has that checksum. Note that the list may
+be longer than one item.
An index node looks like this:
- - first-checksum: e242ed3bffccdf271b7fbaf34ed72d089537b42f
- last-checksum: f1d2d2f924e986ac86fdf7b36c94bcdf32beec15
- leaf-id: "1234567890abcdef.2"
+ [
+ {
+ 'first-checksum': 'e242ed3bffccdf271b7fbaf34ed72d089537b42f',
+ 'last-checksum': 'f1d2d2f924e986ac86fdf7b36c94bcdf32beec15',
+ 'leaf-id': '1234567890abcdef.2',
+ 'index-id': None,
+ },
+ ]
In other words:
* The index node is a list of mappings, where each mapping corresponds
to an object on the next level in the lookup tree.
-* `first-checksum` is the smallest checksum in the node being
+* `first-checksum` is the smallest checksum in the sub-tree being
referenced.
* `last-checksum` is the largest checksum.
* `leaf-id` is the object id of the leaf node, assuming the next level
@@ -170,16 +181,23 @@ Chunk id to client ids
This tree is similar in structure as the checksum tree, but index nodes
look like this:
- first-client-id: "1234567890abcdef.50"
- last-client-id: "1234567890abcdef.51"
- leaf-id: "1234567890abcdef.49"
+ [
+ {
+ 'first-client-id': '1234567890abcdef.50',
+ 'last-client-id': '1234567890abcdef.51',
+ 'leaf-id': '1234567890abcdef.49',
+ },
+ ]
Leaf nodes look like this:
- - chunk-id: "1234567890abcdef.32"
(Diff truncated)
Remove obsolete dependency
diff --git a/development.mdwn b/development.mdwn index c04f9fc..627f228 100644 --- a/development.mdwn +++ b/development.mdwn @@ -30,7 +30,6 @@ Dependencies: * Python 2 * paramiko * larch: <http://liw.fi/larch/> -* python-lru: <http://liw.fi/lru/> * ttystatus: <http://liw.fi/ttystatus/> * CoverageTestRunner: <http://liw.fi/coverage-test-runner/> (you only need this for running the test suite)
Remove obsolete links
diff --git a/development.mdwn b/development.mdwn index 157a50a..c04f9fc 100644 --- a/development.mdwn +++ b/development.mdwn @@ -24,10 +24,6 @@ Other stuff: * Repository [[locking]] * [[bugs]] - see [code.liw.fi](http://liw.fi/code/) for instructions -* Benchmarks: - * [[specification|obnam/benchmarkspec]] - * [[recent benchmark results|benchmark-summary.txt]] -* [[Obnam release procedure|obnam/release]] Dependencies:
Add missing checkpoint field to GEN
diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn
index 8a93cc8..aab574c 100644
--- a/format-green-albatross.mdwn
+++ b/format-green-albatross.mdwn
@@ -195,6 +195,7 @@ Each generation is a GEN object:
started: "2015-04-06T17.35:41"
ended: "2015-04-06T17.35:42"
+ checkpoint: false
root-dir: "1234567890abcdef.77"
Each directory in the live data is stored in a DIR object. The object
Add missing word space
diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn
index 01f5973..8a93cc8 100644
--- a/format-green-albatross.mdwn
+++ b/format-green-albatross.mdwn
@@ -167,7 +167,7 @@ node get updated.
Chunk id to client ids
----------------------
-This tree is similarin structure as the checksum tree, but index nodes
+This tree is similar in structure as the checksum tree, but index nodes
look like this:
first-client-id: "1234567890abcdef.50"
Add further warning to top of page
diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn index 1eaa622..01f5973 100644 --- a/format-green-albatross.mdwn +++ b/format-green-albatross.mdwn @@ -3,7 +3,8 @@ This page describes the repository format called 'Green Albatross'. It is a preliminary format, intended to improve Obnam performance over [[format-6]]. Current development status is PONDERING; -implementatation has not started yet. +implementatation has not started yet and in fact everything on this +page may and probably will change. Introduction
Fix markup typo
diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn index 898d7ab..1eaa622 100644 --- a/format-green-albatross.mdwn +++ b/format-green-albatross.mdwn @@ -1,4 +1,4 @@ -[[!meta title="Repository format 'Green Albatross']] +[[!meta title="Repository format 'Green Albatross'"]] This page describes the repository format called 'Green Albatross'. It is a preliminary format, intended to improve Obnam performance over
Add more text to the Green Albatross format
diff --git a/format-green-albatross.mdwn b/format-green-albatross.mdwn index c89382f..898d7ab 100644 --- a/format-green-albatross.mdwn +++ b/format-green-albatross.mdwn @@ -4,3 +4,217 @@ This page describes the repository format called 'Green Albatross'. It is a preliminary format, intended to improve Obnam performance over [[format-6]]. Current development status is PONDERING; implementatation has not started yet. + + +Introduction +============ + +For background and the big picture, please read the [[ondisk]] page. +This page only discusses details specific to this format. + +This repository format takes the following approach: + +* Don't use the [Larch](http://liw.fi/larch/) B-tree implementation. + It's intricate code and insufficiently fast. Implement any data + structures directly. +* With few specific exceptions, repository files are never updated. + Everything is done copy-on-write. This enables caching. The + exceptions are root nodes of DAGs, so that it's easy to know where + the DAG starts. +* Data is stored in objects of various types. Objects may be small, + and to avoid having a file per object, objects are collected into + bags. This includes chunks. Each bag is stored in its own file. + Objects are identified by the bag identifier, plus an index into the + bag. +* Objects are stored in YAML. + - Note that this may change if YAML turns out to be too slow, but + using YAML gets us running quickly. + - JSON is not used, since JSON is not suitable for storing binary + data, such as filenames, without adding an encoding layer on top + of JSON. +* Encryption is done exactly like in [[format-6]]. + + +Bags +==== + +A bag is also a YAML file, with the following structure: + + - content: !!binary | + ... + +In other words, a bag is a list of YAML mappings, whose only key is +"content", and the corresponding value is a blob. If the bag stores +YAML objects, those are encoded and then stored as binary. + +Note that this bag implementation is chosen for implementation +simplicity. It may change to something more efficient, if need be. + +Bags are identified by a random 64-bit integer. This is used as the +filename of the bag. The bags are stored in a 3-level directory +structure, using the top three octets of the bag id as the directory +names. Thus, a bag whose id in hexadecimal is 0x1234567890abcdef would +be stored as `12/34/56/1234567890abcdef.bag`. + + +Object identifiers +================== + +Object identifiers are a pair consisting of the bag id and an index +into the bag. Since the identifiers are used frequently, it is +practical to store them as a unit rather than as a pair. Further, they +will be visible to the user (and, especially, the developer), so the +following syntax is used: + + BAGID.INDEX + +For example, the first and third objects stored in the bag with id +0x1234567890abcdef would be: + + 1234567890abcdef.0 + 1234567890abcdef.2 + +Note the use of hexadecimal for the bag id (so all bag identifiers are +of the same length), and indexing in decimal, starting from zero. + +Client list +=========== + +The client list is stored as `client-list/yaml` in the repository, +using the following structure: + + "client-foo": + client-id: 123 + encryption-key: "..." + +The client list is not stored in a bag. If the number of clients +sharing a repository grows sufficiently large, the single file will +need to be be split up. However this doesn't look like a short-term +problem. + + +Chunks +====== + +Chunks are also stored in bags. The chunk data is just a binary blob. + + +Chunk indexes +============= + +Chunk indexes map a checksum (using the user's chosen algorithm) to a +list of chunk ids, and a chunk id to a list of client ids. The root +object of the indexes is: + + checksums: + algorithm: SHA-1 + root-id: "1234567890abcdef.1" + used-by-tree: "1234567890abcdef.0" + +Checksum to chunk ids +--------------------- + +The mapping from a checksum value to a list of chunk ids is done using +a lookup tree that is vaguely similar to a B-tree. The tree contains +index nodes and leaf nodes. Leaf nodes store the actual mappings: + + - checksum: e242ed3bffccdf271b7fbaf34ed72d089537b42f + chunk-ids: + - "1234567890abcdef.3" + - "1234567890abcdef.4" + - checksum: f1d2d2f924e986ac86fdf7b36c94bcdf32beec15 + chunk-ids: + - 1234567890abcdef.5 + +In other words, a leaf node is a list of mappings with the following +keys: + +* `checksum` --- the checksum of the data in each chunk +* `chunk-ids` --- a list of chunk ids whose contents have the given + checksum; note that there may be more than one chunk id + +An index node looks like this: + + - first-checksum: e242ed3bffccdf271b7fbaf34ed72d089537b42f + last-checksum: f1d2d2f924e986ac86fdf7b36c94bcdf32beec15 + leaf-id: "1234567890abcdef.2" + +In other words: + +* The index node is a list of mappings, where each mapping corresponds + to an object on the next level in the lookup tree. +* `first-checksum` is the smallest checksum in the node being + referenced. +* `last-checksum` is the largest checksum. +* `leaf-id` is the object id of the leaf node, assuming the next level + is leaf nodes. +* `index-id` is the object id of the index node, assuming the next + level is index nodes. + +The lookup tree is created in a copy-on-write manner. No node is ever +overwritten, but it may be deleted after it is no longer referenced. +The tree is not kept in balance, to keep the code maintaining as +simple as possible. + +When a new checksum is inserted into the lookup tree, it is added to +an existing leaf node by creating a new leaf node that is a copy of +the old one, and adding the new checksum to the new leaf. A big leaf +node is split in half. Any index nodes on the path to an updated leaf +node get updated. + + +Chunk id to client ids +---------------------- + +This tree is similarin structure as the checksum tree, but index nodes +look like this: + + first-client-id: "1234567890abcdef.50" + last-client-id: "1234567890abcdef.51" + leaf-id: "1234567890abcdef.49" + +Leaf nodes look like this: + + - chunk-id: "1234567890abcdef.32" + used-by-clients-ids: + - 123 + - 456 + +Per-client data +=============== + +The data stored for each client separately starts with a CLIENT +object: + + client-name: "foo" + generations: + - generation-id: "1234567890abcdef.77" + +Each generation is a GEN object: + + started: "2015-04-06T17.35:41" + ended: "2015-04-06T17.35:42" + root-dir: "1234567890abcdef.77" + (Diff truncated)
Fix link
diff --git a/ondisk.mdwn b/ondisk.mdwn index 463486c..424ce36 100644 --- a/ondisk.mdwn +++ b/ondisk.mdwn @@ -206,7 +206,7 @@ the abstract features described above in some concrete way. Depending on the quality of the implementation, the resulting format can work better or worse. -* [|Format **6**|format-6]] was introduced prior to version 1.0, and +* [[Format **6**|format-6]] was introduced prior to version 1.0, and is currently the main format. It is intended for real use. * Format **dummy** provides memory-only storage, and is only used for testing and development of the repository interface itself. Its code
Fix ToC
diff --git a/ondisk.mdwn b/ondisk.mdwn index b262dc1..463486c 100644 --- a/ondisk.mdwn +++ b/ondisk.mdwn @@ -8,7 +8,7 @@ repository data structures are like, and the internal abstraction for handling them. It then links to more detailed descriptions of each different repository format. -[[!toc]] +[[!toc levels=2]] Constraints and assumptions
Fix ToC
diff --git a/ondisk.mdwn b/ondisk.mdwn index 3cacfb6..b262dc1 100644 --- a/ondisk.mdwn +++ b/ondisk.mdwn @@ -8,7 +8,7 @@ repository data structures are like, and the internal abstraction for handling them. It then links to more detailed descriptions of each different repository format. -[[!toc startlevel=2 levels=2]] +[[!toc]] Constraints and assumptions
Changed pages:
format-6
format-green-albatross
ondisk
Changed by: liw
Commit type: git
Date: 10:10:42 04/06/15
Restructure on-disk storage descriptions
diff --git a/format-6.mdwn b/format-6.mdwn new file mode 100644 index 0000000..7acfe0f --- /dev/null +++ b/format-6.mdwn @@ -0,0 +1,403 @@ +[[!meta title="Repository format 6"]] + +NOTE: This was originally written as the sole description of Obnam's +on-disk repository storage. At the time, there was no other repository +format, and so this text doesn't separate that which is generic from +that which is specific. This page needs to be updated to discuss the +specific parts only. For the generic parts, see [[ondisk]]. + +[[!toc startlevel=2 levels=2]] + +Obnam is a backup tool for making **backups to hard disks,** either +local ones or over the network. It does not work with tape drives, +or optical disks. The **backup server is stupid:** it does +not run any Obnam specific code, only ssh (sftp). + +The location where +backups are stored is called the **backup store**, and may be shared between +multiple co-operating clients. The backup store resides on a backup server. The +computers that own the data being backed up are the **clients**. + +Performance is obviously a consideration. It has many aspects: + +* run-time +* RAM +* disk space, both server and client +* disk bandwidth, both server and client +* network bandwidth, between server and client + +To achieve its performance goals, +Obnam uses the B-tree variant devised by Ohad Rodeh[1], also +used by btrfs. This document is describes how it does that. + + +Characteristics of the B-tree +----------------------------- + +The **Rodeh B-trees** are designed for shadowing, i.e., +**copy-on-write updates** +of tree nodes. Nodes are not modified, instead a new copy is created +to replace it. This allows an efficient way to copy a tree and to update +the copy, while keeping the original tree intact. +The two trees will share as many nodes as possible. +Obnam calls the collection of related trees a **forest.** + +The trees use **fixed-length binary strings as keys.** +For each key, a **variable-length binary string is stored as the value.** +The tree consists of interior nodes and leaf +nodes; all values are stored in leaves only. There is a maximum size for +nodes, which in some applications would be a raw disk block, but is +a separate file for Obnam. +Value strings can be as big as a node (minus a small header), but no bigger. + +Lookups and removals from the tree can be done using **ranges:** all keys +within a range are looked up, or removed from the tree. This is an important +optimization and opens up some interesting design possibilities. + + +File data (chunk) storage +------------------------- + +File data is not stored directly in B-trees, but externally. Data +is broken up into **chunks** of suitable size (say, 64 KiB) +and assigned identifiers, +which also act as filenames. Given a chunk id, its contents can be easily +retrieved. + +**Chunks are shared** within and across clients. +This allows clients to avoid +uploading data that another client has already uploaded, +or storing the same data twice. The same mechanism allows Obnam to +efficiently back up files that have moved, or that have been modified, +or both at the same time. +If a chunk of data already exists in the store, Obnam tries hard to avoid +having to back it up again. + +To back up a chunk, Obnam chooses a random, unused 64-bit identifier, +and uploads a file of that name. The next chunk uses the next +identifier, until it hits one that has been used, at which point +it picks a new random one. + +Chunks are managed using reference counts. We will cover these later, +after discussing other details of the store. See +section "Chunks, part 2" below. + +Overview of store forests +------------------------- + +The backup store consists of several forests: + +* client list: map client names to 64-bit identifiers +* chunk list: map chunk identifiers to checksums +* chunk checksums: map chunk checksum to identifiers +* per-client data: backup generations for each client + +Locking +------- + +Locking is done only for writes. +Reads are always allowed, even while write locks exist. +This allows race conditions between readers and writers, +but thanks to copy-on-write updates +those are no different from files getting corrupted or +deleted on the server by hardware failures, +and can be treated the same. + +Each forest is locked separately. +If more than one forest needs to be locked at the same time, +they are locked in an order sorted by the name of the forest, +using the C locale. +If a client fails to lock a particular forest, +it releases all the locks it holds, +and tries again. +This avoids deadlocks, +but allows starving. +The most likely reason for starving is too many clients sharing the +same store, and that needs to be solved by increasing backup server +performance, or having more stores. + +Locks are implemented as files, which are created atomically. +Each lock file contains the name of the host that holds it +(which might not be a backup client), +and the process id on that client, and the time of creating the lock. +If the time is very old, another client may decide to break the lock. +If the backup store is encryped, then also the contents of the +lock file is encrypted. + +To reduce lock congestion, each client attempts to keep a lock for as +short a time as possible. For per-client data, this means keeping the lock +for the duration of the backup. For shared forests, updates can be +spooled: the shared forest is used read-only until the end of the backup +run, or until a checkpoint, and updated then, as quickly as possible. + +Client list forest +---------------- + +The client list forest consists of a single tree. +Its key is consists of a tuple: + +* 128 bits of MD5 checksum of the name of the client +* 64 bits of client id (for hash collision avoidance) + +The client name is typically its host name, but might be anything. +It is up to the sysadmins of the clients to co-operate so that each +client has a unique name. + +It is unlikely that there will be checksum collisions for client names, +but it's easy to not have to worry about that. The client id will be +chosen randomly. + +Chunk list forest +----------------- + +The chunk list forest consists of a single tree. +Its key consists of a 64-bit integer containing a chunk identifier. +The value is the MD5 checksum for the chunk. + +This tree is needed so that it is possible to quickly find the +checksum of a chunk, given its identifier, when removing generations +from the per-client forest. + +Chunk checksum forest +--------------------- + +The chunk checksum forest uses a tuple as a key: + +* 128-bit MD5 checksum of the data in the chunk +* 64-bit chunk id +* 64-bit client id + +The chunk id is used to handle checksum collisions. +While MD5 collisions are not likely in general use, +they are certain for people who research cryptographic hashes, +so Obnam needs to be able to handle them. +When Obnam has a chunk it wants to back up, +it does a range lookup from (md5,0) through (md5,chunkd_id_max), +and then checks if the chunk is identical to any of the chunks +already in the store. +This checking is expensive, but safety may more important than +performance here. (Obnam may provide an option to disable the +data comparison check, and rely only on the hashes.) + +The value is empty. + +Per-client forest +--------------- + +The per-client forest has one tree per backup generation or checkpoint, +and represents a snapshot of the filesystem at a specific time, +or as close to a snapshot as Obnam can make it. Obnam does not +freeze the filesystem while the backup runs, so it cannot guarantee +a true snapshot. + +The forest uses a tuple as a key: + (Diff truncated)
Update files for Obnam 1.9
diff --git a/NEWS.mdwn b/NEWS.mdwn
index 770c607..79152fb 100644
--- a/NEWS.mdwn
+++ b/NEWS.mdwn
@@ -3,6 +3,96 @@ Obnam NEWS
This file summarizes changes between releases of Obnam.
+Version 1.9, released 2015-03-22
+--------------------------------
+
+New features:
+
+* James Vasile changed Obnam so it can backup an individual file,
+ instead of an entire directory.
+
+* James Vasile added the `--include` option to Obnam, allowing one to
+ include files that would otherwise be excluded (see `--exclude`).
+
+* Carlo Teubner changed `obnam fsck` to remove unused chunks, if the
+ `--fsck-fix` or `--fsck-rm-unused` settings are used. He also made
+ it not check for unused chunks when it's useless to do so, because
+ of various `--fsck-skip` settings are used.
+
+* A start of a French translation of the manual by pedrito2.
+
+* Ian Cambell provided a new Obnam command, `obnam kdirstat`, which
+ makes the KDE `k4dirstat` utility be able to show graphically which
+ parts of a backup generation use most space.
+
+* Lars Wirzenius added the `simple` repository format, which is for
+ demonstration only. It is much too simplistic to be used for real.
+
+Minor changes:
+
+* The manual page and `obnam --help` are now clearer that the `--root`
+ setting and command line arguments to `obnam backup` can be SFTP
+ URLs. Thanks to Simone Piccardi for reporting the issue.
+
+* David Fries filled in the displayed file permission mode bits.
+
+* Grammar and typo fixes for the obnam.1 manual page, from Jean
+ Jordaan.
+
+* Tom Chiverton suggested a clarification to the manual page for
+ "obnam mount" to say that each generation is a subdirectory.
+
+* David Fries changed restore to set the group ownership if possible
+ even when not root. No warnings are issued if the attempt fails.
+
+* Jan Niggemann added a little to the German translation of the Obnam
+ manual.
+
+* Lars Wirzenius added the path to the error message about a missing
+ chunk (R43272X).
+
+* Lars Wirzenius made the message at the end of a backup report more
+ statistics about transfers during the backup.
+
+Bug fixes:
+
+* The Obnam SFTP plugin would loop infinitely if it lost the
+ connection to the SSH server while creating a temporary file. Itamar
+ Turner-Trauring provided a fix for this.
+
+* Will Dyson fixed a bug about locking while removing checkpoint
+ generations.
+
+* Michel Alexandre Salim fixed a Python 2.6 compatibility problem in
+ the unit tests (use of `assertRaises` as a context manager).
+
+* Lars Kruse fixed a bug with backing up of overlapping backup roots
+ (e.g., / and /boot), given a test case by Adrien Clerc.
+
+* Thomas Eschenbacher fixed a bug in the format 6 repository code that
+ would crash when there is an obscure problem and a B-tree code can't
+ be found in the tree.
+
+* Tom Chiverton pointed out that the manual page was using "obnam
+ restore" instead of "obnam mount" in an example for "obnam mount".
+
+* The yarn test suite now runs FUSE tests (`obnam mount`) when
+ `fusermount` is available, rather than checking for membership in
+ the group `fuse`. The latter is a Debianism (fixed in Debian
+ `jessie`).
+
+* Thomas Waldmann noticed that `obnam verify` didn't notice that a
+ file had new data, when the modification time was the same. Obnam
+ now notices this.
+
+* Thomas Waldmann fixed many typos and minor bugs in the source code.
+
+* Laurence Perkins reported that the Tahoe-LAFS SFTP server returned
+ some `stat` fields as None. Fixed to change those to be 0 instead.
+
+* Lars Wirzenius fixed double-downloading of chunks during restores.
+
+
Version 1.8, released 2014-05-13
--------------------------------
@@ -365,7 +455,7 @@ Bug fixes:
by ROGERIO DE CARVALHO BASTOS and patched by Peter Valdemar Mørch.
* Setuid and setgid bits are now restored correctly, when restore happens
as root. Reported by Pavel Kokolemin.
-* Obnam now complains if no backup roots have been specfied.
+* Obnam now complains if no backup roots have been specified.
Version 1.2, released 2012-10-06
--------------------------------
diff --git a/README.mdwn b/README.mdwn
index 9bf1751..6203274 100644
--- a/README.mdwn
+++ b/README.mdwn
@@ -7,7 +7,7 @@ Obnam is a backup program.
Home page
---------
-The Obnam home page is at <http://liw.fi/obnam/>, see there
+The Obnam home page is at <http://obnam.org/>, see there
for more information.
@@ -44,6 +44,12 @@ dependencies on a Debian system. (That set actually gets tested. The
above list is maintained manually and may get out of date from time
to time.)
+If you want to run obnam from the repository directory (rather than installing
+it), you need to do some setup. Run `./check --unit-tests` for setup and
+to verify with unit tests or `./check --help` to setup without any tests.
+You'll need dev files for python and the Coverage Test Runner python module (on
+Debian, those are the python-dev and python-coverage-test-runner packages).
+
Use
---
@@ -75,14 +81,14 @@ to master when the changes are done and considered good.
To build and run automatic tests:
./check
- ./check --fast # unit tests only, no black box tests
+ ./check --unit-tests # unit tests only, no black box tests
./check --network # requires ssh access to localhost
`check` is a wrapper around `python setup.py`, but since using that
takes several steps, the script makes things easier.
-You need my CoverageTestRunner to run tests, see above for where to get it.
-A couple of scripts exist to run benchmarks and profiles:
+You need my CoverageTestRunner to run tests, see above for where to get it. A
+couple of scripts exist to run benchmarks and profiles:
./metadata-speed 10000
./obnam-benchmark --size=1m/100k --results /tmp/benchmark-results
@@ -111,6 +117,9 @@ If you make any changes, I welcome patches, either as plain diffs,
`git format-patch --cover-letter` mails, or public repositories I can
merge from.
+I would really appreciate if patches for new features and bug fixes
+would update the test suite and documentation as well.
+
The code layout is roughly like this:
obnamlib/ # all the real code
@@ -125,16 +134,12 @@ Mark statements that should not be included in coverage test with
"# pragma: no cover", if you really, really can't write a test.
without-tests lists modules that have no test modules.
-If you want to make a new release of Obnam, I recommend following
-my release checklist: <http://liw.fi/obnam/release/>.
-
Feedback
--------
-I welcome bug fixes, enhancements, bug reports, suggestions, requests,
-and other feedback. I prefer e-mail the mailing list:
-see <http://vlists.pepperfish.net/cgi-bin/mailman/listinfo/obnam-flarn.net>
-for instructions.
+I welcome bug fixes, enhancements, bug reports, suggestions,
+requests, and other feedback. I prefer e-mail the mailing list: see
+<http://obnam.org/contact/> for details.
It would be helpful if you can run `make clean check` before submitting
a patch, but it is not strictly required.
@@ -149,7 +154,7 @@ so that can change.)
This entire work is covered by the GNU General Public
License, version 3 or later.
-> Copyright 2010-2014 Lars Wirzenius
+> Copyright 2010-2015 Lars Wirzenius
>
> This program is free software: you can redistribute it and/or modify
> it under the terms of the GNU General Public License as published by
diff --git a/obnam.1.txt b/obnam.1.txt
index 084fede..1dc9fe9 100644
--- a/obnam.1.txt
+++ b/obnam.1.txt
@@ -1,4 +1,4 @@
-OBNAM(1) OBNAM(1)
+OBNAM(1) General Commands Manual OBNAM(1)
(Diff truncated)
Update donation page (Paypal closed)
diff --git a/donate.mdwn b/donate.mdwn index 331c838..175e0ec 100644 --- a/donate.mdwn +++ b/donate.mdwn @@ -11,9 +11,10 @@ money or gifts: * Amazon wishlist: http://www.amazon.co.uk/registry/wishlist/IAQ38FB6Y27D/ref=cm_wl_act_vv -* Paypal: liw@liw.fi * Bitcoin: bitcoin:1FUKDBy79cC6iTaiyg5i1JVBCemiVdb5QC +(I've moved to a new country, and I've thus closed my Paypal.) + Obnam does not run on donations: it currently mainly runs on my free time. Donations are not required in any way: you're more than welcome to use it without donating, and in any case I prefer a well-formulated @@ -24,17 +25,17 @@ I can also arrange to do Obnam development for specific features, or support, for a fee, if you or your company needs something that isn't happening otherwise. -Lars Wirzenius, 2014-02-09 +Lars Wirzenius, 2015-03-01 -----BEGIN PGP SIGNATURE----- -Version: GnuPG v1.4.12 (GNU/Linux) - -iQEcBAEBCAAGBQJS943KAAoJEIahkFub41rmf1kH/jefKNjYhfKE7xbzZgR1IzXJ -tJIC0a6YTqe+yXNPHOdsCj348v2TTcDl0l7106XprKDos02YZSDGqPbkqlxwF6DH -cPKw7ZAvosagmbRsW/8dctmDeDi5ejvAzUuSbDFiYDYK5f1OyhozVX8VIMvJXCEj -5OSGXzzYtpBuCNvB594u6E5FD3aE/MCT/5HTdnNhAfPK5Gn1bxEsDizB7DOIWD9D -ItAchaeSqTJJ7ODkFB7goRNd/yJiKYcTW3aK3NmuATavq92DWIwvhGTmK5AwbZS/ -mBvmzSLDpqw1cgAJpFCYK9yMBmvZ53okiBFxwuFSXY9FYI8mc8B9HsWK9szycd8= -=tbcm +Version: GnuPG v1 + +iQEcBAEBCAAGBQJU8vwZAAoJEIahkFub41rmPTQIAIBKvd0CI50O3UZ7yZE5DEIR +M49i01pLbR+B2G1+uf1KFD21fonDGIE/YA98yi12NE8sRwIzMkhR9/nhrkikh9lt +rqEsRjllD1rg8jPfU6Dzs0eo8X2A3JPW5pCtdvw97malPoL0mcbr+u8DhLwl7Pa/ +sAE3XLZaOj/yM7bBnlgsToYrzHQ7sfM+B2RhVzd/h4Nfky3pNJ3V1s5DHiyKPyVf +ZACLcHFqVT0s0fHc/yVjJqSKmKNFOaay1DlbWwr86O//b/kI0NSq0IityQPh8mbl +ly6PYZDAZK97vmiGxwrcWHq2e+UawhSEpbq/+lvy/BBsmBF5JjYwnc/A/XSI04A= +=9aqX -----END PGP SIGNATURE----- </pre> diff --git a/donate.txt b/donate.txt index 69fd91b..d57e8c8 100644 --- a/donate.txt +++ b/donate.txt @@ -8,9 +8,10 @@ money or gifts: * Amazon wishlist: http://www.amazon.co.uk/registry/wishlist/IAQ38FB6Y27D/ref=cm_wl_act_vv -* Paypal: liw@liw.fi * Bitcoin: bitcoin:1FUKDBy79cC6iTaiyg5i1JVBCemiVdb5QC +(I've moved to a new country, and I've thus closed my Paypal.) + Obnam does not run on donations: it currently mainly runs on my free time. Donations are not required in any way: you're more than welcome to use it without donating, and in any case I prefer a well-formulated @@ -21,15 +22,15 @@ I can also arrange to do Obnam development for specific features, or support, for a fee, if you or your company needs something that isn't happening otherwise. -Lars Wirzenius, 2014-02-09 +Lars Wirzenius, 2015-03-01 -----BEGIN PGP SIGNATURE----- -Version: GnuPG v1.4.12 (GNU/Linux) +Version: GnuPG v1 -iQEcBAEBCAAGBQJS943KAAoJEIahkFub41rmf1kH/jefKNjYhfKE7xbzZgR1IzXJ -tJIC0a6YTqe+yXNPHOdsCj348v2TTcDl0l7106XprKDos02YZSDGqPbkqlxwF6DH -cPKw7ZAvosagmbRsW/8dctmDeDi5ejvAzUuSbDFiYDYK5f1OyhozVX8VIMvJXCEj -5OSGXzzYtpBuCNvB594u6E5FD3aE/MCT/5HTdnNhAfPK5Gn1bxEsDizB7DOIWD9D -ItAchaeSqTJJ7ODkFB7goRNd/yJiKYcTW3aK3NmuATavq92DWIwvhGTmK5AwbZS/ -mBvmzSLDpqw1cgAJpFCYK9yMBmvZ53okiBFxwuFSXY9FYI8mc8B9HsWK9szycd8= -=tbcm +iQEcBAEBCAAGBQJU8vwZAAoJEIahkFub41rmPTQIAIBKvd0CI50O3UZ7yZE5DEIR +M49i01pLbR+B2G1+uf1KFD21fonDGIE/YA98yi12NE8sRwIzMkhR9/nhrkikh9lt +rqEsRjllD1rg8jPfU6Dzs0eo8X2A3JPW5pCtdvw97malPoL0mcbr+u8DhLwl7Pa/ +sAE3XLZaOj/yM7bBnlgsToYrzHQ7sfM+B2RhVzd/h4Nfky3pNJ3V1s5DHiyKPyVf +ZACLcHFqVT0s0fHc/yVjJqSKmKNFOaay1DlbWwr86O//b/kI0NSq0IityQPh8mbl +ly6PYZDAZK97vmiGxwrcWHq2e+UawhSEpbq/+lvy/BBsmBF5JjYwnc/A/XSI04A= +=9aqX -----END PGP SIGNATURE-----
Add FAQ on why Obnam is not hosted on Github
diff --git a/faq/why-not-on-github.mdwn b/faq/why-not-on-github.mdwn new file mode 100644 index 0000000..6eaa707 --- /dev/null +++ b/faq/why-not-on-github.mdwn @@ -0,0 +1,16 @@ +[[!meta title="Why is Obnam not hosted on Github or other such service?"]] + +Github is a very nice service in many ways, but it is a proprietary +service, running (at least partly) on non-free software. Obnam is free +software, primarily developed by a free software developer, and +hosting it on a proprietary service would be unethical. + +Services such as Github often provide many things on top of mere code +hosting, such as ticketing systems. The services might be very easy to +use. It is true that Obnam, which is self-hosted by its main +developer, could benefit from some of those things. None of this takes +away the ethical dilemma of using proprietary software. + +For the difference between open source and free software, see for +example +[Software freedom: an introduction](http://yakking.branchable.com/archives/2013/08/).
Add mhy quote
diff --git a/quotes.mdwn b/quotes.mdwn index 1ef7c3d..11f9f2c 100644 --- a/quotes.mdwn +++ b/quotes.mdwn @@ -2,5 +2,7 @@ Some quotes about Obnam, from its users. +"Even mhy can't find much bad to say about it." + "Not only does it back up, but it restores too!" says relieved user Kinnison. 2014-01-18 IRC.
Start a quotes page
diff --git a/quotes.mdwn b/quotes.mdwn new file mode 100644 index 0000000..1ef7c3d --- /dev/null +++ b/quotes.mdwn @@ -0,0 +1,6 @@ +[[!meta title="Quotes about Obnam"]] + +Some quotes about Obnam, from its users. + +"Not only does it back up, but it restores too!" says relieved user Kinnison. +2014-01-18 IRC.
Changed pages:
bugs/no-can-two-files-as-roots
Changed by: liw
Commit type: git
Date: 09:04:41 11/30/14
Add new bug report
diff --git a/bugs/no-can-two-files-as-roots.mdwn b/bugs/no-can-two-files-as-roots.mdwn new file mode 100644 index 0000000..456824c --- /dev/null +++ b/bugs/no-can-two-files-as-roots.mdwn @@ -0,0 +1,7 @@ +[[!meta title="Can't backup two plain files as roots"]] + +Only the last file ends up in the repository. + +See +<http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003103.html> +for details.
Add FAQ about the list archive mess
diff --git a/faq/list-archive-mess.mdwn b/faq/list-archive-mess.mdwn new file mode 100644 index 0000000..6169191 --- /dev/null +++ b/faq/list-archive-mess.mdwn @@ -0,0 +1,17 @@ +[[!meta title="Why are messages in the list archive encrypted?"]] + +The Obnam mailing list archives sometimes have messages that look like +they're encrypted. They aren't, it's actually just a base64 encoding, +so it's easy enough to undo. The base64 encoding in the archives +happens due to some bug in the mailing list software, which the people +hosting the lists haven't been able to fix. It will hopefully be fixed +in a future upgrade to a new version of Mailman. + +Meanwhile, [Gmane]() provides a much nicer interface to the mailing +list than Mailman's archives, and doesn't suffer from this problem. +Gmane is highly recommended. See the +<http://dir.gmane.org/gmane.comp.sysutils.backup.obnam> group there +for the Obnam support list. + +[Gmane]: http://gmane.org/ +
Link to gmane
diff --git a/contact.mdwn b/contact.mdwn index 3db43f8..bfe7643 100644 --- a/contact.mdwn +++ b/contact.mdwn @@ -14,11 +14,18 @@ E-mail to know about. If you're using Obnam, you probably should be subscribed to this list. It is very low volume. +The support list is available on the [Gmane]() site, +at <http://dir.gmane.org/gmane.comp.sysutils.backup.obnam>. The +official archives sometimes have messages in base64 encoding, and +their UI isn't great, so the Gmane archives are often a better way to +read the archives. + Please don't e-mail developers directly, unless you know them. [support list]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org [development list]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org [announcement list]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-announce-obnam.org +[Gmane]: http://gmane.org/ IRC ===
creating tag page tag/person
diff --git a/tag/person.mdwn b/tag/person.mdwn new file mode 100644 index 0000000..d3b7dfa --- /dev/null +++ b/tag/person.mdwn @@ -0,0 +1,4 @@ +[[!meta title="pages tagged person"]] + +[[!inline pages="tagged(person)" actions="no" archive="yes" +feedshow=10]]
Update FAQ on missing nodes
diff --git a/faq/missing-node.mdwn b/faq/missing-node.mdwn index a5adc46..808b4f2 100644 --- a/faq/missing-node.mdwn +++ b/faq/missing-node.mdwn @@ -14,16 +14,24 @@ depends on the exact files that get modified during a backup generation. Worse, `obnam fsck` isn't able to fix the corruption, though it would find it. -As a result, a corrupted repository may be unrecoverable. If the -corruption is only in a "per-client B-tree", recovery can happen by -removing that B-tree, but that results in all backup generations for -that client being lost. However, file content will remain in the -repository, so a fresh full backup should be reasonably fast. The -per-client B-tree is a directory that has a large number as its name. -The directory needs to be removed manully, not using Obnam. - -If the corruption is anywhere else in the repository, no recovery is -useful. +As a result, a corrupted repository may be unrecoverable. Sometimes +you can recover at least some of the repository, with some loss of +data: + +* If the corruption is only in a "per-client B-tree", recovery can + happen by removing or renaming that B-tree, but that results in all + backup generations for that client being lost. However, file content + will remain in the repository, so a fresh full backup should be + reasonably fast. The per-client B-tree is a directory that has a + large number as its name. The directory needs to be removed manully, + not using Obnam. Instead of removing it, you can also rename it. + +* If the corruption is in the `chunksums` or `chunklist` B-trees, you + can delete or rename them both. The next backup run will create new + ones. This will reduce the possibility of de-duplication, but + otherwise everything should work. + +If the corruption is any other B-tree, no recovery is possible. When in doubt, avoid using a repository that has ever been used with Obnam prior to version 1.6.1, or Larch prior to version 1.20131130.
Add an FAQ on the old repo corruption problem
diff --git a/faq/missing-node.mdwn b/faq/missing-node.mdwn new file mode 100644 index 0000000..a5adc46 --- /dev/null +++ b/faq/missing-node.mdwn @@ -0,0 +1,34 @@ +[[!meta title="Missing node or KeyError problems"]] + +Obnam prior to version 1.6.1, and [Larch][] prior to version +1.20131130, had serious, but somewhat rare bugs that could result in +an error that said "Missing node" or "KeyError". These bugs would +cause a corruption in the backup repository by either causing a file +to be deleted, when it shouldn't have been, or the reference count for +a B-tree node to be wrong. As a result, Obnam would become unable to +make a backup or, in some cases, to restore everything from a backup +(some files could be restored). + +The corruption is not necessarily noticeable on all backup runs. It +depends on the exact files that get modified during a backup +generation. Worse, `obnam fsck` isn't able to fix the corruption, +though it would find it. + +As a result, a corrupted repository may be unrecoverable. If the +corruption is only in a "per-client B-tree", recovery can happen by +removing that B-tree, but that results in all backup generations for +that client being lost. However, file content will remain in the +repository, so a fresh full backup should be reasonably fast. The +per-client B-tree is a directory that has a large number as its name. +The directory needs to be removed manully, not using Obnam. + +If the corruption is anywhere else in the repository, no recovery is +useful. + +When in doubt, avoid using a repository that has ever been used with +Obnam prior to version 1.6.1, or Larch prior to version 1.20131130. +Instead, start over with a new, empty repository. + +Lars apologises for this. + +[Larch]: http://liw.fi/larch/
Expand the contact page
diff --git a/contact.mdwn b/contact.mdwn index 21cc24c..3db43f8 100644 --- a/contact.mdwn +++ b/contact.mdwn @@ -1,16 +1,57 @@ -Contact -------- - -* [Mailing list](http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org) -* `#obnam` on `irc.oftc.net` for IRC discussions about Obnam - - the mailing list is strongly preferred for asking help with Obnam - - Lars rarely has time to discuss anything serious on IRC - - patches to the mailing list only; post them as patches, not pointers to - code on some website, though an additional pointer to a branch - that can be merged is fine +[[!meta title="Contact"]] + +E-mail +====== + +* If you have any questions about Obnam, or any problems when using + it, e-mail the [support list][]. +* If you have made improvements to Obnam, or want to discuss ways in + which you can implement them, please e-mail the + [development list][]. +* If unsure which list is appropriate, use the support list. +* The [announcement list][] is used by Obnam developers to announce + new releases, or to alert Obnam users about anything that they need + to know about. If you're using Obnam, you probably should be + subscribed to this list. It is very low volume. + +Please don't e-mail developers directly, unless you know them. + +[support list]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org +[development list]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-dev-obnam.org +[announcement list]: http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-announce-obnam.org + +IRC +=== + +The `#obnam` channel on the `irc.oftc.net` IRC network exists for +realtime discussions about Obnam. Note that the [support list][] is +strongly preferred over the IRC channel for support issues, but you're +welcome to hang out and chat if you want. + +Note that the main Obnam developer, Lars Wirzenius, rarely has time +for serious discussions on IRC, especially during working hours. + +Patches and fixes and new features +================================== + +If you want to make changes to Obnam, or this website, please clone +the git repository and send patches to the [development list][], +preferably using `git format-patch --cover-letter`. + +For learning to use git, start with these pages: + +* [Basics of version control systems][] on Yakking. +* [Git tutorial][] + +[Basics of version control systems]: http://yakking.branchable.com/posts/basics-of-cs/ +[Git tutorial]: http://www.git-scm.com/book/en/Getting-Started + + +More information +================ + * [Lars's blog posts about Obnam](http://blog.liw.fi/tag/obnam/) - and [btree](http://blog.liw.fi/tag/btree/). + and [larch](http://blog.liw.fi/tag/larch/). * Also, [Lars's private journal entries](http://liw.fi/obnam/journal-dump/) about obnam and btree. * [Sharon Kimble's blog posts about Obnam](http://www.sharons.org.uk/tag/obnam/) -* If all else fails, [[/contact]] gives contact information for Lars.
Fix Jan's last name (sorry)
diff --git a/index.mdwn b/index.mdwn
index 3a80f0f..c497793 100644
--- a/index.mdwn
+++ b/index.mdwn
@@ -48,7 +48,7 @@ Documentation
[web page](http://code.liw.fi/obnam/yarns.html),
[PDF](http://code.liw.fi/obnam/yarns.pdf).
- See also manuals built automatically from version control
- by Jan Niggeman:
+ by Jan Niggemann:
<http://hz6.de/obnam-downloads/>
* [[README]] (updated at release time)
* [[NEWS]] (updated at release time)
Add link to Jan's CI-built manuals
diff --git a/index.mdwn b/index.mdwn
index 725aa5b..3a80f0f 100644
--- a/index.mdwn
+++ b/index.mdwn
@@ -47,6 +47,9 @@ Documentation
technical users of Obnam to read:
[web page](http://code.liw.fi/obnam/yarns.html),
[PDF](http://code.liw.fi/obnam/yarns.pdf).
+ - See also manuals built automatically from version control
+ by Jan Niggeman:
+ <http://hz6.de/obnam-downloads/>
* [[README]] (updated at release time)
* [[NEWS]] (updated at release time)
* [[obnam manual page|obnam.1.txt]]
Keep old table
diff --git a/faq/tuning.mdwn b/faq/tuning.mdwn
index 1ac5eba..bb2b5de 100644
--- a/faq/tuning.mdwn
+++ b/faq/tuning.mdwn
@@ -78,29 +78,6 @@ Default values as fetched from `__init__.py` are: l=256, q=128.
| l=default, q=1000 | 02m19s - 02m20s | ~266M | 2 |
| l=default, q=10000 | 02m23s - 02m35s | ~266M | 2 |
-foo.
-
-[[!table data="""
-Conditions | Time | Memory |Number of runs
-default values | 22m21s - 24m51s | ~260M | 2
-l=10000, q=default | 13m45s - 15m03s | ~332M | 2
-l=default, q=250 | 08m23s - 10m29s | ~278M | 5
-l=default, q=350 | 02m42s - 02m49s | 272-276M | 2
-l=default, q=400 | 02m13s - 02m18s | 268-272M | 3
-l=default, q=500 | 02m10s - 02m16s | 267-272M | 3
-l=default, q=512 | 02m13s - 02m14s | 265-269M | 2
-l=512, q=512 | 01m55s - 02m06s | 322-326M | 3
-l=768, q=512 | 01m55s - 01m58s | 397-418M | 3
-l=1024, q=512 | 01m53s - 01m55s | 403-418M | 3
-l=2048, q=512 | 01m55s - 01m59s | 408-410M | 3
-l=4096, q=512 | ~01m58s | ~419M | 1
-l=default, q=600 | 02m14s - 02m26s | 269-272M | 4
-l=default, q=750 | 02m13s - 02m15s | 266-272M | 2
-l=default, q=1000 | 02m19s - 02m20s | ~266M | 2
-l=default, q=10000 | 02m23s - 02m35s | ~266M | 2
-"""]]
-
-
So in my configuration, when nearly no data changes between backups,
`--lru-size=1024 --upload-queue-size=512` is at least 11x faster than the
default configuration.
Try native table
diff --git a/faq/tuning.mdwn b/faq/tuning.mdwn
index bb2b5de..1ac5eba 100644
--- a/faq/tuning.mdwn
+++ b/faq/tuning.mdwn
@@ -78,6 +78,29 @@ Default values as fetched from `__init__.py` are: l=256, q=128.
| l=default, q=1000 | 02m19s - 02m20s | ~266M | 2 |
| l=default, q=10000 | 02m23s - 02m35s | ~266M | 2 |
+foo.
+
+[[!table data="""
+Conditions | Time | Memory |Number of runs
+default values | 22m21s - 24m51s | ~260M | 2
+l=10000, q=default | 13m45s - 15m03s | ~332M | 2
+l=default, q=250 | 08m23s - 10m29s | ~278M | 5
+l=default, q=350 | 02m42s - 02m49s | 272-276M | 2
+l=default, q=400 | 02m13s - 02m18s | 268-272M | 3
+l=default, q=500 | 02m10s - 02m16s | 267-272M | 3
+l=default, q=512 | 02m13s - 02m14s | 265-269M | 2
+l=512, q=512 | 01m55s - 02m06s | 322-326M | 3
+l=768, q=512 | 01m55s - 01m58s | 397-418M | 3
+l=1024, q=512 | 01m53s - 01m55s | 403-418M | 3
+l=2048, q=512 | 01m55s - 01m59s | 408-410M | 3
+l=4096, q=512 | ~01m58s | ~419M | 1
+l=default, q=600 | 02m14s - 02m26s | 269-272M | 4
+l=default, q=750 | 02m13s - 02m15s | 266-272M | 2
+l=default, q=1000 | 02m19s - 02m20s | ~266M | 2
+l=default, q=10000 | 02m23s - 02m35s | ~266M | 2
+"""]]
+
+
So in my configuration, when nearly no data changes between backups,
`--lru-size=1024 --upload-queue-size=512` is at least 11x faster than the
default configuration.
Markup tweak
diff --git a/faq/tuning.mdwn b/faq/tuning.mdwn index 5fd463a..bb2b5de 100644 --- a/faq/tuning.mdwn +++ b/faq/tuning.mdwn @@ -15,7 +15,7 @@ the e-mails: [first] and [second]. Measurements ============ -Tuning lru-size and/or upload-queue-size can make a significant +Tuning `lru-size` and/or `upload-queue-size` can make a significant difference in performance. Here follows some test results for this situation:
Add section title to FAQ entry
diff --git a/faq/tuning.mdwn b/faq/tuning.mdwn index 7fc4d03..5fd463a 100644 --- a/faq/tuning.mdwn +++ b/faq/tuning.mdwn @@ -1,5 +1,8 @@ [[!meta title="Performance tuning"]] +Introduction +============ + Obnam has a number of options for performance tuning. See the [[manual page|obnam.1.txt]] for all the details. Below is an adapted excerpt of e-mails written by Lionel Bouton of how to test various @@ -9,6 +12,9 @@ the e-mails: [first] and [second]. [first]: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003086.html [second]: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003090.html +Measurements +============ + Tuning lru-size and/or upload-queue-size can make a significant difference in performance. @@ -76,6 +82,9 @@ So in my configuration, when nearly no data changes between backups, `--lru-size=1024 --upload-queue-size=512` is at least 11x faster than the default configuration. +Discussion +========== + `--upload-queue-size` seems to have the greatest effect without any adverse effect (memory usage remains at the same level). For a little extra boost with a small impact on memory usage, I can
Order FAQ entries by page title
diff --git a/faq.mdwn b/faq.mdwn index fbef8d8..75dc8e0 100644 --- a/faq.mdwn +++ b/faq.mdwn @@ -1,3 +1,3 @@ [[!meta title="Obnam FAQ"]] -[[!inline pages="page(faq/*)" archive=yes template=titlepage]] +[[!inline pages="page(faq/*)" archive=yes template=titlepage sort=meta(title)]]
Add FAQ on performance tuning
diff --git a/faq/tuning.mdwn b/faq/tuning.mdwn new file mode 100644 index 0000000..7fc4d03 --- /dev/null +++ b/faq/tuning.mdwn @@ -0,0 +1,100 @@ +[[!meta title="Performance tuning"]] + +Obnam has a number of options for performance tuning. See the +[[manual page|obnam.1.txt]] for all the details. Below is an adapted +excerpt of e-mails written by Lionel Bouton of how to test various +values to find a good set for your situation. See the list archive for +the e-mails: [first] and [second]. + +[first]: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003086.html +[second]: http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-June/003090.html + +Tuning lru-size and/or upload-queue-size can make a significant +difference in performance. + +Here follows some test results for this situation: + +- Data to backup stored on a btrfs volume on SSD: ~155000 files, 3.66GiB. + +- Local system: 64 bit Linux, Python 2.7.5, OpenSSH 6.6p1 with hpn patches. + +- Local CPU: Intel(R) Core(TM) i5-3317U CPU @ 1.70GHz (mostly idle). + +- Remote system: 64bit Linux, OpenSSH 6.6p1 with hpn patches, + repository data on ext4 on standard SATA 7200rpm disk, large memory + (everything should fit in memory, only writes should hit disk). + +- Very minimal changes in the data backed up during tests so + successive backups only check for differences and backup transfers + nearly no content. + +- Backup over WiFi (~1ms RTT, max speed over sftp ~3MB/s). + +I use this command line without any configuration file: + + obnam -r sftp://obnam@SERVER/~/repo --compress-with=deflate \ + --client-name=CLIENT backup DIR + +During testing I added `--lru-size <l> --upload-queue-size <q>` with +different `<l>` and `<q>` values. + +The resident memory of the Obnam process grows steadily (probably +filling caches) until it hits a pretty stable ceiling (cache full or +nothing new to put in cache) during the backup. It raises again +rapidly at the very end (during commits/unlock/...). The value +reported below is obtained either through the RES column reported by +the htop utility or the RSS column reported by "ps aux" and is the max +witnessed near the end of the backup. + +Each combination was tested at least twice unless it was considered +not interesting after the first run. Timing seems consistent enough +given the systems involved (the system hosting the repository is often +busy) and memory usage is very consistent across runs. + +Default values as fetched from `__init__.py` are: l=256, q=128. + + | Conditions | Time | Memory |Number of runs | + +--------------------+-----------------+------------+----------------+ + | default values | 22m21s - 24m51s | ~260M | 2 | + | l=10000, q=default | 13m45s - 15m03s | ~332M | 2 | + | l=default, q=250 | 08m23s - 10m29s | ~278M | 5 | + | l=default, q=350 | 02m42s - 02m49s | 272-276M | 2 | + | l=default, q=400 | 02m13s - 02m18s | 268-272M | 3 | + | l=default, q=500 | 02m10s - 02m16s | 267-272M | 3 | + | l=default, q=512 | 02m13s - 02m14s | 265-269M | 2 | + | l=512, q=512 | 01m55s - 02m06s | 322-326M | 3 | + | l=768, q=512 | 01m55s - 01m58s | 397-418M | 3 | + | l=1024, q=512 | 01m53s - 01m55s | 403-418M | 3 | + | l=2048, q=512 | 01m55s - 01m59s | 408-410M | 3 | + | l=4096, q=512 | ~01m58s | ~419M | 1 | + | l=default, q=600 | 02m14s - 02m26s | 269-272M | 4 | + | l=default, q=750 | 02m13s - 02m15s | 266-272M | 2 | + | l=default, q=1000 | 02m19s - 02m20s | ~266M | 2 | + | l=default, q=10000 | 02m23s - 02m35s | ~266M | 2 | + +So in my configuration, when nearly no data changes between backups, +`--lru-size=1024 --upload-queue-size=512` is at least 11x faster than the +default configuration. + +`--upload-queue-size` seems to have the greatest effect without any +adverse effect (memory usage remains at the same level). +For a little extra boost with a small impact on memory usage, I can +increase `--lru-size` to 1024. + +Note that Obnam was using 100% of the CPU for most of the time in the +fastest configuration, replacing `--verbose` with `--quiet` didn't +change the running time. + +Please note that the ideal settings for my backup configuration might +differ from the ones for yours. You might get even better results after +tuning of your own. + +These parameters have a nice behaviour for tuning: `upload-queue-size` +doesn't seem to have much drawback if at all when being increased (it +begins to give signs of slowing down obnam at 10000 here but it might be +the performance variance inherent in my configuration) and increasing +`lru-size` only increases memory usage a bit without slowing things +noticeably after reaching the ideal spot. + +A good rule of thumb seems to try increasing one of these parameters by +2x or 4x and keep going at it until performance stops increasing.
Fix paths in inline
diff --git a/faq.mdwn b/faq.mdwn index 46adeda..fbef8d8 100644 --- a/faq.mdwn +++ b/faq.mdwn @@ -1,3 +1,3 @@ [[!meta title="Obnam FAQ"]] -[[!inline pages="page(obnam/faq/*)" archive=yes template=titlepage]] +[[!inline pages="page(faq/*)" archive=yes template=titlepage]]
Fix typo
diff --git a/1.0.mdwn b/1.0.mdwn index 487485c..96449ae 100644 --- a/1.0.mdwn +++ b/1.0.mdwn @@ -1,7 +1,7 @@ [[!meta title="Obnam 1.0 (backup software); a story in many words"]] [[!meta date="Fri, 8 Jun 2012 07:09:33 +0000"]] -Date:"Fri, 8 Jun 2012 07:09:33 +0000 +Date: Fri, 8 Jun 2012 07:09:33 +0000 **tl;dr**: Version 1.0 of [Obnam](http://liw.fi/obnam/), my snapshotting, de-duplicating, encrypting backup program is released.
Put a date on Obnam 1.0 announcement
diff --git a/1.0.mdwn b/1.0.mdwn index a49a246..487485c 100644 --- a/1.0.mdwn +++ b/1.0.mdwn @@ -1,4 +1,7 @@ [[!meta title="Obnam 1.0 (backup software); a story in many words"]] +[[!meta date="Fri, 8 Jun 2012 07:09:33 +0000"]] + +Date:"Fri, 8 Jun 2012 07:09:33 +0000 **tl;dr**: Version 1.0 of [Obnam](http://liw.fi/obnam/), my snapshotting, de-duplicating, encrypting backup program is released.
Fix mailing list URLs to point at new list
diff --git a/contact.mdwn b/contact.mdwn
index 3e48ccf..21cc24c 100644
--- a/contact.mdwn
+++ b/contact.mdwn
@@ -1,7 +1,7 @@
Contact
-------
-* [Mailing list](http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-flarn.net)
+* [Mailing list](http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org)
* `#obnam` on `irc.oftc.net` for IRC discussions about Obnam
- the mailing list is strongly preferred for asking help with Obnam
- Lars rarely has time to discuss anything serious on IRC
diff --git a/status.mdwn b/status.mdwn
index 89c6981..5a67dfb 100644
--- a/status.mdwn
+++ b/status.mdwn
@@ -13,7 +13,7 @@ Using the Obnam web pages for bug reporting is not a good idea,
since they are not a good way to conduct a discussion.
* E-mail to the
- [mailing list](http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-flarn.net).
+ [mailing list](http://listmaster.pepperfish.net/cgi-bin/mailman/listinfo/obnam-support-obnam.org).
See [[contact page|contact]] for more instructions on the mailing list.
Changed pages:
bugs/symlink-as-parent-of-backup-root
Changed by: liw
Commit type: git
Date: 16:23:59 06/04/14
Add bug about backup parents being symlinks
diff --git a/bugs/symlink-as-parent-of-backup-root.mdwn b/bugs/symlink-as-parent-of-backup-root.mdwn new file mode 100644 index 0000000..0034461 --- /dev/null +++ b/bugs/symlink-as-parent-of-backup-root.mdwn @@ -0,0 +1,9 @@ +Valery Yundin reports: + +> It is a problem not only when backup root is a symbolic link, but also when +> any parent directory of backup root is a symbolic link. Unless you +> explicitly ask to restore a directory which is already below symlink. + +<http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-April/002976.html> + +--liw
Changed pages:
bugs/backup-downloads-too-much-data
Changed by: liw
Commit type: git
Date: 14:35:55 06/04/14
Add bug from Andrew Ruthwen about excessive downloads
diff --git a/bugs/backup-downloads-too-much-data.mdwn b/bugs/backup-downloads-too-much-data.mdwn new file mode 100644 index 0000000..2487feb --- /dev/null +++ b/bugs/backup-downloads-too-much-data.mdwn @@ -0,0 +1,21 @@ +Andrew Ruthwen reported that a small backup delta was very slow, and had these +performance stats: + + 2014-04-17 18:56:34 INFO Backup performance statistics: + 2014-04-17 18:56:34 INFO * files found: 3643 + 2014-04-17 18:56:34 INFO * files backed up: 7 + 2014-04-17 18:56:34 INFO * uploaded chunk data: 18818 bytes (18 KiB) + 2014-04-17 18:56:34 INFO * total uploaded data (incl. metadata): 3097917 + bytes (2 MiB) + 2014-04-17 18:56:34 INFO * total downloaded data (incl. metadata): + 89711629838 bytes (83 GiB) + 2014-04-17 18:56:34 INFO * transfer overhead: 89714708937 bytes (83 GiB) + 2014-04-17 18:56:34 INFO * duration: 29275.9613361 s (8h7m56s) + 2014-04-17 18:56:34 INFO * average speed: 0.642779917077 B/s + +This is way too much downloaded data. + +<http://listmaster.pepperfish.net/pipermail/obnam-support-obnam.org/2014-April/002979.html> + +--liw +
Changed pages:
bugs/fsck-rechecks-same-chunks
Changed by: liw
Commit type: git
Date: 14:26:55 06/04/14
Add bug about fsck re-checking same chunks
diff --git a/bugs/fsck-rechecks-same-chunks.mdwn b/bugs/fsck-rechecks-same-chunks.mdwn new file mode 100644 index 0000000..208a2bb --- /dev/null +++ b/bugs/fsck-rechecks-same-chunks.mdwn @@ -0,0 +1,5 @@ +Alexander Sitnik reported that he'd noticed that "obnam fsck" checks +all chunks in all generations, resulting in the same chunks being checked +over and over. It should speed up "obnam fsck" if each chunk was checked +only once. I note that this may need to be done with some care so that +memory use doesn't increase too much. --liw