Chunking

Attic uses the FastCDC algorithm to split uploaded NARs into chunks for deduplication. There are four main parameters that control chunking in Attic:

• nar-size-threshold: The minimum NAR size to trigger chunking
  • When set to 0, chunking is disabled entirely for newly-uploaded NARs
  • When set to 1, all newly-uploaded NARs are chunked
• min-size: The preferred minimum size of a chunk, in bytes
• avg-size: The preferred average size of a chunk, in bytes
• max-size: The preferred maximum size of a chunk, in bytes

Configuration

When upgrading from an older version without support for chunking, you must include the new [chunking] section:

# Data chunking
#
# Warning: If you change any of the values here, it will be
# difficult to reuse existing chunks for newly-uploaded NARs
# since the cutpoints will be different. As a result, the
# deduplication ratio will suffer for a while after the change.
[chunking]
# The minimum NAR size to trigger chunking
#
# If 0, chunking is disabled entirely for newly-uploaded NARs.
# If 1, all newly-uploaded NARs are chunked.
nar-size-threshold = 131072 # chunk files that are 128 KiB or larger

# The preferred minimum size of a chunk, in bytes
min-size = 65536            # 64 KiB

# The preferred average size of a chunk, in bytes
avg-size = 131072           # 128 KiB

# The preferred maximum size of a chunk, in bytes
max-size = 262144           # 256 KiB