feat: bundled attachments

CHANGELOG.md

@@ -4,6 +4,21 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.9.956] - 2026-04-17
+### Added
+- Outbox attachment bundling: when the `use_bundled_attachments` config flag
+  is on (off by default), the outbox packs as many pending items' attachment
+  files as fit into a single zip capped at 8 MiB and uploads the bundle as
+  one Matrix file event, then sends each bundled item's text event with the
+  bundled paths short-circuited. Items whose own attachments exceed the cap
+  fall through to the existing single-item send path. Receivers recognize
+  the `com.lotti.bundle` event marker and unpack the zip to each entry's
+  target relative path unconditionally, so turning the flag on is forward-
+  compatible with peers already on this release.
+- Attachment enumerator that derives the file-on-disk set (JSON plus any
+  journal audio/image file) for a `SyncMessage`. Used by the outbox bundling
+  path and available for future packing strategies.
+
 ## [0.9.954] - 2026-04-17
 ### Added
 - Optional gzip compression for JSON sync attachments, gated by the

flatpak/com.matthiasn.lotti.metainfo.xml

@@ -31,6 +31,11 @@
   <launchable type="desktop-id">com.matthiasn.lotti.desktop</launchable>
   <icon type="stock">com.matthiasn.lotti</icon>
   <releases>
+    <release version="0.9.956" date="2026-04-17">
+      <description>
+          <p>Outbox attachment bundling: when the new config flag is on, pending sync items' attachment files are packed into a single zip (capped at 8 MiB) and uploaded as one Matrix file event, collapsing many per-file uploads into one round-trip over slow links. Items whose own attachments exceed the cap fall through to the existing per-file path. Receivers recognize the bundle marker and unpack entries to their target paths unconditionally, so enabling bundling on a single device stays compatible with peers already on this release.</p>
+      </description>
+    </release>
     <release version="0.9.954" date="2026-04-17">
       <description>
           <p>Optional gzip compression for JSON sync attachments over Matrix, gated by a new config flag that is off by default. Receivers always decompress when the encoding marker is present on an attachment event, so turning compression on for a single device stays compatible with peers on this release. Only .json attachments are compressed; media files are already on compressed formats and skip the branch.</p>

lib/database/journal_db/config_flags.dart

@@ -115,6 +115,14 @@ Future<void> initConfigFlags(
     ),
   );
 
+  await db.insertFlagIfNotExists(
+    const ConfigFlag(
+      name: useBundledAttachmentsFlag,
+      description: 'Pack multiple sync attachments into one zip on send?',
+      status: false,
+    ),
+  );
+
   await db.insertFlagIfNotExists(
     const ConfigFlag(
       name: enableAgentsFlag,

lib/features/sync/README.md

@@ -350,6 +350,32 @@ When the flag is on, the uploaded file name gains a `.gz` suffix and the
 event content includes the encoding header; otherwise bytes are sent
 verbatim with no header and no suffix.
 
+### Attachment Bundling
+
+Attachment events may carry a `com.lotti.bundle: true` marker in the Matrix
+event content. The event's payload is then a zip archive whose entries are
+named by their logical `relativePath` values. When a receiver sees this
+marker it unpacks the zip instead of writing the zip itself, routing every
+entry through the same `_targetFile()` path-traversal guard and the
+non-agent-file existing-on-disk dedup that single-file saves use. The outer
+`relativePath` of a bundle event is `.bundles/<uuid>.zip`, a location no
+sync payload refers to, so older receivers that predate bundle support
+store the zip harmlessly while newer receivers recognize the marker.
+
+On the send side, bundling is gated by the `use_bundled_attachments` config
+flag (off by default). When the flag is on, `OutboxProcessor.processQueue`
+enumerates attachments across the current pending batch via
+`enumerateAttachments`, greedily packs as many as fit into one zip up to
+`SyncTuning.outboxBundleMaxBytes` (8 MiB), uploads the bundle as one Matrix
+file event, and then sends each bundled item's text event with the bundled
+relative paths in `MatrixMessageContext.skipAttachmentPaths`. The sender's
+`_sendFile` short-circuits for any path in that set. Items whose own
+attachments exceed the bundle cap fall through to the existing single-item
+path on the next tick, so large media uploads are never hidden behind a
+bundle. The bundle marker and the `com.lotti.encoding` marker are
+independent; a bundled event is always a zip and does not set the encoding
+header.
+
 ```mermaid
 flowchart TD
   Event["Matrix event"] --> Decode["Decode SyncMessage"]

lib/features/sync/matrix/attachment_enumerator.dart

@@ -0,0 +1,155 @@
+import 'dart:convert';
+import 'dart:io';
+import 'dart:typed_data';
+
+import 'package:lotti/classes/journal_entities.dart';
+import 'package:lotti/features/sync/model/sync_message.dart';
+import 'package:lotti/utils/audio_utils.dart';
+import 'package:lotti/utils/image_utils.dart';
+import 'package:path/path.dart' as p;
+
+/// Describes a single file that a [SyncMessage] would upload individually via
+/// `MatrixMessageSender._sendFile` when bundling is disabled.
+///
+/// When the file is the JSON payload the enumerator already read to discover
+/// media paths, `bytes` is populated so downstream callers (e.g. the
+/// bundler) can avoid re-reading the same file from disk. For media files
+/// `bytes` is null and callers must load the contents themselves.
+typedef AttachmentDescriptor = ({
+  String fullPath,
+  String relativePath,
+  int size,
+  Uint8List? bytes,
+});
+
+/// Enumerates the on-disk files a [message] would upload individually.
+///
+/// Files that are missing or empty are silently omitted. A message that
+/// carries no file attachments returns an empty list. Callers should treat
+/// a partial enumeration (e.g. the JSON is readable but the media file is
+/// missing) as a signal to fall back to the solo send path, where the
+/// existing error handling applies.
+///
+/// This helper never throws; it returns an empty list on any I/O failure.
+Future<List<AttachmentDescriptor>> enumerateAttachments({
+  required SyncMessage message,
+  required Directory documentsDirectory,
+}) async {
+  switch (message) {
+    case final SyncJournalEntity msg:
+      return _forJournalEntity(
+        message: msg,
+        documentsDirectory: documentsDirectory,
+      );
+    case final SyncAgentEntity msg:
+      return _forOptionalJsonPath(msg.jsonPath, documentsDirectory);
+    case final SyncAgentLink msg:
+      return _forOptionalJsonPath(msg.jsonPath, documentsDirectory);
+    case _:
+      return const [];
+  }
+}
+
+Future<List<AttachmentDescriptor>> _forOptionalJsonPath(
+  String? relativePath,
+  Directory documentsDirectory,
+) async {
+  if (relativePath == null) return const [];
+  return _forFile(
+    relativePath: relativePath,
+    documentsDirectory: documentsDirectory,
+  );
+}
+
+Future<List<AttachmentDescriptor>> _forJournalEntity({
+  required SyncJournalEntity message,
+  required Directory documentsDirectory,
+}) async {
+  // Read once and cache: the same bytes feed the JournalEntity.fromJson pass
+  // below (to discover audio/image paths) and any downstream consumer such as
+  // the bundler, so subsequent reads can short-circuit.
+  final jsonFullPath = _resolveWithinDocuments(
+    relativePath: message.jsonPath,
+    documentsDirectory: documentsDirectory,
+  );
+  if (jsonFullPath == null) return const [];
+  Uint8List bytes;
+  try {
+    bytes = await File(jsonFullPath).readAsBytes();
+  } on FileSystemException {
+    return const [];
+  }
+  if (bytes.isEmpty) return const [];
+
+  final base = (
+    fullPath: jsonFullPath,
+    relativePath: message.jsonPath,
+    size: bytes.length,
+    bytes: bytes,
+  );
+
+  try {
+    final entity = JournalEntity.fromJson(
+      json.decode(utf8.decode(bytes)) as Map<String, dynamic>,
+    );
+    final media = await entity.maybeMap(
+      journalAudio: (JournalAudio audio) async => _forFile(
+        relativePath: AudioUtils.getRelativeAudioPath(audio),
+        documentsDirectory: documentsDirectory,
+      ),
+      journalImage: (JournalImage image) async => _forFile(
+        relativePath: getRelativeImagePath(image),
+        documentsDirectory: documentsDirectory,
+      ),
+      orElse: () async => const <AttachmentDescriptor>[],
+    );
+    return [base, ...media];
+  } catch (_) {
+    return [base];
+  }
+}
+
+Future<List<AttachmentDescriptor>> _forFile({
+  required String relativePath,
+  required Directory documentsDirectory,
+}) async {
+  final fullPath = _resolveWithinDocuments(
+    relativePath: relativePath,
+    documentsDirectory: documentsDirectory,
+  );
+  if (fullPath == null) return const [];
+  try {
+    final size = await File(fullPath).length();
+    if (size <= 0) return const [];
+    return [
+      (
+        fullPath: fullPath,
+        relativePath: relativePath,
+        size: size,
+        bytes: null,
+      ),
+    ];
+  } on FileSystemException {
+    return const [];
+  }
+}
+
+/// Resolves [relativePath] against [documentsDirectory] and rejects anything
+/// that escapes the documents tree — defends the enumerator (and therefore
+/// any downstream read/upload) against `..` segments or absolute fragments
+/// in a crafted `jsonPath`. Returns the absolute path on success, or null
+/// when the path is outside the documents directory or cannot be normalized.
+String? _resolveWithinDocuments({
+  required String relativePath,
+  required Directory documentsDirectory,
+}) {
+  var rel = relativePath;
+  if (p.isAbsolute(rel)) {
+    final prefix = p.rootPrefix(rel);
+    rel = rel.substring(prefix.length);
+  }
+  final joined = p.joinAll(rel.split('/').where((part) => part.isNotEmpty));
+  final resolved = p.normalize(p.join(documentsDirectory.path, joined));
+  if (!p.isWithin(documentsDirectory.path, resolved)) return null;
+  return resolved;
+}

lib/features/sync/matrix/consts.dart

@@ -14,3 +14,17 @@ const String attachmentEncodingKey = 'com.lotti.encoding';
 /// Value for [attachmentEncodingKey] indicating the attachment bytes are
 /// gzip-compressed; the receiver must decompress before writing to disk.
 const String attachmentEncodingGzip = 'gzip';
+
+/// Key in an attachment event's content that marks the file as a zip
+/// containing multiple inner attachments. Entries inside the zip are named
+/// after their target relative paths. Receivers that understand this marker
+/// unpack the zip and write each entry to its inner path instead of writing
+/// the zip itself. Receivers that do not understand the marker will still
+/// write the zip to the outer `relativePath`, which is harmless because that
+/// path is under [attachmentBundleDirPrefix] and is not referenced by any
+/// sync payload.
+const String attachmentBundleKey = 'com.lotti.bundle';
+
+/// Documents-relative directory prefix used for the outer `relativePath` of
+/// bundle events. No sync payload resolves to a path under this prefix.
+const String attachmentBundleDirPrefix = '.bundles/';