diff --git a/README.md b/README.md index 893c155..5216ace 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,8 @@ # Papyrus Reader `papyrus_reader` is a self-contained Flutter library for reading reflowable -EPUB 2/3 and PDF documents on Android, iOS, Web, Linux, macOS, and Windows. +EPUB 2/3, PDF, MOBI, AZW3, TXT, CBZ and CBR documents on Android, iOS, Web, +Linux, macOS, and Windows. It provides versioned EPUB content locations, PDF page locations, TOC/outline navigation, scroll and paginated layouts, configurable appearance, and a responsive @@ -67,16 +68,27 @@ flutter run ``` The six-platform example includes deterministic long EPUB and three-page PDF -assets, a local EPUB/PDF file picker, light/dark themes, and in-memory +assets, a local book file picker, light/dark themes, and in-memory position/preference restoration. Example positions reset when the app restarts; production persistence belongs to the host application. ## Current scope -The initial release supports reflowable EPUB 2/3 and PDF. Fixed-layout EPUBs -return `ReaderErrorCode.unsupportedFixedLayout`. MOBI/AZW3, TXT, comic -archives, search, bookmarks, highlights, and notes are planned extensions. -Encrypted/DRM documents are unsupported. EPUB uses semantic reflow: publisher +Supported inputs are reflowable EPUB 2/3, PDF, DRM-free MOBI/AZW3 (KF8), plain +text (UTF-8, BOM-marked UTF-16 and Windows-1252), CBZ and CBR (RAR4/RAR5). +Comic pages use natural filename order and support JPEG, PNG, GIF, WebP and BMP; +animations display as still pages. Password-protected archives, multi-volume +comics, Kindle Print Replica, Topaz and encrypted/DRM books are unsupported. +Fixed-layout EPUBs return `ReaderErrorCode.unsupportedFixedLayout`. + +Text and Kindle inputs use the reflow engine. Comics use a transient in-memory +page document for zoom, scrolling and facing pages; original library files are +never changed. Conversion runs in a cancellable isolate/browser worker before +opening; large comics may take longer and require memory for the page document. +Reflow uses EPUB-style content locators; comics use PDF-style page locators, so +saved positions, bookmarks and reading activity use the existing contracts. + +EPUB and Kindle use semantic reflow: publisher stylesheets, embedded fonts, advanced SVG/math, RTL progression and footnote interaction are incomplete; complex tables render as readable rows. diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md index a6a783a..3709208 100644 --- a/docs/ARCHITECTURE.md +++ b/docs/ARCHITECTURE.md @@ -42,6 +42,25 @@ Custom HTML rendering restores approximate progression rather than the default renderer’s precise content offset. Supported-format limits are in the [README](../README.md#current-scope). +## Additional input formats + +The shared document worker prepares TXT and MOBI/AZW3 as transient EPUB content, +then applies the same sanitizer, semantic offsets and layout pipeline. Text +chapters use deterministic bounded chunks. The vendored Kindle decoder records +its upstream version, license and browser adjustments in +`third_party/kindle_unpack/UPSTREAM.md`. + +CBZ and CBR are decoded in the worker into a transient PDF page document. Natural +path ordering determines page indices. Images preserve their aspect ratios and +JPEG data; archive readers release their resources after preparation. Binary +responses transfer typed byte buffers in browsers instead of serializing them +as large JSON arrays. Cancelling a load terminates its conversion worker. + +Input format remains on `ReaderDocument`; reflow inputs use `EpubReaderLocator` +and fixed-page inputs use `PdfReaderLocator`. The host retains the original file, +and no conversion cache or second persisted copy is introduced. Comic colors +are preserved in dark mode. + ## PDF resources `PdfReaderEngine` uses a `PdfFacade` so engine tests do not depend on PDFium. diff --git a/docs/VALIDATION.md b/docs/VALIDATION.md index c7b8c5a..d62b314 100644 --- a/docs/VALIDATION.md +++ b/docs/VALIDATION.md @@ -13,7 +13,11 @@ python3 tool/check_worker_build.py Widget tests inject an in-process EPUB transport because fake clocks do not model resident-isolate scheduling. Native transport tests and the compiled browser suite -exercise the real transports. Browser checks require the built example and its +exercise the real transports. Additional-format tests decode real Gutenberg MOBI/KF8 +text and covers, compare rendered comic pixels for natural page order, restore +serialized positions, and exercise cancellation. Browser checks also open every +format through the example file picker and record animation-frame gaps during +worker processing. Browser checks require the built example and its packaged worker. `PAPYRUS_CHROME_EXECUTABLE` selects an installed Chrome; `PAPYRUS_SCREENSHOT_DIR` saves review images outside source control. @@ -53,3 +57,44 @@ The driver writes screenshots and timings under `client/app/build/`. Keep run results with the relevant PR, including the tested revisions, platform and unrun checks. Debug turn timings include deliberate observation frames and test overhead; they are not production latency benchmarks. + +## Large-book stress checks + +Keep large downloads outside Git. The opt-in native corpus expects Webster's +Unabridged Dictionary (Gutenberg 29765) as `websters-dictionary.epub`, `.mobi`, +`.azw3`, and `.txt`, plus `alice-high-resolution.pdf`, `whiz-272-pages.cbz`, and +`whiz-272-pages.cbr`. Record source URLs, checksums and any generated/repeated +content alongside the files. Never present a repeated-page fixture as a newly +published book. + +```sh +PAPYRUS_LARGE_BOOKS=/absolute/path/to/corpus \ + ../tools/flutter test test/stress/large_books_test.dart --concurrency=1 +``` + +This checks native loading, seeking and locator restoration, and reports event-loop +gaps and sampled test-process RSS. It does not render Flutter viewports or establish +device memory limits. The corpus is skipped in ordinary test runs. + +For actual browser viewports, build the example, serve `example/build/web` on +localhost port 8791, and run: + +```sh +PAPYRUS_CHROME_EXECUTABLE=/path/to/chrome \ +PAPYRUS_STRESS_OUTPUT=/absolute/path/to/results \ + node tool/browser/stress.mjs /absolute/path/to/corpus/*.epub +``` + +Pass any supported book files as arguments. `PAPYRUS_STRESS_ORIGIN` overrides the +server URL. `PAPYRUS_STRESS_CANCEL_ONLY=1` checks cancellation during preparation +without waiting for full reading. The runner records idle and loading frame gaps, +long tasks, page-turn screenshots, worker closure and Back response. macOS/Linux +RSS sampling covers all processes belonging to the test browser, including GPU +and workers; it is not the reader's isolated heap. Each file gets a fresh browser. +Screenshots are excluded from the navigation frame measurements. + +Functional success does not mean acceptable performance: review load duration, +long tasks and memory separately. Compare frame gaps with the idle baseline, avoid +concurrent builds, and record other desktop load. Browser page turns and native +headless seeks do not certify long-distance Flutter pagination, phone behavior, +or every page in a large publication. diff --git a/example/lib/main.dart b/example/lib/main.dart index f4a7f00..2171fd5 100644 --- a/example/lib/main.dart +++ b/example/lib/main.dart @@ -290,19 +290,25 @@ final class _DemoLibrary extends StatelessWidget { acceptedTypeGroups: const [ XTypeGroup( label: 'Books', - extensions: ['epub', 'pdf'], - mimeTypes: ['application/epub+zip', 'application/pdf'], + extensions: ['epub', 'pdf', 'mobi', 'azw3', 'txt', 'cbz', 'cbr'], + uniformTypeIdentifiers: [ 'org.idpf.epub-container', 'com.adobe.pdf', + 'public.data', ], ), ], ); if (file == null || !context.mounted) return; final extension = file.name.split('.').last.toLowerCase(); - if (extension != 'epub' && extension != 'pdf') { - throw const FormatException('Choose an EPUB or PDF document.'); + final format = ReaderFormat.values + .where((format) => format.name == extension) + .firstOrNull; + if (format == null) { + throw const FormatException( + 'Choose an EPUB, PDF, MOBI, AZW3, TXT, CBZ or CBR document.', + ); } final id = 'file:${file.name}:${await file.length()}:${await file.lastModified()}'; @@ -310,7 +316,7 @@ final class _DemoLibrary extends StatelessWidget { final document = ReaderDocument( id: id, title: file.name, - format: extension == 'epub' ? ReaderFormat.epub : ReaderFormat.pdf, + format: format, loadBytes: file.readAsBytes, ); Navigator.of(context).push( diff --git a/example/pubspec.lock b/example/pubspec.lock index 814d9ac..0742219 100644 --- a/example/pubspec.lock +++ b/example/pubspec.lock @@ -25,6 +25,22 @@ packages: url: "https://pub.dev" source: hosted version: "2.13.1" + barcode: + dependency: transitive + description: + name: barcode + sha256: "7b6729c37e3b7f34233e2318d866e8c48ddb46c1f7ad01ff7bb2a8de1da2b9f4" + url: "https://pub.dev" + source: hosted + version: "2.2.9" + bidi: + dependency: transitive + description: + name: bidi + sha256: "77f475165e94b261745cf1032c751e2032b8ed92ccb2bf5716036db79320637d" + url: "https://pub.dev" + source: hosted + version: "2.0.13" boolean_selector: dependency: transitive description: @@ -272,6 +288,37 @@ packages: url: "https://pub.dev" source: hosted version: "1.0.1" + kindle_unpack: + dependency: transitive + description: + path: "../third_party/kindle_unpack" + relative: true + source: path + version: "0.2.0" + koni_archive_core: + dependency: transitive + description: + name: koni_archive_core + sha256: f323296866d6224725839e2b8ef25dfeba5e0c2476877fa7726f4b966a4c7a84 + url: "https://pub.dev" + source: hosted + version: "0.9.0" + koni_codecs: + dependency: transitive + description: + name: koni_codecs + sha256: "0a0ffa23f62f627bc50db6de79e88a6a41e019d3a21c6a455ac5ae7c1677a8bd" + url: "https://pub.dev" + source: hosted + version: "0.9.0" + koni_rar: + dependency: transitive + description: + name: koni_rar + sha256: c8f1a5b2f1b9d8f6cfd981c67e8c34298f722806f7288f06faabb40004362f56 + url: "https://pub.dev" + source: hosted + version: "0.9.0" leak_tracker: dependency: transitive description: @@ -375,6 +422,14 @@ packages: url: "https://pub.dev" source: hosted version: "1.9.1" + path_parsing: + dependency: transitive + description: + name: path_parsing + sha256: "883402936929eac138ee0a45da5b0f2c80f89913e6dc3bf77eb65b84b409c6ca" + url: "https://pub.dev" + source: hosted + version: "1.1.0" path_provider: dependency: transitive description: @@ -423,6 +478,14 @@ packages: url: "https://pub.dev" source: hosted version: "2.3.0" + pdf: + dependency: transitive + description: + name: pdf + sha256: e47a275b267873d5944ad5f5ff0dcc7ac2e36c02b3046a0ffac9b72fd362c44b + url: "https://pub.dev" + source: hosted + version: "3.12.0" pdfium_dart: dependency: transitive description: @@ -495,6 +558,14 @@ packages: url: "https://pub.dev" source: hosted version: "2.2.0" + qr: + dependency: transitive + description: + name: qr + sha256: "2c0189dca64a3ab08dda7acd7911c89af2374f08eb4eaded3e8ef4609f3b9c8b" + url: "https://pub.dev" + source: hosted + version: "3.0.3" record_use: dependency: transitive description: diff --git a/lib/src/domain/reader_types.dart b/lib/src/domain/reader_types.dart index b5da8a0..7d9417c 100644 --- a/lib/src/domain/reader_types.dart +++ b/lib/src/domain/reader_types.dart @@ -1,4 +1,17 @@ -enum ReaderFormat { epub, pdf } +enum ReaderFormat { + epub, + pdf, + mobi, + azw3, + txt, + cbr, + cbz; + + bool get isReflowable => switch (this) { + epub || mobi || azw3 || txt => true, + pdf || cbr || cbz => false, + }; +} enum ReaderLayoutMode { paginated, scroll } diff --git a/lib/src/engine/epub/epub_reader_engine.dart b/lib/src/engine/epub/epub_reader_engine.dart index 7392d4f..be72e2a 100644 --- a/lib/src/engine/epub/epub_reader_engine.dart +++ b/lib/src/engine/epub/epub_reader_engine.dart @@ -85,7 +85,12 @@ final class EpubReaderEngine extends ReaderEngine { ReaderPreferences get preferences => _preferences; @override - Set get supportedFormats => const {ReaderFormat.epub}; + Set get supportedFormats => const { + ReaderFormat.epub, + ReaderFormat.mobi, + ReaderFormat.azw3, + ReaderFormat.txt, + }; @override ReaderCapabilities get capabilities => const ReaderCapabilities( supportsPagination: true, @@ -103,11 +108,11 @@ final class EpubReaderEngine extends ReaderEngine { ReaderLocator? initialLocator, required ReaderPreferences preferences, }) async { - if (document.format != ReaderFormat.epub || + if (!supportedFormats.contains(document.format) || (initialLocator != null && initialLocator is! EpubReaderLocator)) { throw const ReaderException( ReaderErrorCode.invalidDocument, - 'The EPUB document or initial locator has the wrong format.', + 'The text document or initial locator has the wrong format.', ); } cancelPendingSeek(); @@ -119,7 +124,12 @@ final class EpubReaderEngine extends ReaderEngine { candidate = await _workerFactory(); _requireCurrent(generation); _openingWorker = candidate; - final metadata = await candidate.open(bytes); + final metadata = await candidate.request( + document.format == ReaderFormat.epub + ? 'open' + : 'open_${document.format.name}', + bytes, + ); _requireCurrent(generation); final count = metadata['count'] as int; final restored = initialLocator as EpubReaderLocator?; @@ -162,7 +172,7 @@ final class EpubReaderEngine extends ReaderEngine { if (!identical(candidate, _worker)) candidate?.dispose(); throw ReaderException( ReaderErrorCode.invalidDocument, - 'The EPUB document is invalid or malformed.', + 'The text document is invalid or malformed.', cause: error, ); } finally { diff --git a/lib/src/engine/epub/worker/epub_processor.dart b/lib/src/engine/epub/worker/epub_processor.dart index a4a1c55..2abce3a 100644 --- a/lib/src/engine/epub/worker/epub_processor.dart +++ b/lib/src/engine/epub/worker/epub_processor.dart @@ -5,6 +5,12 @@ import 'package:epub_pro/epub_pro.dart'; import 'package:html/dom.dart' as html_dom; import 'package:html/parser.dart' as html_parser; +import 'package:kindle_unpack/kindle_unpack.dart'; +import 'package:koni_archive_core/koni_archive_core.dart' + show EncryptedArchiveException; + +import '../../formats/comic_document.dart'; +import '../../formats/reflow_document.dart'; import 'epub_content.dart'; /// Pure Dart document session, shared by native isolates and browser workers. @@ -18,6 +24,18 @@ final class EpubProcessor { Object? argument, ) async { if (command == 'open') return _open(argument as Uint8List); + if (command == 'open_txt') return _open(textToEpub(argument as Uint8List)); + if (command == 'open_mobi' || command == 'open_azw3') { + return _open(kindleToEpub(argument as Uint8List)); + } + if (command == 'prepare_cbz' || command == 'prepare_cbr') { + return { + 'bytes': await comicToPdf( + argument as Uint8List, + isRar: command == 'prepare_cbr', + ), + }; + } if (command == 'chapter') return _chapter(argument as int); throw ArgumentError('Unknown EPUB worker command: $command'); } @@ -409,11 +427,27 @@ Future> processEpubMessage( }; } on EpubProcessingException catch (error) { return {'id': message['id'], 'error': error.code, 'message': error.message}; + } on HeaderException catch (error) { + return { + 'id': message['id'], + 'error': error.message.contains('encrypted') + ? 'encryptedDocument' + : 'invalidDocument', + 'message': error.message.contains('encrypted') + ? 'This book is encrypted. Open a DRM-free copy.' + : 'The Kindle document is invalid or malformed.', + }; + } on EncryptedArchiveException { + return { + 'id': message['id'], + 'error': 'encryptedDocument', + 'message': 'This comic archive requires a password.', + }; } catch (_) { return { 'id': message['id'], 'error': 'invalidDocument', - 'message': 'The EPUB document or chapter is invalid or malformed.', + 'message': 'The book document or chapter is invalid or malformed.', }; } } diff --git a/lib/src/engine/epub/worker/epub_worker_native.dart b/lib/src/engine/epub/worker/epub_worker_native.dart index 3dfebf8..9cc26d0 100644 --- a/lib/src/engine/epub/worker/epub_worker_native.dart +++ b/lib/src/engine/epub/worker/epub_worker_native.dart @@ -66,9 +66,10 @@ final class _NativeEpubWorker implements EpubWorker { if (message['error'] != null) { pending.completeError( ReaderException( - message['error'] == 'unsupportedFixedLayout' - ? ReaderErrorCode.unsupportedFixedLayout - : ReaderErrorCode.invalidDocument, + ReaderErrorCode.values.firstWhere( + (code) => code.name == message['error'], + orElse: () => ReaderErrorCode.invalidDocument, + ), message['message'] as String, ), ); diff --git a/lib/src/engine/epub/worker/epub_worker_web.dart b/lib/src/engine/epub/worker/epub_worker_web.dart index d31308b..173165d 100644 --- a/lib/src/engine/epub/worker/epub_worker_web.dart +++ b/lib/src/engine/epub/worker/epub_worker_web.dart @@ -26,15 +26,19 @@ final class _WebEpubWorker implements EpubWorker { _WebEpubWorker(this.worker) { worker.onmessage = ((web.MessageEvent event) { final message = - jsonDecode((event.data as JSString).toDart) as Map; + (event.data.isA() + ? jsonDecode((event.data as JSString).toDart) + : event.data.dartify()) + as Map; final result = _pending.remove(message['id']); if (result == null) return; if (message['error'] != null) { result.completeError( ReaderException( - message['error'] == 'unsupportedFixedLayout' - ? ReaderErrorCode.unsupportedFixedLayout - : ReaderErrorCode.invalidDocument, + ReaderErrorCode.values.firstWhere( + (code) => code.name == message['error'], + orElse: () => ReaderErrorCode.invalidDocument, + ), message['message'] as String, ), ); diff --git a/lib/src/engine/formats/comic_document.dart b/lib/src/engine/formats/comic_document.dart new file mode 100644 index 0000000..a3b42ec --- /dev/null +++ b/lib/src/engine/formats/comic_document.dart @@ -0,0 +1,128 @@ +import 'dart:typed_data'; + +import 'package:archive/archive.dart'; +import 'package:koni_archive_core/koni_archive_core.dart' as rar; +import 'package:koni_rar/koni_rar.dart'; +import 'package:pdf/pdf.dart'; + +/// Build a transient page document inside the worker. Pages retain their +/// original dimensions and JPEG data; no library file or converted copy is saved. +Future comicToPdf(Uint8List bytes, {required bool isRar}) async { + final document = PdfDocument(); + var count = 0; + + void addPage(Uint8List bytes) { + final image = PdfImage.file(document, bytes: bytes); + final page = PdfPage( + document, + pageFormat: PdfPageFormat( + image.width.toDouble(), + image.height.toDouble(), + ), + ); + page.getGraphics().drawImage( + image, + 0, + 0, + image.width.toDouble(), + image.height.toDouble(), + ); + count++; + } + + if (isRar) { + final source = rar.MemoryByteSource(bytes); + rar.ArchiveReader? archive; + + try { + archive = await const RarFormat().openReader( + source, + const rar.ArchiveReadOptions(), + ); + final pages = + archive.entries + .where( + (entry) => + entry.type == rar.ArchiveEntryType.file && + isComicPage(entry.path), + ) + .toList() + ..sort((left, right) => compareComicPaths(left.path, right.path)); + + for (final page in pages) { + final content = BytesBuilder(copy: false); + + await for (final chunk in archive.openRead(page)) { + content.add(chunk); + } + + addPage(content.takeBytes()); + } + } finally { + await archive?.close(); + await source.close(); + } + } else { + final archive = ZipDecoder().decodeBytes(bytes); + + try { + final pages = + archive.files + .where((entry) => entry.isFile && isComicPage(entry.name)) + .toList() + ..sort((left, right) => compareComicPaths(left.name, right.name)); + + for (final page in pages) { + addPage(page.content); + page.clear(); + } + } finally { + archive.clear(); + } + } + + if (count == 0) { + throw const FormatException('The comic archive has no readable images.'); + } + + return document.save(); +} + +bool isComicPage(String path) { + final segments = path.replaceAll('\\', '/').split('/'); + return !segments.any((part) => part.startsWith('.') || part == '__MACOSX') && + RegExp( + r'\.(jpe?g|png|gif|webp|bmp)$', + caseSensitive: false, + ).hasMatch(path); +} + +/// Natural ordering: page2 precedes page10, including nested chapter folders. +int compareComicPaths(String left, String right) { + final parts = RegExp(r'\d+|\D+'); + final first = parts + .allMatches(left.toLowerCase()) + .map((match) => match[0]!) + .toList(); + final second = parts + .allMatches(right.toLowerCase()) + .map((match) => match[0]!) + .toList(); + + for (var index = 0; index < first.length && index < second.length; index++) { + final a = first[index]; + final b = second[index]; + final numeric = int.tryParse(a); + final other = int.tryParse(b); + final comparison = numeric != null && other != null + ? numeric.compareTo(other) + : a.compareTo(b); + + if (comparison != 0) { + return comparison; + } + } + + final length = first.length.compareTo(second.length); + return length != 0 ? length : left.compareTo(right); +} diff --git a/lib/src/engine/formats/reflow_document.dart b/lib/src/engine/formats/reflow_document.dart new file mode 100644 index 0000000..2256237 --- /dev/null +++ b/lib/src/engine/formats/reflow_document.dart @@ -0,0 +1,208 @@ +import 'dart:convert'; +import 'dart:typed_data'; + +import 'package:html/parser.dart' as html; +import 'package:kindle_unpack/kindle_unpack.dart'; + +/// Runs exclusively in the document worker. The source file is never replaced. +Uint8List textToEpub(Uint8List bytes) { + final text = decodeBookText( + bytes, + ).replaceAll('\r\n', '\n').replaceAll('\r', '\n'); + + if (text.trim().isEmpty) { + throw const FormatException('The text document is empty.'); + } + + final parts = []; + var start = 0; + + // Stable, bounded chapters keep text measurement and seek work manageable. + while (start < text.length) { + var end = (start + 32768).clamp(0, text.length); + + if (end < text.length) { + final boundary = text.lastIndexOf('\n', end); + + if (boundary > start) { + end = boundary + 1; + } else if (text.codeUnitAt(end - 1) >= 0xd800 && + text.codeUnitAt(end - 1) <= 0xdbff) { + end--; + } + } + + final paragraphs = text.substring(start, end).split(RegExp(r'\n[ \t]*\n')); + final body = paragraphs + .map( + (paragraph) => + '

${const HtmlEscape().convert(paragraph).replaceAll('\n', '
')}

', + ) + .join(); + parts.add( + XhtmlPart( + fileNumber: parts.length, + bytes: utf8.encode('$body'), + ), + ); + start = end; + } + + return EpubBuilder.build( + metadata: const EpubMetadata( + identifier: 'papyrus-text', + title: 'Text', + language: 'und', + ), + parts: parts, + ); +} + +Uint8List kindleToEpub(Uint8List bytes) { + final book = KindleBook.fromBytes(bytes); + final images = {for (final image in book.images.all) image.blockIndex: image}; + final parts = []; + + for (final part in book.parts) { + final source = decodeBookText( + part.bytes, + windows1252: book.mobi.textEncoding == 1252, + ); + final document = html.parse(source); + + for (final image in document.querySelectorAll('img')) { + final flowLink = RegExp( + r'^kindle:flow:([0-9A-V]+)', + caseSensitive: false, + ).firstMatch(image.attributes['src'] ?? ''); + final index = flowLink == null + ? null + : int.tryParse(flowLink.group(1)!, radix: 32); + final flow = book.flows?.flows + .where((flow) => flow.index == index && flow.kind == FlowKind.svg) + .firstOrNull; + + if (flow != null) { + final svg = html + .parseFragment(utf8.decode(flow.bytes)) + .querySelector('svg'); + + if (svg != null) { + image.replaceWith(svg); + } + } + } + + for (final image in document.querySelectorAll('img, image')) { + final source = + image.attributes['src'] ?? + image.attributes['href'] ?? + image.attributes['xlink:href'] ?? + image.attributes.entries + .where((entry) => entry.key.toString() == 'xlink:href') + .map((entry) => entry.value) + .firstOrNull ?? + ''; + final embedded = RegExp( + r'^kindle:embed:([0-9A-V]+)', + caseSensitive: false, + ).firstMatch(source); + final number = embedded == null + ? int.tryParse(image.attributes['recindex'] ?? '') + : int.tryParse(embedded.group(1)!, radix: 32); + final resource = number == null ? null : images[number - 1]; + + if (resource != null) { + image.attributes[image.localName == 'img' ? 'src' : 'href'] = + '../Images/${resource.name}'; + } + } + + parts.add( + XhtmlPart( + fileNumber: part.fileNumber, + bytes: utf8.encode(document.outerHtml), + ), + ); + } + + if (parts.isEmpty) { + throw const FormatException('The Kindle document has no readable content.'); + } + + return EpubBuilder.build( + metadata: EpubMetadata( + identifier: 'papyrus-kindle-${book.mobi.uniqueId}', + title: book.title, + language: book.exth?.language ?? 'und', + creators: book.exth?.authors ?? const [], + ), + parts: parts, + images: book.images.all, + ); +} + +String decodeBookText(Uint8List bytes, {bool windows1252 = false}) { + if (bytes.length >= 2 && + ((bytes[0] == 0xff && bytes[1] == 0xfe) || + (bytes[0] == 0xfe && bytes[1] == 0xff))) { + if (bytes.length.isOdd) { + throw const FormatException('Truncated UTF-16 text.'); + } + + final data = ByteData.sublistView(bytes); + final endian = bytes[0] == 0xff ? Endian.little : Endian.big; + return String.fromCharCodes([ + for (var offset = 2; offset < bytes.length; offset += 2) + data.getUint16(offset, endian), + ]); + } + + if (!windows1252) { + try { + return utf8.decode(bytes).replaceFirst(RegExp('^\uFEFF'), ''); + } on FormatException { + // Unmarked legacy text files commonly use Windows-1252. + } + } + + const controls = [ + 0x20ac, + 0x81, + 0x201a, + 0x192, + 0x201e, + 0x2026, + 0x2020, + 0x2021, + 0x2c6, + 0x2030, + 0x160, + 0x2039, + 0x152, + 0x8d, + 0x17d, + 0x8f, + 0x90, + 0x2018, + 0x2019, + 0x201c, + 0x201d, + 0x2022, + 0x2013, + 0x2014, + 0x2dc, + 0x2122, + 0x161, + 0x203a, + 0x153, + 0x9d, + 0x17e, + 0x178, + ]; + return String.fromCharCodes( + bytes.map( + (byte) => byte >= 0x80 && byte <= 0x9f ? controls[byte - 0x80] : byte, + ), + ); +} diff --git a/lib/src/engine/pdf/pdf_facade.dart b/lib/src/engine/pdf/pdf_facade.dart index 443b4ef..3d518ae 100644 --- a/lib/src/engine/pdf/pdf_facade.dart +++ b/lib/src/engine/pdf/pdf_facade.dart @@ -90,6 +90,7 @@ typedef PdfMetadataDocumentLoader = Future Function(); Future createPdfrxFacade( Uint8List bytes, { required String sourceName, + bool preserveColors = false, PdfMetadataDocumentLoader? metadataDocumentLoader, }) async { final reference = PdfDocumentRefData( @@ -114,6 +115,7 @@ Future createPdfrxFacade( final facade = _PdfrxFacade( documentRef: reference, + preserveColors: preserveColors, pageCount: metadataDocument.pageCount, outline: List.unmodifiable(outline), releaseDocument: releaseDocument, @@ -167,6 +169,7 @@ PdfFacadeOutlineEntry _convertOutline(PdfOutlineNode node) { final class _PdfrxFacade implements PdfFacade, DisposablePdfFacade { _PdfrxFacade({ required PdfDocumentRef documentRef, + this.preserveColors = false, required this.pageCount, required this.outline, VoidCallback? releaseDocument, @@ -176,6 +179,7 @@ final class _PdfrxFacade implements PdfFacade, DisposablePdfFacade { } final PdfDocumentRef _documentRef; + final bool preserveColors; final PdfViewerController _controller = PdfViewerController(); int _pageIndex = 0; double _pageOffset = 0; @@ -343,7 +347,7 @@ final class _PdfrxFacade implements PdfFacade, DisposablePdfFacade { } final revision = _viewerRevision; return applyPdfBrightness( - configuration.brightness, + preserveColors ? Brightness.light : configuration.brightness, Listener( onPointerSignal: (event) { if (configuration.layoutMode != ReaderLayoutMode.paginated || diff --git a/lib/src/engine/pdf/pdf_reader_engine.dart b/lib/src/engine/pdf/pdf_reader_engine.dart index 77eacfe..1a58719 100644 --- a/lib/src/engine/pdf/pdf_reader_engine.dart +++ b/lib/src/engine/pdf/pdf_reader_engine.dart @@ -1,3 +1,5 @@ +import 'dart:typed_data'; + import 'package:flutter/widgets.dart'; import '../../domain/reader_capabilities.dart'; @@ -11,15 +13,26 @@ import '../../domain/reader_toc_entry.dart'; import '../../domain/reader_types.dart'; import '../reader_engine.dart'; import 'pdf_facade.dart'; +import '../epub/worker/epub_worker.dart'; final class PdfReaderEngine extends ReaderEngine { - PdfReaderEngine({PdfFacadeFactory? facadeFactory}) - : _facadeFactory = facadeFactory; + PdfReaderEngine({ + PdfFacadeFactory? facadeFactory, + Future Function() workerFactory = createEpubWorker, + }) : _facadeFactory = facadeFactory, + _workerFactory = workerFactory; final PdfFacadeFactory? _facadeFactory; + final Future Function() _workerFactory; + EpubWorker? _preparingWorker; + int _generation = 0; @override - Set get supportedFormats => const {ReaderFormat.pdf}; + Set get supportedFormats => const { + ReaderFormat.pdf, + ReaderFormat.cbz, + ReaderFormat.cbr, + }; @override ReaderCapabilities get capabilities => const ReaderCapabilities( @@ -46,10 +59,10 @@ final class PdfReaderEngine extends ReaderEngine { ReaderLocator? initialLocator, required ReaderPreferences preferences, }) async { - if (document.format != ReaderFormat.pdf) { + if (!supportedFormats.contains(document.format)) { throw const ReaderException( ReaderErrorCode.invalidDocument, - 'The PDF engine can only load PDF documents.', + 'This engine requires a PDF or comic document.', ); } if (initialLocator != null && initialLocator is! PdfReaderLocator) { @@ -61,10 +74,37 @@ final class PdfReaderEngine extends ReaderEngine { _exposedPages = const []; _viewportReady = null; + final generation = ++_generation; + _preparingWorker?.dispose(); + _preparingWorker = null; PdfFacade? candidate; + EpubWorker? worker; try { - final bytes = await document.loadBytes(); - if (_disposed) { + var bytes = await document.loadBytes(); + if (_disposed || generation != _generation) { + throw const ReaderException( + ReaderErrorCode.engineUnavailable, + 'The document was closed.', + ); + } + if (document.format != ReaderFormat.pdf) { + worker = await _workerFactory(); + if (_disposed || generation != _generation) { + throw const ReaderException( + ReaderErrorCode.engineUnavailable, + 'The document was closed.', + ); + } + _preparingWorker = worker; + final prepared = await worker.request( + 'prepare_${document.format.name}', + bytes, + ); + bytes = prepared['bytes'] as Uint8List; + worker.dispose(); + if (identical(_preparingWorker, worker)) _preparingWorker = null; + } + if (_disposed || generation != _generation) { throw const ReaderException( ReaderErrorCode.engineUnavailable, 'The document was closed.', @@ -72,10 +112,14 @@ final class PdfReaderEngine extends ReaderEngine { } final factory = _facadeFactory ?? - (bytes) => createPdfrxFacade(bytes, sourceName: document.id); + (bytes) => createPdfrxFacade( + bytes, + sourceName: document.id, + preserveColors: document.format != ReaderFormat.pdf, + ); final facade = await factory(bytes); candidate = facade; - if (_disposed) { + if (_disposed || generation != _generation) { throw const ReaderException( ReaderErrorCode.engineUnavailable, 'The document was closed.', @@ -98,7 +142,7 @@ final class PdfReaderEngine extends ReaderEngine { .map((entry) => _tocEntry(entry, facade.pageCount)) .toList(); await facade.showPage(target.pageIndex, target.pageOffset); - if (_disposed) { + if (_disposed || generation != _generation) { throw const ReaderException( ReaderErrorCode.engineUnavailable, 'The document was closed.', @@ -127,7 +171,7 @@ final class PdfReaderEngine extends ReaderEngine { : ReaderErrorCode.invalidDocument, error.error == PdfFacadeError.encrypted ? 'The PDF document is encrypted.' - : 'The PDF document is invalid or malformed.', + : 'The page document is invalid or malformed.', cause: error.cause, ); } catch (error) { @@ -136,9 +180,12 @@ final class PdfReaderEngine extends ReaderEngine { } throw ReaderException( ReaderErrorCode.invalidDocument, - 'The PDF document is invalid or malformed.', + 'The page document is invalid or malformed.', cause: error, ); + } finally { + worker?.dispose(); + if (identical(_preparingWorker, worker)) _preparingWorker = null; } } @@ -387,6 +434,9 @@ final class PdfReaderEngine extends ReaderEngine { void dispose() { if (_disposed) return; _disposed = true; + _generation++; + _preparingWorker?.dispose(); + _preparingWorker = null; final facade = _facade; _facade = null; if (facade != null) _release(facade); diff --git a/lib/src/engine/reader_engine_registry.dart b/lib/src/engine/reader_engine_registry.dart index 4333364..a8988fb 100644 --- a/lib/src/engine/reader_engine_registry.dart +++ b/lib/src/engine/reader_engine_registry.dart @@ -27,11 +27,16 @@ final class ReaderEngineRegistry { factory ReaderEngineRegistry.defaults() { return ReaderEngineRegistry([ ReaderEngineRegistration( - formats: const {ReaderFormat.epub}, + formats: const { + ReaderFormat.epub, + ReaderFormat.mobi, + ReaderFormat.azw3, + ReaderFormat.txt, + }, factory: EpubReaderEngine.new, ), ReaderEngineRegistration( - formats: const {ReaderFormat.pdf}, + formats: const {ReaderFormat.pdf, ReaderFormat.cbr, ReaderFormat.cbz}, factory: PdfReaderEngine.new, ), ]); diff --git a/lib/src/presentation/papyrus_reader.dart b/lib/src/presentation/papyrus_reader.dart index 9fb3caa..1a285e8 100644 --- a/lib/src/presentation/papyrus_reader.dart +++ b/lib/src/presentation/papyrus_reader.dart @@ -469,15 +469,31 @@ final class _PapyrusReaderState extends State { ReaderThemeData theme, ) { final builder = widget.builders.loading; - if (builder != null) { - return builder(context, _stateContext(snapshot)); - } + final content = builder != null + ? builder(context, _stateContext(snapshot)) + : Center( + child: CircularProgressIndicator( + color: theme.accentColor, + semanticsLabel: 'Opening document', + ), + ); - return Center( - child: CircularProgressIndicator( - color: theme.accentColor, - semanticsLabel: 'Opening document', - ), + return Stack( + fit: StackFit.expand, + children: [ + content, + if (widget.onBack != null) + PositionedDirectional( + top: 0, + start: 0, + child: ReaderIconButton( + icon: Icons.arrow_back_rounded, + tooltip: 'Back', + onPressed: widget.onBack, + theme: theme, + ), + ), + ], ); } diff --git a/pubspec.lock b/pubspec.lock index 471b102..f78fbce 100644 --- a/pubspec.lock +++ b/pubspec.lock @@ -2,13 +2,13 @@ # See https://dart.dev/tools/pub/glossary#lockfile packages: archive: - dependency: transitive + dependency: "direct main" description: name: archive - sha256: ace891da0862b0e4cabbb064ee3fd87b2728b898949fdb366d83fe98342c9f19 + sha256: a96e8b390886ee8abb49b7bd3ac8df6f451c621619f52a26e815fdcf568959ff url: "https://pub.dev" source: hosted - version: "4.2.0" + version: "4.0.9" args: dependency: transitive description: @@ -25,6 +25,22 @@ packages: url: "https://pub.dev" source: hosted version: "2.13.1" + barcode: + dependency: transitive + description: + name: barcode + sha256: "7b6729c37e3b7f34233e2318d866e8c48ddb46c1f7ad01ff7bb2a8de1da2b9f4" + url: "https://pub.dev" + source: hosted + version: "2.2.9" + bidi: + dependency: transitive + description: + name: bidi + sha256: "77f475165e94b261745cf1032c751e2032b8ed92ccb2bf5716036db79320637d" + url: "https://pub.dev" + source: hosted + version: "2.0.13" boolean_selector: dependency: transitive description: @@ -169,7 +185,7 @@ packages: source: hosted version: "4.1.2" image: - dependency: transitive + dependency: "direct main" description: name: image sha256: "1976370a4df3091bb0f72409c187ad1f9132a818bc6b95ca59c0bae1c75c688e" @@ -200,6 +216,37 @@ packages: url: "https://pub.dev" source: hosted version: "1.0.0" + kindle_unpack: + dependency: "direct main" + description: + path: "third_party/kindle_unpack" + relative: true + source: path + version: "0.2.0" + koni_archive_core: + dependency: "direct main" + description: + name: koni_archive_core + sha256: f323296866d6224725839e2b8ef25dfeba5e0c2476877fa7726f4b966a4c7a84 + url: "https://pub.dev" + source: hosted + version: "0.9.0" + koni_codecs: + dependency: transitive + description: + name: koni_codecs + sha256: "0a0ffa23f62f627bc50db6de79e88a6a41e019d3a21c6a455ac5ae7c1677a8bd" + url: "https://pub.dev" + source: hosted + version: "0.9.0" + koni_rar: + dependency: "direct main" + description: + name: koni_rar + sha256: c8f1a5b2f1b9d8f6cfd981c67e8c34298f722806f7288f06faabb40004362f56 + url: "https://pub.dev" + source: hosted + version: "0.9.0" leak_tracker: dependency: transitive description: @@ -296,6 +343,14 @@ packages: url: "https://pub.dev" source: hosted version: "1.9.1" + path_parsing: + dependency: transitive + description: + name: path_parsing + sha256: "883402936929eac138ee0a45da5b0f2c80f89913e6dc3bf77eb65b84b409c6ca" + url: "https://pub.dev" + source: hosted + version: "1.1.0" path_provider: dependency: transitive description: @@ -344,6 +399,14 @@ packages: url: "https://pub.dev" source: hosted version: "2.3.0" + pdf: + dependency: "direct main" + description: + name: pdf + sha256: e47a275b267873d5944ad5f5ff0dcc7ac2e36c02b3046a0ffac9b72fd362c44b + url: "https://pub.dev" + source: hosted + version: "3.12.0" pdfium_dart: dependency: transitive description: @@ -416,6 +479,14 @@ packages: url: "https://pub.dev" source: hosted version: "2.2.0" + qr: + dependency: transitive + description: + name: qr + sha256: "2c0189dca64a3ab08dda7acd7911c89af2374f08eb4eaded3e8ef4609f3b9c8b" + url: "https://pub.dev" + source: hosted + version: "3.0.3" record_use: dependency: transitive description: diff --git a/pubspec.yaml b/pubspec.yaml index b11b3f7..9cda5fc 100644 --- a/pubspec.yaml +++ b/pubspec.yaml @@ -8,13 +8,21 @@ environment: flutter: ">=3.47.0" dependencies: + archive: ^4.0.7 characters: ^1.4.1 + kindle_unpack: + path: third_party/kindle_unpack epub_pro: ^5.6.0 flutter: sdk: flutter flutter_html: ^3.0.0 # flutter_html 3.0.0 uses the private matches API removed in html 0.15.7. html: 0.15.6 + # pdf 3.12 requires archive <4.1; image 4.10+ uses newer archive APIs. + image: ">=4.3.0 <4.10.0" + koni_archive_core: ^0.9.0 + koni_rar: ^0.9.0 + pdf: ^3.11.3 pdfrx: ^2.4.8 web: ^1.1.1 diff --git a/test/engine/additional_formats_test.dart b/test/engine/additional_formats_test.dart new file mode 100644 index 0000000..b6e8104 --- /dev/null +++ b/test/engine/additional_formats_test.dart @@ -0,0 +1,353 @@ +import 'dart:async'; +import 'dart:convert'; +import 'dart:io'; +import 'dart:typed_data'; + +import 'package:flutter_test/flutter_test.dart'; +import 'package:kindle_unpack/kindle_unpack.dart'; +import 'package:archive/archive.dart' show ZLibEncoder; +import 'package:papyrus_reader/papyrus_reader.dart'; +import 'package:papyrus_reader/src/engine/epub/worker/epub_worker.dart'; +import 'package:pdfrx/pdfrx.dart'; + +import '../support/pdfium_test_setup.dart'; + +void main() { + TestWidgetsFlutterBinding.ensureInitialized(); + setUpAll(initializePdfiumForTest); + + for (final format in [ + ReaderFormat.txt, + ReaderFormat.mobi, + ReaderFormat.azw3, + ]) { + test( + '$format opens real text, navigates and restores a serialized position', + () async { + final bytes = format == ReaderFormat.txt + ? utf8.encode( + List.filled( + 3000, + 'Alice read the book. Ελληνικά 日本語 🐇\n\n', + ).join(), + ) + : await File( + 'test/fixtures/gutenberg/alice.${format.name}', + ).readAsBytes(); + final document = ReaderDocument( + id: format.name, + format: format, + loadBytes: () async => bytes, + ); + final engine = + ReaderEngineRegistry.defaults().resolve(format) as EpubReaderEngine; + addTearDown(engine.dispose); + var ticks = 0; + final heartbeat = Timer.periodic( + const Duration(milliseconds: 1), + (_) => ticks++, + ); + await engine.load(document, preferences: const ReaderPreferences()); + heartbeat.cancel(); + expect( + ticks, + greaterThan(0), + reason: 'Opening must allow the host event loop to advance', + ); + expect(engine.contentLength, greaterThan(0)); + await engine.goToProgress(.5); + expect(engine.currentChapterHtml, contains('Alice')); + final locator = ReaderLocator.fromJson( + (await engine.currentLocator())!.toJson(), + ); + final reopened = + ReaderEngineRegistry.defaults().resolve(format) as EpubReaderEngine; + addTearDown(reopened.dispose); + await reopened.load( + document, + initialLocator: locator, + preferences: const ReaderPreferences(), + ); + expect(await reopened.currentLocator(), locator); + expect(reopened.currentChapterHtml, engine.currentChapterHtml); + await reopened.goToProgress(1); + expect(reopened.contentLength, greaterThan(0)); + }, + ); + } + + for (final fixture in [ + 'pages.cbz', + 'pages-rar4.cbr', + 'pages-rar4-compressed.cbr', + 'pages-rar5.cbr', + ]) { + test( + '$fixture decodes real pages in natural order and opens in PDFium', + () async { + final worker = await createEpubWorker(); + addTearDown(worker.dispose); + final bytes = await File('test/fixtures/comics/$fixture').readAsBytes(); + var ticks = 0; + final heartbeat = Timer.periodic( + const Duration(milliseconds: 1), + (_) => ticks++, + ); + final converted = await worker.request( + fixture.endsWith('cbz') ? 'prepare_cbz' : 'prepare_cbr', + bytes, + ); + heartbeat.cancel(); + expect(ticks, greaterThan(0)); + final pdf = await PdfDocument.openData(converted['bytes'] as Uint8List); + addTearDown(pdf.dispose); + expect(pdf.pages, hasLength(3)); + + for (var index = 0; index < 3; index++) { + final page = pdf.pages[index]; + expect(page.width, 320); + expect(page.height, 480); + final rendered = await page.render(width: 32, height: 48); + expect(rendered, isNotNull); + final pixel = rendered!.pixels.sublist(0, 4); + // PDFium pixels are BGRA: red, green, blue prove natural page order. + expect(pixel[2 - index], greaterThan(150)); + expect(pixel[2 - ((index + 1) % 3)], lessThan(30)); + rendered.dispose(); + } + + final engine = ReaderEngineRegistry.defaults().resolve( + fixture.endsWith('cbz') ? ReaderFormat.cbz : ReaderFormat.cbr, + ); + addTearDown(engine.dispose); + final location = PdfReaderLocator( + pageIndex: 1, + pageOffset: .2, + totalProgression: .5, + ); + await engine.load( + ReaderDocument( + id: fixture, + format: fixture.endsWith('cbz') + ? ReaderFormat.cbz + : ReaderFormat.cbr, + loadBytes: () async => bytes, + ), + initialLocator: ReaderLocator.fromJson(location.toJson()), + preferences: const ReaderPreferences(), + ); + expect(await engine.currentLocator(), location); + await engine.goNext(); + expect( + (await engine.currentLocator() as PdfReaderLocator).pageIndex, + 2, + ); + await engine.goPrevious(); + expect( + (await engine.currentLocator() as PdfReaderLocator).pageIndex, + 1, + ); + }, + ); + } + + test( + 'text decoding preserves UTF-16 and Windows-1252 without interpreting markup', + () async { + final worker = await createEpubWorker(); + addTearDown(worker.dispose); + const text = '日本語 🐇 '; + final bytes = ByteData(2 + text.length * 2) + ..setUint16(0, 0xfeff, Endian.little); + + for (var index = 0; index < text.length; index++) { + bytes.setUint16(2 + index * 2, text.codeUnitAt(index), Endian.little); + } + + await worker.request('open_txt', bytes.buffer.asUint8List()); + var chapter = await worker.chapter(0); + expect( + chapter['html'], + contains('<script>literal text</script>'), + ); + expect(chapter['html'], contains('日本語 🐇')); + await worker.request( + 'open_txt', + Uint8List.fromList([0x93, 0x41, 0x94, 0x20, 0x80]), + ); + chapter = await worker.chapter(0); + expect(chapter['html'], contains('“A” €')); + }, + ); + + test('Kindle embedded font inflation uses the portable decoder', () { + final font = Uint8List.fromList([0, 1, 0, 0, ...List.filled(128, 42)]); + final compressed = ZLibEncoder().encodeBytes(font); + final record = Uint8List(24 + compressed.length); + record.setRange(0, 4, 'FONT'.codeUnits); + final header = ByteData.sublistView(record); + header.setUint32(4, font.length); + header.setUint32(8, 1); + header.setUint32(12, 24); + record.setRange(24, record.length, compressed); + expect(FontResource.parse(record).payload, font); + }); + + for (final pointsAtSentinel in [true, false]) { + test( + 'dual MOBI/KF8 uses relative section records, sentinel=$pointsAtSentinel', + () async { + final legacy = await File( + 'test/fixtures/gutenberg/alice.mobi', + ).readAsBytes(); + final modern = await File( + 'test/fixtures/gutenberg/alice.azw3', + ).readAsBytes(); + final bytes = _dualKindle( + legacy, + modern, + pointsAtSentinel: pointsAtSentinel, + ); + final book = KindleBook.fromBytes(bytes); + expect(book.format, KindleFormat.combo); + expect(book.parts.length, greaterThan(10)); + expect(utf8.decode(book.rawML), contains('Alice')); + final worker = await createEpubWorker(); + addTearDown(worker.dispose); + final metadata = await worker.request('open_mobi', bytes); + expect(metadata['count'], 19); + final cover = await worker.chapter(0); + expect( + (cover['blocks'] as List).where((block) => block['kind'] == 'image'), + isNotEmpty, + ); + }, + ); + } + + test('encrypted Kindle files report a specific error', () async { + final bytes = await File( + 'test/fixtures/gutenberg/alice.mobi', + ).readAsBytes(); + final header = ByteData.sublistView(bytes); + header.setUint16(header.getUint32(78) + 12, 2); + final worker = await createEpubWorker(); + addTearDown(worker.dispose); + await expectLater( + worker.request('open_mobi', bytes), + throwsA( + isA().having( + (error) => error.code, + 'code', + ReaderErrorCode.encryptedDocument, + ), + ), + ); + }); + + test('PDF opening leaves the host event loop running', () async { + final engine = PdfReaderEngine(); + addTearDown(engine.dispose); + final bytes = await File('example/assets/a_tiny_pdf.pdf').readAsBytes(); + var ticks = 0; + final heartbeat = Timer.periodic( + const Duration(milliseconds: 1), + (_) => ticks++, + ); + await engine.load( + ReaderDocument( + id: 'pdf', + format: ReaderFormat.pdf, + loadBytes: () async => bytes, + ), + preferences: const ReaderPreferences(), + ); + heartbeat.cancel(); + expect(ticks, greaterThan(0)); + expect(engine.snapshot.status, ReaderStatus.ready); + }); + + test( + 'worker rejects empty comics, malformed Kindle books and empty text', + () async { + for (final command in [ + 'prepare_cbz', + 'prepare_cbr', + 'open_mobi', + 'open_azw3', + 'open_txt', + ]) { + final worker = await createEpubWorker(); + addTearDown(worker.dispose); + await expectLater( + worker.request(command, Uint8List(0)), + throwsA(isA()), + ); + } + }, + ); + + test('closing the worker cancels in-flight format decoding', () async { + final worker = await createEpubWorker(); + final opening = worker.request( + 'open_txt', + utf8.encode(List.filled(100000, 'Long text\n\n').join()), + ); + final expectation = expectLater(opening, throwsA(isA())); + worker.dispose(); + await expectation; + }); +} + +// Combine the independent real editions without changing their text records. +// This deliberately tests section-relative record indices and shared images. +Uint8List _dualKindle( + Uint8List legacy, + Uint8List modern, { + required bool pointsAtSentinel, +}) { + final first = PdbFile.parse(legacy); + final second = PdbFile.parse(modern); + final original = first.records.first.data; + final originalHeader = ByteData.sublistView(original); + final exth = 16 + originalHeader.getUint32(20); + final count = originalHeader.getUint32(exth + 8); + var insertAt = exth + 12; + + for (var index = 0; index < count; index++) { + insertAt += originalHeader.getUint32(insertAt + 4); + } + + final record = Uint8List(original.length + 12); + record.setRange(0, insertAt, original); + record.setRange(insertAt + 12, record.length, original, insertAt); + final header = ByteData.sublistView(record); + header.setUint32(insertAt, 121); + header.setUint32(insertAt + 4, 12); + header.setUint32( + insertAt + 8, + first.records.length + (pointsAtSentinel ? 0 : 1), + ); + header.setUint32(exth + 4, originalHeader.getUint32(exth + 4) + 12); + header.setUint32(exth + 8, count + 1); + header.setUint32(84, originalHeader.getUint32(84) + 12); + final records = [ + record, + ...first.records.skip(1).map((record) => record.data), + utf8.encode('BOUNDARY'), + ...second.records.map((record) => record.data), + ]; + final table = Uint8List(78 + records.length * 8 + 2)..setRange(0, 78, legacy); + final offsets = ByteData.sublistView(table)..setUint16(76, records.length); + var offset = table.length; + + for (var index = 0; index < records.length; index++) { + offsets.setUint32(78 + index * 8, offset); + offset += records[index].length; + } + + return (BytesBuilder(copy: false) + ..add(table) + ..add(records.expand((record) => record).toList())) + .takeBytes(); +} diff --git a/test/fixtures/comics/README.md b/test/fixtures/comics/README.md new file mode 100644 index 0000000..a16b0ce --- /dev/null +++ b/test/fixtures/comics/README.md @@ -0,0 +1,16 @@ +# Comic fixtures + +These owned, generated pages use red, green and blue to verify decoded pixels +and natural filename ordering (`page1`, `page2`, `page10`). Regenerate PNG, CBZ, +and RAR4 files with `python3 test/support/generate_comic_fixtures.py`. + +`pages-rar5.cbr` uses RAR 7.23, maximum compression and solid mode: + +``` +rar a -ep -m5 -s pages-rar5.cbr page10.png page2.png page1.png +``` + +The RAR binary is only a fixture-generation tool, not a runtime dependency. + +`pages-rar4-compressed.cbr` uses RAR 6.24 with `-ma4 -ep -m5 -s`, exercising +compressed, solid RAR4 in addition to the generated stored archive. diff --git a/test/fixtures/comics/page1.png b/test/fixtures/comics/page1.png new file mode 100644 index 0000000..1e06924 Binary files /dev/null and b/test/fixtures/comics/page1.png differ diff --git a/test/fixtures/comics/page10.png b/test/fixtures/comics/page10.png new file mode 100644 index 0000000..9223244 Binary files /dev/null and b/test/fixtures/comics/page10.png differ diff --git a/test/fixtures/comics/page2.png b/test/fixtures/comics/page2.png new file mode 100644 index 0000000..0a0e1fd Binary files /dev/null and b/test/fixtures/comics/page2.png differ diff --git a/test/fixtures/comics/pages-rar4-compressed.cbr b/test/fixtures/comics/pages-rar4-compressed.cbr new file mode 100644 index 0000000..a3a3ab9 Binary files /dev/null and b/test/fixtures/comics/pages-rar4-compressed.cbr differ diff --git a/test/fixtures/comics/pages-rar4.cbr b/test/fixtures/comics/pages-rar4.cbr new file mode 100644 index 0000000..74dae00 Binary files /dev/null and b/test/fixtures/comics/pages-rar4.cbr differ diff --git a/test/fixtures/comics/pages-rar5.cbr b/test/fixtures/comics/pages-rar5.cbr new file mode 100644 index 0000000..64a7c88 Binary files /dev/null and b/test/fixtures/comics/pages-rar5.cbr differ diff --git a/test/fixtures/comics/pages.cbz b/test/fixtures/comics/pages.cbz new file mode 100644 index 0000000..1835406 Binary files /dev/null and b/test/fixtures/comics/pages.cbz differ diff --git a/test/fixtures/gutenberg/README.md b/test/fixtures/gutenberg/README.md index f536c93..e2e37f9 100644 --- a/test/fixtures/gutenberg/README.md +++ b/test/fixtures/gutenberg/README.md @@ -16,3 +16,10 @@ without adding the 24 MB illustrated edition to the repository. The user's SQL Performance Explained EPUB is a local-only validation input and must not be copied into this fixture directory. + +The Kindle fixtures were retrieved for format decoding regressions on 2026-11-08: + +| File | Source | SHA-256 | +| --- | --- | --- | +| `alice.mobi` | [Older Kindle edition](https://www.gutenberg.org/ebooks/11.kindle.images) | `1b4222ffdf1862a39d0fae6d42c588f28a34c12ad73cc21c2849fe2646cac3e1` | +| `alice.azw3` | [KF8 edition](https://www.gutenberg.org/ebooks/11.kf8.images) | `d8211269aed55ded8302f289950936db8368b4d4d384c463b722034152f8f1d2` | diff --git a/test/fixtures/gutenberg/alice.azw3 b/test/fixtures/gutenberg/alice.azw3 new file mode 100644 index 0000000..2f3e6e3 Binary files /dev/null and b/test/fixtures/gutenberg/alice.azw3 differ diff --git a/test/fixtures/gutenberg/alice.mobi b/test/fixtures/gutenberg/alice.mobi new file mode 100644 index 0000000..d177b00 Binary files /dev/null and b/test/fixtures/gutenberg/alice.mobi differ diff --git a/test/presentation/papyrus_reader_test.dart b/test/presentation/papyrus_reader_test.dart index df30114..bbc7fc0 100644 --- a/test/presentation/papyrus_reader_test.dart +++ b/test/presentation/papyrus_reader_test.dart @@ -698,17 +698,20 @@ void main() { final gate = Completer(); final engine = _UiReaderEngine(loadGate: gate); final controller = controllerFor([engine]); + var backCalls = 0; addTearDown(controller.dispose); await pumpReader( tester, document: document('Loading'), controller: controller, - onBack: () {}, + onBack: () => backCalls++, ); expect(find.byType(CircularProgressIndicator), findsOneWidget); - expect(find.byTooltip('Back'), findsNothing); + expect(find.byTooltip('Back'), findsOneWidget); + await tester.tap(find.byTooltip('Back')); + expect(backCalls, 1); expect(find.text('Opening The Loading Reader…'), findsNothing); expect( tester.getCenter(find.byType(CircularProgressIndicator)), diff --git a/test/stress/large_books_test.dart b/test/stress/large_books_test.dart new file mode 100644 index 0000000..d7503a0 --- /dev/null +++ b/test/stress/large_books_test.dart @@ -0,0 +1,91 @@ +import 'dart:async'; +import 'dart:convert'; +import 'dart:io'; + +import 'package:flutter_test/flutter_test.dart'; +import 'package:papyrus_reader/papyrus_reader.dart'; + +import '../support/pdfium_test_setup.dart'; + +// Opt in with PAPYRUS_LARGE_BOOKS pointing to the downloaded stress corpus. +void main() { + TestWidgetsFlutterBinding.ensureInitialized(); + final directory = Platform.environment['PAPYRUS_LARGE_BOOKS']; + const cases = { + 'websters-dictionary.epub': ReaderFormat.epub, + 'websters-dictionary.txt': ReaderFormat.txt, + 'websters-dictionary.mobi': ReaderFormat.mobi, + 'websters-dictionary.azw3': ReaderFormat.azw3, + 'alice-high-resolution.pdf': ReaderFormat.pdf, + 'whiz-272-pages.cbz': ReaderFormat.cbz, + 'whiz-272-pages.cbr': ReaderFormat.cbr, + }; + + group( + 'large real-book corpus', + () { + setUpAll(initializePdfiumForTest); + + for (final entry in cases.entries) { + test(entry.key, () async { + final document = ReaderDocument( + id: entry.key, + format: entry.value, + loadBytes: () => File('$directory/${entry.key}').readAsBytes(), + ); + final engine = ReaderEngineRegistry.defaults().resolve(entry.value); + addTearDown(engine.dispose); + final clock = Stopwatch()..start(); + var ticks = 0; + var lastTick = 0; + var longestGap = 0; + var peakRss = ProcessInfo.currentRss; + final heartbeat = Timer.periodic(const Duration(milliseconds: 16), ( + _, + ) { + final now = clock.elapsedMilliseconds; + final gap = now - lastTick; + if (gap > longestGap) longestGap = gap; + lastTick = now; + ticks++; + final rss = ProcessInfo.currentRss; + if (rss > peakRss) peakRss = rss; + }); + addTearDown(heartbeat.cancel); + + await engine.load(document, preferences: const ReaderPreferences()); + final openMs = clock.elapsedMilliseconds; + expect(engine.snapshot.status, ReaderStatus.ready); + expect(ticks, greaterThan(0)); + await engine.goToProgress(.9); + final locator = (await engine.currentLocator())!; + final serialized = ReaderLocator.fromJson(locator.toJson()); + expect(serialized, locator); + final progress = switch (locator) { + EpubReaderLocator() => locator.totalProgression, + PdfReaderLocator() => locator.totalProgression, + }; + expect(progress, closeTo(.9, .02)); + await engine.goToProgress(.1); + await engine.goTo(serialized); + expect(await engine.currentLocator(), locator); + expect(engine.snapshot.status, ReaderStatus.ready); + heartbeat.cancel(); + stdout.writeln( + jsonEncode({ + 'file': entry.key, + 'openMs': openMs, + 'openAndSeekMs': clock.elapsedMilliseconds, + 'heartbeatTicks': ticks, + 'longestEventLoopGapMs': longestGap, + 'peakTestProcessRssMiB': peakRss / (1024 * 1024), + }), + ); + }, timeout: const Timeout(Duration(minutes: 5))); + } + }, + skip: directory == null + ? 'Set PAPYRUS_LARGE_BOOKS to run the opt-in corpus.' + : false, + ); +} diff --git a/test/support/generate_comic_fixtures.py b/test/support/generate_comic_fixtures.py new file mode 100644 index 0000000..4576fb0 --- /dev/null +++ b/test/support/generate_comic_fixtures.py @@ -0,0 +1,46 @@ +"""Generate owned PNG pages, CBZ, and stored RAR4 fixtures (stdlib only). + +For a compressed RAR5 fixture, run RAR against the generated pages: + rar a -ep -m5 -s pages-rar5.cbr page10.png page2.png page1.png +""" +import binascii +from pathlib import Path +import struct +import zlib +import zipfile + +root = Path(__file__).resolve().parents[1] / 'fixtures' / 'comics' +root.mkdir(exist_ok=True) + + +def png_chunk(kind, data): + return struct.pack('>I', len(data)) + kind + data + struct.pack('>I', binascii.crc32(kind + data)) + + +def header(kind, flags, body): + value = struct.pack('IIBBBBB', width, height, 8, 2, 0, 0, 0)) + data += png_chunk(b'IDAT', zlib.compress(raw)) + png_chunk(b'IEND', b'') + pages[name] = data + (root / name).write_bytes(data) + +with zipfile.ZipFile(root / 'pages.cbz', 'w', zipfile.ZIP_DEFLATED) as archive: + for name, data in pages.items(): + archive.writestr(name, data) + archive.writestr('__MACOSX/._page0.png', b'not an image') + archive.writestr('ComicInfo.xml', 'Test comic') + +rar = b'Rar!\x1a\x07\0' + header(0x73, 0, b'\0' * 6) +for name, data in pages.items(): + encoded = name.encode() + body = struct.pack(' + Everyone is permitted to copy and distribute verbatim copies + of this license document, but changing it is not allowed. + + Preamble + + The GNU General Public License is a free, copyleft license for +software and other kinds of works. + + The licenses for most software and other practical works are designed +to take away your freedom to share and change the works. By contrast, +the GNU General Public License is intended to guarantee your freedom to +share and change all versions of a program--to make sure it remains free +software for all its users. We, the Free Software Foundation, use the +GNU General Public License for most of our software; it applies also to +any other work released this way by its authors. You can apply it to +your programs, too. + + When we speak of free software, we are referring to freedom, not +price. Our General Public Licenses are designed to make sure that you +have the freedom to distribute copies of free software (and charge for +them if you wish), that you receive source code or can get it if you +want it, that you can change the software or use pieces of it in new +free programs, and that you know you can do these things. + + To protect your rights, we need to prevent others from denying you +these rights or asking you to surrender the rights. Therefore, you have +certain responsibilities if you distribute copies of the software, or if +you modify it: responsibilities to respect the freedom of others. + + For example, if you distribute copies of such a program, whether +gratis or for a fee, you must pass on to the recipients the same +freedoms that you received. You must make sure that they, too, receive +or can get the source code. And you must show them these terms so they +know their rights. + + Developers that use the GNU GPL protect your rights with two steps: +(1) assert copyright on the software, and (2) offer you this License +giving you legal permission to copy, distribute and/or modify it. + + For the developers' and authors' protection, the GPL clearly explains +that there is no warranty for this free software. For both users' and +authors' sake, the GPL requires that modified versions be marked as +changed, so that their problems will not be attributed erroneously to +authors of previous versions. + + Some devices are designed to deny users access to install or run +modified versions of the software inside them, although the manufacturer +can do so. This is fundamentally incompatible with the aim of +protecting users' freedom to change the software. The systematic +pattern of such abuse occurs in the area of products for individuals to +use, which is precisely where it is most unacceptable. Therefore, we +have designed this version of the GPL to prohibit the practice for those +products. If such problems arise substantially in other domains, we +stand ready to extend this provision to those domains in future versions +of the GPL, as needed to protect the freedom of users. + + Finally, every program is threatened constantly by software patents. +States should not allow patents to restrict development and use of +software on general-purpose computers, but in those that do, we wish to +avoid the special danger that patents applied to a free program could +make it effectively proprietary. To prevent this, the GPL assures that +patents cannot be used to render the program non-free. + + The precise terms and conditions for copying, distribution and +modification follow. + + TERMS AND CONDITIONS + + 0. Definitions. + + "This License" refers to version 3 of the GNU General Public License. + + "Copyright" also means copyright-like laws that apply to other kinds of +works, such as semiconductor masks. + + "The Program" refers to any copyrightable work licensed under this +License. Each licensee is addressed as "you". "Licensees" and +"recipients" may be individuals or organizations. + + To "modify" a work means to copy from or adapt all or part of the work +in a fashion requiring copyright permission, other than the making of an +exact copy. The resulting work is called a "modified version" of the +earlier work or a work "based on" the earlier work. + + A "covered work" means either the unmodified Program or a work based +on the Program. + + To "propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification), making available to the +public, and in some countries other activities as well. + + To "convey" a work means any kind of propagation that enables other +parties to make or receive copies. Mere interaction with a user through +a computer network, with no transfer of a copy, is not conveying. + + An interactive user interface displays "Appropriate Legal Notices" +to the extent that it includes a convenient and prominently visible +feature that (1) displays an appropriate copyright notice, and (2) +tells the user that there is no warranty for the work (except to the +extent that warranties are provided), that licensees may convey the +work under this License, and how to view a copy of this License. If +the interface presents a list of user commands or options, such as a +menu, a prominent item in the list meets this criterion. + + 1. Source Code. + + The "source code" for a work means the preferred form of the work +for making modifications to it. "Object code" means any non-source +form of a work. + + A "Standard Interface" means an interface that either is an official +standard defined by a recognized standards body, or, in the case of +interfaces specified for a particular programming language, one that +is widely used among developers working in that language. + + The "System Libraries" of an executable work include anything, other +than the work as a whole, that (a) is included in the normal form of +packaging a Major Component, but which is not part of that Major +Component, and (b) serves only to enable use of the work with that +Major Component, or to implement a Standard Interface for which an +implementation is available to the public in source code form. A +"Major Component", in this context, means a major essential component +(kernel, window system, and so on) of the specific operating system +(if any) on which the executable work runs, or a compiler used to +produce the work, or an object code interpreter used to run it. + + The "Corresponding Source" for a work in object code form means all +the source code needed to generate, install, and (for an executable +work) run the object code and to modify the work, including scripts to +control those activities. However, it does not include the work's +System Libraries, or general-purpose tools or generally available free +programs which are used unmodified in performing those activities but +which are not part of the work. For example, Corresponding Source +includes interface definition files associated with source files for +the work, and the source code for shared libraries and dynamically +linked subprograms that the work is specifically designed to require, +such as by intimate data communication or control flow between those +subprograms and other parts of the work. + + The Corresponding Source need not include anything that users +can regenerate automatically from other parts of the Corresponding +Source. + + The Corresponding Source for a work in source code form is that +same work. + + 2. Basic Permissions. + + All rights granted under this License are granted for the term of +copyright on the Program, and are irrevocable provided the stated +conditions are met. This License explicitly affirms your unlimited +permission to run the unmodified Program. The output from running a +covered work is covered by this License only if the output, given its +content, constitutes a covered work. This License acknowledges your +rights of fair use or other equivalent, as provided by copyright law. + + You may make, run and propagate covered works that you do not +convey, without conditions so long as your license otherwise remains +in force. You may convey covered works to others for the sole purpose +of having them make modifications exclusively for you, or provide you +with facilities for running those works, provided that you comply with +the terms of this License in conveying all material for which you do +not control copyright. Those thus making or running the covered works +for you must do so exclusively on your behalf, under your direction +and control, on terms that prohibit them from making any copies of +your copyrighted material outside their relationship with you. + + Conveying under any other circumstances is permitted solely under +the conditions stated below. Sublicensing is not allowed; section 10 +makes it unnecessary. + + 3. Protecting Users' Legal Rights From Anti-Circumvention Law. + + No covered work shall be deemed part of an effective technological +measure under any applicable law fulfilling obligations under article +11 of the WIPO copyright treaty adopted on 20 December 1996, or +similar laws prohibiting or restricting circumvention of such +measures. + + When you convey a covered work, you waive any legal power to forbid +circumvention of technological measures to the extent such circumvention +is effected by exercising rights under this License with respect to +the covered work, and you disclaim any intention to limit operation or +modification of the work as a means of enforcing, against the work's +users, your or third parties' legal rights to forbid circumvention of +technological measures. + + 4. Conveying Verbatim Copies. + + You may convey verbatim copies of the Program's source code as you +receive it, in any medium, provided that you conspicuously and +appropriately publish on each copy an appropriate copyright notice; +keep intact all notices stating that this License and any +non-permissive terms added in accord with section 7 apply to the code; +keep intact all notices of the absence of any warranty; and give all +recipients a copy of this License along with the Program. + + You may charge any price or no price for each copy that you convey, +and you may offer support or warranty protection for a fee. + + 5. Conveying Modified Source Versions. + + You may convey a work based on the Program, or the modifications to +produce it from the Program, in the form of source code under the +terms of section 4, provided that you also meet all of these conditions: + + a) The work must carry prominent notices stating that you modified + it, and giving a relevant date. + + b) The work must carry prominent notices stating that it is + released under this License and any conditions added under section + 7. This requirement modifies the requirement in section 4 to + "keep intact all notices". + + c) You must license the entire work, as a whole, under this + License to anyone who comes into possession of a copy. This + License will therefore apply, along with any applicable section 7 + additional terms, to the whole of the work, and all its parts, + regardless of how they are packaged. This License gives no + permission to license the work in any other way, but it does not + invalidate such permission if you have separately received it. + + d) If the work has interactive user interfaces, each must display + Appropriate Legal Notices; however, if the Program has interactive + interfaces that do not display Appropriate Legal Notices, your + work need not make them do so. + + A compilation of a covered work with other separate and independent +works, which are not by their nature extensions of the covered work, +and which are not combined with it such as to form a larger program, +in or on a volume of a storage or distribution medium, is called an +"aggregate" if the compilation and its resulting copyright are not +used to limit the access or legal rights of the compilation's users +beyond what the individual works permit. Inclusion of a covered work +in an aggregate does not cause this License to apply to the other +parts of the aggregate. + + 6. Conveying Non-Source Forms. + + You may convey a covered work in object code form under the terms +of sections 4 and 5, provided that you also convey the +machine-readable Corresponding Source under the terms of this License, +in one of these ways: + + a) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by the + Corresponding Source fixed on a durable physical medium + customarily used for software interchange. + + b) Convey the object code in, or embodied in, a physical product + (including a physical distribution medium), accompanied by a + written offer, valid for at least three years and valid for as + long as you offer spare parts or customer support for that product + model, to give anyone who possesses the object code either (1) a + copy of the Corresponding Source for all the software in the + product that is covered by this License, on a durable physical + medium customarily used for software interchange, for a price no + more than your reasonable cost of physically performing this + conveying of source, or (2) access to copy the + Corresponding Source from a network server at no charge. + + c) Convey individual copies of the object code with a copy of the + written offer to provide the Corresponding Source. This + alternative is allowed only occasionally and noncommercially, and + only if you received the object code with such an offer, in accord + with subsection 6b. + + d) Convey the object code by offering access from a designated + place (gratis or for a charge), and offer equivalent access to the + Corresponding Source in the same way through the same place at no + further charge. You need not require recipients to copy the + Corresponding Source along with the object code. If the place to + copy the object code is a network server, the Corresponding Source + may be on a different server (operated by you or a third party) + that supports equivalent copying facilities, provided you maintain + clear directions next to the object code saying where to find the + Corresponding Source. Regardless of what server hosts the + Corresponding Source, you remain obligated to ensure that it is + available for as long as needed to satisfy these requirements. + + e) Convey the object code using peer-to-peer transmission, provided + you inform other peers where the object code and Corresponding + Source of the work are being offered to the general public at no + charge under subsection 6d. + + A separable portion of the object code, whose source code is excluded +from the Corresponding Source as a System Library, need not be +included in conveying the object code work. + + A "User Product" is either (1) a "consumer product", which means any +tangible personal property which is normally used for personal, family, +or household purposes, or (2) anything designed or sold for incorporation +into a dwelling. In determining whether a product is a consumer product, +doubtful cases shall be resolved in favor of coverage. For a particular +product received by a particular user, "normally used" refers to a +typical or common use of that class of product, regardless of the status +of the particular user or of the way in which the particular user +actually uses, or expects or is expected to use, the product. A product +is a consumer product regardless of whether the product has substantial +commercial, industrial or non-consumer uses, unless such uses represent +the only significant mode of use of the product. + + "Installation Information" for a User Product means any methods, +procedures, authorization keys, or other information required to install +and execute modified versions of a covered work in that User Product from +a modified version of its Corresponding Source. The information must +suffice to ensure that the continued functioning of the modified object +code is in no case prevented or interfered with solely because +modification has been made. + + If you convey an object code work under this section in, or with, or +specifically for use in, a User Product, and the conveying occurs as +part of a transaction in which the right of possession and use of the +User Product is transferred to the recipient in perpetuity or for a +fixed term (regardless of how the transaction is characterized), the +Corresponding Source conveyed under this section must be accompanied +by the Installation Information. But this requirement does not apply +if neither you nor any third party retains the ability to install +modified object code on the User Product (for example, the work has +been installed in ROM). + + The requirement to provide Installation Information does not include a +requirement to continue to provide support service, warranty, or updates +for a work that has been modified or installed by the recipient, or for +the User Product in which it has been modified or installed. Access to a +network may be denied when the modification itself materially and +adversely affects the operation of the network or violates the rules and +protocols for communication across the network. + + Corresponding Source conveyed, and Installation Information provided, +in accord with this section must be in a format that is publicly +documented (and with an implementation available to the public in +source code form), and must require no special password or key for +unpacking, reading or copying. + + 7. Additional Terms. + + "Additional permissions" are terms that supplement the terms of this +License by making exceptions from one or more of its conditions. +Additional permissions that are applicable to the entire Program shall +be treated as though they were included in this License, to the extent +that they are valid under applicable law. If additional permissions +apply only to part of the Program, that part may be used separately +under those permissions, but the entire Program remains governed by +this License without regard to the additional permissions. + + When you convey a copy of a covered work, you may at your option +remove any additional permissions from that copy, or from any part of +it. (Additional permissions may be written to require their own +removal in certain cases when you modify the work.) You may place +additional permissions on material, added by you to a covered work, +for which you have or can give appropriate copyright permission. + + Notwithstanding any other provision of this License, for material you +add to a covered work, you may (if authorized by the copyright holders of +that material) supplement the terms of this License with terms: + + a) Disclaiming warranty or limiting liability differently from the + terms of sections 15 and 16 of this License; or + + b) Requiring preservation of specified reasonable legal notices or + author attributions in that material or in the Appropriate Legal + Notices displayed by works containing it; or + + c) Prohibiting misrepresentation of the origin of that material, or + requiring that modified versions of such material be marked in + reasonable ways as different from the original version; or + + d) Limiting the use for publicity purposes of names of licensors or + authors of the material; or + + e) Declining to grant rights under trademark law for use of some + trade names, trademarks, or service marks; or + + f) Requiring indemnification of licensors and authors of that + material by anyone who conveys the material (or modified versions of + it) with contractual assumptions of liability to the recipient, for + any liability that these contractual assumptions directly impose on + those licensors and authors. + + All other non-permissive additional terms are considered "further +restrictions" within the meaning of section 10. If the Program as you +received it, or any part of it, contains a notice stating that it is +governed by this License along with a term that is a further +restriction, you may remove that term. If a license document contains +a further restriction but permits relicensing or conveying under this +License, you may add to a covered work material governed by the terms +of that license document, provided that the further restriction does +not survive such relicensing or conveying. + + If you add terms to a covered work in accord with this section, you +must place, in the relevant source files, a statement of the +additional terms that apply to those files, or a notice indicating +where to find the applicable terms. + + Additional terms, permissive or non-permissive, may be stated in the +form of a separately written license, or stated as exceptions; +the above requirements apply either way. + + 8. Termination. + + You may not propagate or modify a covered work except as expressly +provided under this License. Any attempt otherwise to propagate or +modify it is void, and will automatically terminate your rights under +this License (including any patent licenses granted under the third +paragraph of section 11). + + However, if you cease all violation of this License, then your +license from a particular copyright holder is reinstated (a) +provisionally, unless and until the copyright holder explicitly and +finally terminates your license, and (b) permanently, if the copyright +holder fails to notify you of the violation by some reasonable means +prior to 60 days after the cessation. + + Moreover, your license from a particular copyright holder is +reinstated permanently if the copyright holder notifies you of the +violation by some reasonable means, this is the first time you have +received notice of violation of this License (for any work) from that +copyright holder, and you cure the violation prior to 30 days after +your receipt of the notice. + + Termination of your rights under this section does not terminate the +licenses of parties who have received copies or rights from you under +this License. If your rights have been terminated and not permanently +reinstated, you do not qualify to receive new licenses for the same +material under section 10. + + 9. Acceptance Not Required for Having Copies. + + You are not required to accept this License in order to receive or +run a copy of the Program. Ancillary propagation of a covered work +occurring solely as a consequence of using peer-to-peer transmission +to receive a copy likewise does not require acceptance. However, +nothing other than this License grants you permission to propagate or +modify any covered work. These actions infringe copyright if you do +not accept this License. Therefore, by modifying or propagating a +covered work, you indicate your acceptance of this License to do so. + + 10. Automatic Licensing of Downstream Recipients. + + Each time you convey a covered work, the recipient automatically +receives a license from the original licensors, to run, modify and +propagate that work, subject to this License. You are not responsible +for enforcing compliance by third parties with this License. + + An "entity transaction" is a transaction transferring control of an +organization, or substantially all assets of one, or subdividing an +organization, or merging organizations. If propagation of a covered +work results from an entity transaction, each party to that +transaction who receives a copy of the work also receives whatever +licenses to the work the party's predecessor in interest had or could +give under the previous paragraph, plus a right to possession of the +Corresponding Source of the work from the predecessor in interest, if +the predecessor has it or can get it with reasonable efforts. + + You may not impose any further restrictions on the exercise of the +rights granted or affirmed under this License. For example, you may +not impose a license fee, royalty, or other charge for exercise of +rights granted under this License, and you may not initiate litigation +(including a cross-claim or counterclaim in a lawsuit) alleging that +any patent claim is infringed by making, using, selling, offering for +sale, or importing the Program or any portion of it. + + 11. Patents. + + A "contributor" is a copyright holder who authorizes use under this +License of the Program or a work on which the Program is based. The +work thus licensed is called the contributor's "contributor version". + + A contributor's "essential patent claims" are all patent claims +owned or controlled by the contributor, whether already acquired or +hereafter acquired, that would be infringed by some manner, permitted +by this License, of making, using, or selling its contributor version, +but do not include claims that would be infringed only as a +consequence of further modification of the contributor version. For +purposes of this definition, "control" includes the right to grant +patent sublicenses in a manner consistent with the requirements of +this License. + + Each contributor grants you a non-exclusive, worldwide, royalty-free +patent license under the contributor's essential patent claims, to +make, use, sell, offer for sale, import and otherwise run, modify and +propagate the contents of its contributor version. + + In the following three paragraphs, a "patent license" is any express +agreement or commitment, however denominated, not to enforce a patent +(such as an express permission to practice a patent or covenant not to +sue for patent infringement). To "grant" such a patent license to a +party means to make such an agreement or commitment not to enforce a +patent against the party. + + If you convey a covered work, knowingly relying on a patent license, +and the Corresponding Source of the work is not available for anyone +to copy, free of charge and under the terms of this License, through a +publicly available network server or other readily accessible means, +then you must either (1) cause the Corresponding Source to be so +available, or (2) arrange to deprive yourself of the benefit of the +patent license for this particular work, or (3) arrange, in a manner +consistent with the requirements of this License, to extend the patent +license to downstream recipients. "Knowingly relying" means you have +actual knowledge that, but for the patent license, your conveying the +covered work in a country, or your recipient's use of the covered work +in a country, would infringe one or more identifiable patents in that +country that you have reason to believe are valid. + + If, pursuant to or in connection with a single transaction or +arrangement, you convey, or propagate by procuring conveyance of, a +covered work, and grant a patent license to some of the parties +receiving the covered work authorizing them to use, propagate, modify +or convey a specific copy of the covered work, then the patent license +you grant is automatically extended to all recipients of the covered +work and works based on it. + + A patent license is "discriminatory" if it does not include within +the scope of its coverage, prohibits the exercise of, or is +conditioned on the non-exercise of one or more of the rights that are +specifically granted under this License. You may not convey a covered +work if you are a party to an arrangement with a third party that is +in the business of distributing software, under which you make payment +to the third party based on the extent of your activity of conveying +the work, and under which the third party grants, to any of the +parties who would receive the covered work from you, a discriminatory +patent license (a) in connection with copies of the covered work +conveyed by you (or copies made from those copies), or (b) primarily +for and in connection with specific products or compilations that +contain the covered work, unless you entered into that arrangement, +or that patent license was granted, prior to 28 March 2007. + + Nothing in this License shall be construed as excluding or limiting +any implied license or other defenses to infringement that may +otherwise be available to you under applicable patent law. + + 12. No Surrender of Others' Freedom. + + If conditions are imposed on you (whether by court order, agreement or +otherwise) that contradict the conditions of this License, they do not +excuse you from the conditions of this License. If you cannot convey a +covered work so as to satisfy simultaneously your obligations under this +License and any other pertinent obligations, then as a consequence you may +not convey it at all. For example, if you agree to terms that obligate you +to collect a royalty for further conveying from those to whom you convey +the Program, the only way you could satisfy both those terms and this +License would be to refrain entirely from conveying the Program. + + 13. Use with the GNU Affero General Public License. + + Notwithstanding any other provision of this License, you have +permission to link or combine any covered work with a work licensed +under version 3 of the GNU Affero General Public License into a single +combined work, and to convey the resulting work. The terms of this +License will continue to apply to the part which is the covered work, +but the special requirements of the GNU Affero General Public License, +section 13, concerning interaction through a network will apply to the +combination as such. + + 14. Revised Versions of this License. + + The Free Software Foundation may publish revised and/or new versions of +the GNU General Public License from time to time. Such new versions will +be similar in spirit to the present version, but may differ in detail to +address new problems or concerns. + + Each version is given a distinguishing version number. If the +Program specifies that a certain numbered version of the GNU General +Public License "or any later version" applies to it, you have the +option of following the terms and conditions either of that numbered +version or of any later version published by the Free Software +Foundation. If the Program does not specify a version number of the +GNU General Public License, you may choose any version ever published +by the Free Software Foundation. + + If the Program specifies that a proxy can decide which future +versions of the GNU General Public License can be used, that proxy's +public statement of acceptance of a version permanently authorizes you +to choose that version for the Program. + + Later license versions may give you additional or different +permissions. However, no additional obligations are imposed on any +author or copyright holder as a result of your choosing to follow a +later version. + + 15. Disclaimer of Warranty. + + THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY +APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT +HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY +OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, +THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM +IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF +ALL NECESSARY SERVICING, REPAIR OR CORRECTION. + + 16. Limitation of Liability. + + IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING +WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS +THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY +GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE +USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF +DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD +PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), +EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF +SUCH DAMAGES. + + 17. Interpretation of Sections 15 and 16. + + If the disclaimer of warranty and limitation of liability provided +above cannot be given local legal effect according to their terms, +reviewing courts shall apply local law that most closely approximates +an absolute waiver of all civil liability in connection with the +Program, unless a warranty or assumption of liability accompanies a +copy of the Program in return for a fee. + + END OF TERMS AND CONDITIONS + + How to Apply These Terms to Your New Programs + + If you develop a new program, and you want it to be of the greatest +possible use to the public, the best way to achieve this is to make it +free software which everyone can redistribute and change under these terms. + + To do so, attach the following notices to the program. It is safest +to attach them to the start of each source file to most effectively +state the exclusion of warranty; and each file should have at least +the "copyright" line and a pointer to where the full notice is found. + + + Copyright (C) + + This program is free software: you can redistribute it and/or modify + it under the terms of the GNU General Public License as published by + the Free Software Foundation, either version 3 of the License, or + (at your option) any later version. + + This program is distributed in the hope that it will be useful, + but WITHOUT ANY WARRANTY; without even the implied warranty of + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the + GNU General Public License for more details. + + You should have received a copy of the GNU General Public License + along with this program. If not, see . + +Also add information on how to contact you by electronic and paper mail. + + If the program does terminal interaction, make it output a short +notice like this when it starts in an interactive mode: + + Copyright (C) + This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'. + This is free software, and you are welcome to redistribute it + under certain conditions; type `show c' for details. + +The hypothetical commands `show w' and `show c' should show the appropriate +parts of the General Public License. Of course, your program's commands +might be different; for a GUI interface, you would use an "about box". + + You should also get your employer (if you work as a programmer) or school, +if any, to sign a "copyright disclaimer" for the program, if necessary. +For more information on this, and how to apply and follow the GNU GPL, see +. + + The GNU General Public License does not permit incorporating your program +into proprietary programs. If your program is a subroutine library, you +may consider it more useful to permit linking proprietary applications with +the library. If this is what you want to do, use the GNU Lesser General +Public License instead of this License. But first, please read +. diff --git a/third_party/kindle_unpack/UPSTREAM.md b/third_party/kindle_unpack/UPSTREAM.md new file mode 100644 index 0000000..db91d16 --- /dev/null +++ b/third_party/kindle_unpack/UPSTREAM.md @@ -0,0 +1,16 @@ +# Kindle decoder provenance + +Vendored from [`kindle_unpack` 0.2.0](https://pub.dev/packages/kindle_unpack/versions/0.2.0), +[upstream repository](https://github.com/emaurel/kindle_unpack). The upstream GPL-3.0 +license is retained in LICENSE and included in generated browser worker notices. + +Papyrus changes: + +- Use archive's pure Dart ZLibDecoder for embedded fonts instead of dart:io, + so the same parser runs in browser workers and native isolates. + +- Resolve KF8 indices relative to the KF8 section in dual-format files, retain + shared images from the MOBI section, and recognize both boundary conventions. + +Keep format-specific integration and sanitization in the reader's formats folder. +The source book is never rewritten. This package does not remove DRM. diff --git a/third_party/kindle_unpack/lib/kindle_unpack.dart b/third_party/kindle_unpack/lib/kindle_unpack.dart new file mode 100644 index 0000000..29f960d --- /dev/null +++ b/third_party/kindle_unpack/lib/kindle_unpack.dart @@ -0,0 +1,27 @@ +/// A pure-Dart library for reading Amazon Kindle ebook files +/// (MOBI / AZW / AZW3 / KF8). +/// +/// This is the public entry point; everything else lives in `src/`. +library; + +export 'src/decompress/book_text.dart'; +export 'src/decompress/huff_cdic.dart'; +export 'src/epub.dart'; +export 'src/exception.dart'; +export 'src/decompress/palmdoc.dart'; +export 'src/decompress/trailing_data.dart'; +export 'src/headers/exth.dart'; +export 'src/headers/header_exception.dart'; +export 'src/headers/mobi_header.dart'; +export 'src/headers/palmdoc_header.dart'; +export 'src/images.dart'; +export 'src/kindle_book.dart'; +export 'src/kf8/boundary.dart'; +export 'src/kf8/fdst.dart'; +export 'src/kf8/flows.dart'; +export 'src/kf8/font.dart'; +export 'src/kf8/indx.dart'; +export 'src/kf8/resc.dart'; +export 'src/kf8/skeleton_fragment.dart'; +export 'src/kf8/xhtml_split.dart'; +export 'src/pdb.dart'; diff --git a/third_party/kindle_unpack/lib/src/decompress/book_text.dart b/third_party/kindle_unpack/lib/src/decompress/book_text.dart new file mode 100644 index 0000000..7461350 --- /dev/null +++ b/third_party/kindle_unpack/lib/src/decompress/book_text.dart @@ -0,0 +1,90 @@ +import 'dart:typed_data'; + +import '../headers/header_exception.dart'; +import '../headers/mobi_header.dart'; +import '../headers/palmdoc_header.dart'; +import '../pdb.dart'; +import 'huff_cdic.dart'; +import 'palmdoc.dart'; +import 'trailing_data.dart'; + +/// Decompress and concatenate every text record in a MOBI book. +/// +/// Text records are PDB records 1..[PalmDocHeader.textRecordCount]. Each +/// record is first stripped of its trailing-data entries (per +/// [MobiHeader.extraDataFlags]) and then run through the appropriate +/// decompressor for [PalmDocHeader.compression]. For HUFF/CDIC files, +/// the HUFF + CDIC records pointed at by [MobiHeader.huffmanRecordOffset] +/// / [MobiHeader.huffmanRecordCount] are loaded once and shared across +/// all text records. +/// +/// Returns the raw uncompressed bytes of the book's text. Decoding to a +/// string is the caller's job — the bytes are in [MobiHeader.textEncoding]. +/// +/// Throws [HeaderException] for encrypted files or malformed metadata, +/// and [HuffCdicException] for malformed HUFF/CDIC records. +Uint8List decompressBookText({ + required PdbFile pdb, + required PalmDocHeader palmDoc, + required MobiHeader mobi, +}) { + if (palmDoc.isEncrypted || mobi.hasDrm) { + throw HeaderException( + 'cannot decompress encrypted MOBI (encryption=${palmDoc.encryption}, ' + 'hasDrm=${mobi.hasDrm})', + ); + } + + final count = palmDoc.textRecordCount; + if (count == 0) return Uint8List(0); + if (1 + count > pdb.records.length) { + throw HeaderException( + 'PalmDOC header advertises $count text records but PDB only has ' + '${pdb.records.length - 1} records past record 0', + ); + } + + // Build the HUFF/CDIC decoder up front so we don't re-parse the tables + // for every text record. + HuffTable? huff; + CdicTable? cdic; + if (palmDoc.compression == CompressionType.huffCdic) { + final huffOff = mobi.huffmanRecordOffset; + final huffCount = mobi.huffmanRecordCount; + if (huffOff == 0 || huffCount < 2) { + throw HeaderException( + 'HUFF/CDIC compression but MOBI header has no HUFF/CDIC records ' + '(offset=$huffOff, count=$huffCount)', + ); + } + if (huffOff + huffCount > pdb.records.length) { + throw HeaderException( + 'HUFF/CDIC records [$huffOff..${huffOff + huffCount}) extend past ' + 'PDB record list (${pdb.records.length})', + ); + } + huff = HuffTable.parse(pdb.records[huffOff].data); + cdic = CdicTable.parse([ + for (var i = huffOff + 1; i < huffOff + huffCount; i++) + pdb.records[i].data, + ]); + } + + final flags = mobi.extraDataFlags; + final out = BytesBuilder(copy: false); + for (var i = 1; i <= count; i++) { + final raw = pdb.records[i].data; + final payload = stripTrailingDataEntries(raw, flags); + switch (palmDoc.compression) { + case CompressionType.none: + out.add(payload); + case CompressionType.palmDoc: + out.add(decompressPalmDoc(payload)); + case CompressionType.huffCdic: + out.add( + decompressHuffCdic(input: payload, huff: huff!, cdic: cdic!), + ); + } + } + return out.toBytes(); +} diff --git a/third_party/kindle_unpack/lib/src/decompress/huff_cdic.dart b/third_party/kindle_unpack/lib/src/decompress/huff_cdic.dart new file mode 100644 index 0000000..56cfc31 --- /dev/null +++ b/third_party/kindle_unpack/lib/src/decompress/huff_cdic.dart @@ -0,0 +1,348 @@ +import 'dart:convert'; +import 'dart:typed_data'; + +import '../exception.dart'; + +/// Thrown when a HUFF/CDIC record can't be parsed or a compressed text +/// record can't be decoded. +class HuffCdicException extends KindleUnpackException { + HuffCdicException(super.message); +} + +/// One entry in the HUFF cache (the 256-entry table indexed by the top +/// 8 bits of the input). Real-world MOBI files carry these as packed +/// uint32s; we unpack into a typed record here for clarity. +class HuffCacheEntry { + const HuffCacheEntry({ + required this.codeLen, + required this.terminal, + required this.maxCode, + }); + + /// Number of bits this code consumes (1..32). + final int codeLen; + + /// True when the cache entry alone resolves the codeword. False means + /// we still need to scan [HuffTable.minCode]/[HuffTable.maxCodeByLen] + /// to find the actual length. + final bool terminal; + + /// Pre-shifted maximum code value with this prefix, used to compute + /// the dictionary index as `(maxCode - code) >>> (32 - codeLen)`. + final int maxCode; +} + +/// Parsed HUFF record. The body is two tables: a 256-entry cache for the +/// fast path, and a 32-pair (mincode, maxcode) table indexed by code +/// length 1..32 for the slow path on non-terminal cache hits. +class HuffTable { + HuffTable._({ + required this.cache, + required this.minCode, + required this.maxCodeByLen, + }); + + /// 256 entries indexed by the top 8 bits of the input bitstream. + final List cache; + + /// Indexed by code length 0..32. `minCode[i]` is the smallest 32-bit + /// code value that has length `i`. Index 0 is a sentinel. + final List minCode; + + /// Indexed by code length 0..32. `maxCodeByLen[i]` is the largest + /// 32-bit code value with length `i` (inclusive). + final List maxCodeByLen; + + static HuffTable parse(Uint8List record) { + if (record.length < 24) { + throw HuffCdicException( + 'HUFF record too short: ${record.length} (need 24+ for header)', + ); + } + final view = ByteData.sublistView(record); + + // The signature includes the fixed header length (0x18 = 24). + final sig = latin1.decode(Uint8List.sublistView(record, 0, 4)); + if (sig != 'HUFF') { + throw HuffCdicException('expected "HUFF" signature, got "$sig"'); + } + final headerLen = view.getUint32(4); + if (headerLen != 0x18) { + throw HuffCdicException( + 'unsupported HUFF header length: 0x${headerLen.toRadixString(16)}', + ); + } + + final off1 = view.getUint32(8); + final off2 = view.getUint32(12); + if (off1 + 256 * 4 > record.length) { + throw HuffCdicException( + 'HUFF cache table at offset $off1 extends past record end', + ); + } + if (off2 + 64 * 4 > record.length) { + throw HuffCdicException( + 'HUFF mincode/maxcode table at offset $off2 extends past record end', + ); + } + + final cache = List.generate(256, (i) { + final v = view.getUint32(off1 + i * 4); + final codeLen = v & 0x1F; + final terminal = (v & 0x80) != 0; + final maxCodeRaw = v >> 8; // 24 bits + if (codeLen == 0) { + throw HuffCdicException('HUFF cache entry $i has codeLen 0'); + } + if (codeLen <= 8 && !terminal) { + throw HuffCdicException( + 'HUFF cache entry $i has codeLen $codeLen but is non-terminal', + ); + } + // Promote the 24-bit raw value to a 32-bit ceiling. + final maxCode = ((maxCodeRaw + 1) << (32 - codeLen)) - 1; + return HuffCacheEntry( + codeLen: codeLen, + terminal: terminal, + maxCode: maxCode & 0xFFFFFFFF, + ); + }, growable: false); + + // Build min/max-code arrays. Index 0 is sentinel; indices 1..32 hold + // the (mincode, maxcode) for codes of that length. + final minCode = List.filled(33, 0); + final maxCodeByLen = List.filled(33, 0); + // Length 0 sentinels: mincode=0, maxcode = ((0+1) << 32) - 1. + // (Dart's int is 64-bit signed, so we just store 0xFFFFFFFF here.) + maxCodeByLen[0] = 0xFFFFFFFF; + for (var i = 1; i <= 32; i++) { + final mincodeRaw = view.getUint32(off2 + (i - 1) * 8); + final maxcodeRaw = view.getUint32(off2 + (i - 1) * 8 + 4); + minCode[i] = (mincodeRaw << (32 - i)) & 0xFFFFFFFF; + maxCodeByLen[i] = (((maxcodeRaw + 1) << (32 - i)) - 1) & 0xFFFFFFFF; + } + + return HuffTable._( + cache: cache, + minCode: minCode, + maxCodeByLen: maxCodeByLen, + ); + } +} + +/// One entry in the merged CDIC dictionary. [bytes] is either a +/// pre-decoded literal phrase ([precoded]=true) or itself a HUFF-encoded +/// payload that must be recursively expanded on first use. +class CdicEntry { + CdicEntry({required this.bytes, required this.precoded}); + + Uint8List bytes; + bool precoded; +} + +/// The merged dictionary built from one HUFF record's worth of CDIC +/// records (typically 1..a few). Index space matches the integers +/// produced by [decompressHuffCdic] when it resolves a codeword. +class CdicTable { + CdicTable._({required this.entries, required this.codeBits}); + + final List entries; + + /// Code length in bits (typically 16 — i.e. up to 65 536 entries per + /// CDIC record). Stored mostly for diagnostics. + final int codeBits; + + /// Build the dictionary from [records] in order. The first CDIC + /// declares the total phrase count and the per-record bits; subsequent + /// CDICs continue filling the same flat list. + static CdicTable parse(List records) { + if (records.isEmpty) { + throw HuffCdicException('CDIC table needs at least one record'); + } + var totalPhrases = 0; + var bits = 0; + final entries = []; + + for (var ri = 0; ri < records.length; ri++) { + final rec = records[ri]; + if (rec.length < 16) { + throw HuffCdicException( + 'CDIC record $ri too short: ${rec.length} (need 16+)', + ); + } + final view = ByteData.sublistView(rec); + final sig = latin1.decode(Uint8List.sublistView(rec, 0, 4)); + if (sig != 'CDIC') { + throw HuffCdicException( + 'CDIC record $ri: expected "CDIC" signature, got "$sig"', + ); + } + final headerLen = view.getUint32(4); + if (headerLen != 0x10) { + throw HuffCdicException( + 'CDIC record $ri: unsupported header length ' + '0x${headerLen.toRadixString(16)}', + ); + } + final phrases = view.getUint32(8); + final recBits = view.getUint32(12); + if (ri == 0) { + totalPhrases = phrases; + bits = recBits; + } else if (phrases != totalPhrases || recBits != bits) { + throw HuffCdicException( + 'CDIC record $ri header disagrees with first record ' + '(phrases=$phrases vs $totalPhrases, bits=$recBits vs $bits)', + ); + } + // How many entries this record carries: at most 1< rec.length) { + throw HuffCdicException( + 'CDIC record $ri offset table runs past record end', + ); + } + for (var i = 0; i < n; i++) { + final off = view.getUint16(16 + i * 2); + if (16 + off + 2 > rec.length) { + throw HuffCdicException( + 'CDIC record $ri entry $i offset $off out of range', + ); + } + final lengthAndFlag = view.getUint16(16 + off); + final blen = lengthAndFlag & 0x7FFF; + final precoded = (lengthAndFlag & 0x8000) != 0; + final start = 16 + off + 2; + final end = start + blen; + if (end > rec.length) { + throw HuffCdicException( + 'CDIC record $ri entry $i payload runs past record end', + ); + } + entries.add( + CdicEntry( + bytes: Uint8List.sublistView(rec, start, end), + precoded: precoded, + ), + ); + } + } + + if (entries.length != totalPhrases) { + throw HuffCdicException( + 'CDIC: filled ${entries.length} entries but header advertises ' + '$totalPhrases', + ); + } + return CdicTable._(entries: entries, codeBits: bits); + } +} + +/// Decompress a single HUFF/CDIC-compressed payload. Caches the result +/// of recursively expanded dictionary entries on the [cdic] in place +/// (precoded=true after first use), so subsequent records reuse the +/// work. +/// +/// Algorithm port of `HuffcdicReader.unpack` from KindleUnpack +/// (mobi_uncompress.py). The 64-bit sliding window is maintained as two +/// big-endian uint32s so we don't depend on 64-bit unsigned semantics +/// the way the Python original does. +Uint8List decompressHuffCdic({ + required Uint8List input, + required HuffTable huff, + required CdicTable cdic, +}) { + return _DecompressContext(huff, cdic).run(input); +} + +class _DecompressContext { + _DecompressContext(this.huff, this.cdic); + + final HuffTable huff; + final CdicTable cdic; + + /// Cycle-detection set: dictionary indices currently being expanded. + /// A precoded entry that recursively references itself would otherwise + /// loop forever. + final Set _expanding = {}; + + Uint8List run(Uint8List input) { + // Pad with 8 zero bytes so the 64-bit-window reads at the tail are + // safe. Real text records are 4 KiB so this is cheap. + final padded = Uint8List(input.length + 8); + padded.setRange(0, input.length, input); + final view = ByteData.sublistView(padded); + + var bitsLeft = input.length * 8; + var pos = 0; + var high = view.getUint32(0); + var low = view.getUint32(4); + var n = 32; + + final out = BytesBuilder(copy: false); + + while (true) { + if (n <= 0) { + pos += 4; + high = low; + low = view.getUint32(pos + 4); + n += 32; + } + + // code = top 32 bits of (high::low) shifted right by n. + final int code; + if (n == 32) { + code = high; + } else if (n == 0) { + code = low; + } else { + code = ((high << (32 - n)) | (low >>> n)) & 0xFFFFFFFF; + } + + final cacheEntry = huff.cache[code >>> 24]; + var codeLen = cacheEntry.codeLen; + var maxCode = cacheEntry.maxCode; + if (!cacheEntry.terminal) { + while (code < huff.minCode[codeLen]) { + codeLen++; + if (codeLen > 32) { + throw HuffCdicException('codeword exceeds 32 bits'); + } + } + maxCode = huff.maxCodeByLen[codeLen]; + } + + n -= codeLen; + bitsLeft -= codeLen; + if (bitsLeft < 0) break; + + final r = (maxCode - code) >>> (32 - codeLen); + if (r < 0 || r >= cdic.entries.length) { + throw HuffCdicException( + 'dictionary index $r out of range ' + '(table size ${cdic.entries.length})', + ); + } + final entry = cdic.entries[r]; + if (entry.precoded) { + out.add(entry.bytes); + } else { + if (!_expanding.add(r)) { + throw HuffCdicException('cycle in CDIC entry $r'); + } + final expanded = run(entry.bytes); + _expanding.remove(r); + // Cache for subsequent records of the same book. + entry.bytes = expanded; + entry.precoded = true; + out.add(expanded); + } + } + + return out.toBytes(); + } +} diff --git a/third_party/kindle_unpack/lib/src/decompress/palmdoc.dart b/third_party/kindle_unpack/lib/src/decompress/palmdoc.dart new file mode 100644 index 0000000..9b13d5b --- /dev/null +++ b/third_party/kindle_unpack/lib/src/decompress/palmdoc.dart @@ -0,0 +1,85 @@ +import 'dart:typed_data'; + +import '../exception.dart'; + +/// Thrown when a PalmDOC-compressed input can't be decoded. +class PalmDocDecompressException extends KindleUnpackException { + PalmDocDecompressException(super.message); +} + +/// Decompress a single PalmDOC-compressed byte stream. +/// +/// PalmDOC is a one-pass byte-coded LZ77 variant. Each control byte `c` +/// dispatches to one of four opcode classes: +/// +/// - `0x00` and `0x09..0x7F`: emit `c` literally. +/// - `0x01..0x08`: read the next `c` bytes verbatim and emit them. +/// - `0x80..0xBF`: a 2-byte back-reference. The 14 bits below the `10` +/// prefix split into 11 bits of distance (1..2047, measured back from +/// the current output cursor) and 3 bits of length (encoded as +/// `length - 3`, so the actual length is 3..10). Bytes are copied one +/// at a time, so overlapping copies (`distance < length`) produce +/// repeating patterns. +/// - `0xC0..0xFF`: a "space pair" — emit `0x20` followed by `c ^ 0x80`. +/// +/// The decoder is strict: a back-reference whose distance is zero or +/// would read before the start of the output buffer throws. A literal +/// run that would read past the end of the input throws. +Uint8List decompressPalmDoc(Uint8List input) { + final out = []; + var i = 0; + final n = input.length; + + while (i < n) { + final c = input[i++]; + + if (c == 0 || (c >= 0x09 && c <= 0x7F)) { + out.add(c); + } else if (c >= 0x01 && c <= 0x08) { + if (i + c > n) { + throw PalmDocDecompressException( + 'literal run of $c bytes at offset ${i - 1} runs past input ' + '(length $n)', + ); + } + for (var j = 0; j < c; j++) { + out.add(input[i + j]); + } + i += c; + } else if (c >= 0x80 && c <= 0xBF) { + if (i >= n) { + throw PalmDocDecompressException( + 'back-reference at offset ${i - 1} truncated — missing low byte', + ); + } + final c2 = input[i++]; + final combined = ((c << 8) | c2) & 0x3FFF; + final distance = combined >> 3; + final length = (combined & 0x07) + 3; + if (distance == 0) { + throw PalmDocDecompressException( + 'back-reference at offset ${i - 2} has distance 0', + ); + } + final src = out.length - distance; + if (src < 0) { + throw PalmDocDecompressException( + 'back-reference at offset ${i - 2} reads before start of output ' + '(distance $distance, output ${out.length})', + ); + } + // Copy byte-by-byte from the live output. When distance < length, + // each appended byte feeds the next read, producing run-length + // patterns — that's the LZ77 trick that compresses repeated chars. + for (var j = 0; j < length; j++) { + out.add(out[src + j]); + } + } else { + // c >= 0xC0: space-pair shortcut. + out.add(0x20); + out.add(c ^ 0x80); + } + } + + return Uint8List.fromList(out); +} diff --git a/third_party/kindle_unpack/lib/src/decompress/trailing_data.dart b/third_party/kindle_unpack/lib/src/decompress/trailing_data.dart new file mode 100644 index 0000000..553f5f2 --- /dev/null +++ b/third_party/kindle_unpack/lib/src/decompress/trailing_data.dart @@ -0,0 +1,72 @@ +import 'dart:typed_data'; + +import '../headers/header_exception.dart'; + +/// Compute the total length of trailing-data entries appended to a +/// MOBI text record, given the `extra record data flags` bitfield from +/// the MOBI header (offset 0xF2). +/// +/// The flags bitfield works as follows: +/// - Each bit `i > 0` set means the record has a length-prefixed trailer +/// for category `i` appended at its end. The length is a backwards +/// variable-width integer: bytes are read from the *end* of the record +/// going backward, each contributing 7 bits to the result; a byte with +/// the high bit set marks the last (most-significant) byte of the int. +/// The decoded value is the trailer's total length, including the +/// bytes that encode it. +/// - Bit 0 set means the record ends with a "multibyte char overlap" +/// indicator: a single byte whose low 2 bits + 1 give how many bytes +/// at the end of the record belong to a multi-byte char that started +/// in the previous record. We strip that 1 indicator byte plus the +/// bytes it points at. +/// +/// All bit-`i>0` trailers are stripped first (each pointing at the *new* +/// end as we shrink), then the bit-0 indicator is consumed last. +int sizeOfTrailingDataEntries(Uint8List record, int flags) { + int sizeOfOne(int end) { + var bitpos = 0; + var result = 0; + var pos = end; + while (pos > 0) { + final v = record[pos - 1]; + result |= (v & 0x7F) << bitpos; + bitpos += 7; + pos -= 1; + if ((v & 0x80) != 0 || bitpos >= 28) break; + } + return result; + } + + var num = 0; + var testflags = flags >> 1; + while (testflags != 0) { + if ((testflags & 1) != 0) { + num += sizeOfOne(record.length - num); + } + testflags >>= 1; + } + if ((flags & 1) != 0) { + if (record.length - num - 1 < 0) { + throw HeaderException( + 'multibyte-overlap indicator points before record start', + ); + } + num += (record[record.length - num - 1] & 0x3) + 1; + } + return num; +} + +/// Return a view of [record] with its trailing data entries stripped, so +/// the result is the raw compressed payload (PalmDOC or HUFF/CDIC). +/// When [flags] is 0 this is a no-op. +Uint8List stripTrailingDataEntries(Uint8List record, int flags) { + if (flags == 0) return record; + final size = sizeOfTrailingDataEntries(record, flags); + if (size < 0 || size > record.length) { + throw HeaderException( + 'trailing-data size $size out of range for record length ' + '${record.length}', + ); + } + return Uint8List.sublistView(record, 0, record.length - size); +} diff --git a/third_party/kindle_unpack/lib/src/epub.dart b/third_party/kindle_unpack/lib/src/epub.dart new file mode 100644 index 0000000..5d8b718 --- /dev/null +++ b/third_party/kindle_unpack/lib/src/epub.dart @@ -0,0 +1,290 @@ +import 'dart:convert'; +import 'dart:typed_data'; + +import 'package:archive/archive.dart'; + +import 'images.dart'; +import 'kf8/xhtml_split.dart'; + +/// Minimal EPUB-3 builder: takes the building blocks Phases 5–10b +/// produced (XHTML parts, images, optional CSS / fonts) plus a small +/// metadata bundle, and emits a valid \[EPUB 3.x\] zip suitable for any +/// EPUB reader. The packager isn't ambitious — it doesn't try to +/// preserve Kindlegen's original spine ordering refinements or NCX +/// playOrder, just produces an EPUB that opens. +abstract final class EpubBuilder { + /// Build the EPUB zip. Returns the byte buffer (not written to disk). + /// + /// Layout matches the EPUB OCF spec: + /// ``` + /// mimetype (uncompressed, must be first) + /// META-INF/container.xml + /// OEBPS/content.opf + /// OEBPS/nav.xhtml + /// OEBPS/toc.ncx + /// OEBPS/Text/.xhtml + /// OEBPS/Images/. + /// OEBPS/Styles/