* feat: Add automated bookmark backup system
Implements a comprehensive automated backup feature for user bookmarks with the following capabilities:
Database Schema:
- Add backupSettings table to store user backup preferences (enabled, frequency, retention)
- Add backups table to track backup records with status and metadata
- Add BACKUP asset type for storing compressed backup files
- Add migration 0066_add_backup_tables.sql
Background Workers:
- Implement BackupSchedulingWorker cron job (runs daily at midnight UTC)
- Create BackupWorker to process individual backup jobs
- Deterministic scheduling spreads backup jobs across 24 hours based on user ID hash
- Support for daily and weekly backup frequencies
- Automated retention cleanup to delete old backups based on user settings
Export & Compression:
- Reuse existing export functionality for bookmark data
- Compress exports using Node.js built-in zlib (gzip level 9)
- Store compressed backups as assets with proper metadata
- Track backup size and bookmark count for statistics
tRPC API:
- backups.getSettings - Retrieve user backup configuration
- backups.updateSettings - Update backup preferences
- backups.list - List all user backups with metadata
- backups.get - Get specific backup details
- backups.delete - Delete a backup
- backups.download - Download backup file (base64 encoded)
- backups.triggerBackup - Manually trigger backup creation
UI Components:
- BackupSettings component with configuration form
- Enable/disable automatic backups toggle
- Frequency selection (daily/weekly)
- Retention period configuration (1-365 days)
- Backup list table with download and delete actions
- Manual backup trigger button
- Display backup stats (size, bookmark count, status)
- Added backups page to settings navigation
Technical Details:
- Uses Restate queue system for distributed job processing
- Implements idempotency keys to prevent duplicate backups
- Background worker concurrency: 2 jobs at a time
- 10-minute timeout for large backup exports
- Proper error handling and logging throughout
- Type-safe implementation with Zod schemas
* refactor: simplify backup settings and asset handling
- Move backup settings from separate table to user table columns
- Update BackupSettings model to use static methods with users table
- Remove download mutation in favor of direct asset links
- Implement proper quota checks using QuotaService.checkStorageQuota
- Update UI to use new property names and direct asset downloads
- Update shared types to match new schema
Key changes:
- backupSettingsTable removed, settings now in users table
- Backup downloads use direct /api/assets/{id} links
- Quota properly validated before creating backup assets
- Cleaner separation of concerns in tRPC models
* migration
* use zip instead of gzip
* fix drizzle
* fix settings
* streaming json
* remove more dead code
* add e2e tests
* return backup
* poll for backups
* more fixes
* more fixes
* fix test
* fix UI
* fix delete asset
* fix ui
* redirect for backup download
* cleanups
* fix idempotency
* fix tests
* add ratelimit
* add error handling for background backups
* i18n
* model changes
---------
Co-authored-by: Claude <noreply@anthropic.com>
* chore: metascraper 5.x comes with its own types, including @types/metascraper is now redundant; also updating to latest versions of metascraper libraries
* feat (workers): creating a local metascraper plugin for Reddit posts
In the past, the preview images for bookmarks from Reddit links were
poorly chosen. Reddit does not use opengraph tags, so metascraper-images
simply looked for all images on the page and returned the first. This
tended to be the profile picture for the poster for the Reddit link.
This new plugin, using the existing metascraper framework, provides a
better selection of image for the bookmark when the URL domain is
'reddit'.
In addition, recent changes (I believe this was a side effect of adding
the metascraper-author and/or the metascaper-publisher plugins, but it
could also be related to the metascraper-readibility plugin) broke what
used to be a good choice of bookmark title. Previously, titles looked
like 'Tinyauth just reached 1000 stars! : r/selfhosted' with both thread
title and subreddit mentioned. After this update, all Reddit posts now
have the same title: 'The heart of the internet'.
To return to the better format, this new metascraper-reddit plugin now
attempts to retrieve the better title from reddit URLs. Note that in
order to gain precendence in title selection, the 'metascraperReddit()'
inclusion in the crawlerWorkers.ts metascraper instantiation list had to
be moved above metascraperReadability().
* chore: updated Hoarder in text to Karakeep
* chore: update metascraper versions
fix for metascraper types has been merged; the expect-error comment can
be removed
* chore: merge with master
---------
Co-authored-by: Mohamed Bassem <me@mbassem.com>
* feat: convert to playwright
Convert crawling to use Playwright instead of Chrome.
- Update Dockerfile to include Playwright
- Update crawler worker to use Playwright API
- Update dependencies
* feat: convert from Puppeteer to Playwright for crawling
* feat: update docker-compose
* use separate browser context for better isolation
* skip chrome download in linux script
* readd the stealth plugin
---------
Co-authored-by: Mohamed Bassem <me@mbassem.com>
* Updated pdf2json to 3.1.5
* Extract and store a screenshot from PDF files using pdf2pic
* Installing graphicsmagick and ghostscript
* Generate Missing PDF screenshot with tidyAssets worker for backward support
* Display PDF screenshot instead of the PDF in web if it exists.
* Display PDF screenshot in mobile app if exists.
* Updated pnpm-lock.yaml
* Removed console.log
* Revert the unnecessary changes in package.json
* Revert pnpm-lock changes
* Prevent rendering PDF files if the screenshot is not generated
* refactor: replace useEffect with useMemo for section initialization
* feat: show PDF file download button and handle large PDFs by defaulting to screenshot view
* feat: add file size to openapi spec
* feature: Add Assets preprocessing in fix mode to admin actions
* i18n: add reprocess_assets_fix_mode translation
* i18n: Add missing ar translations
* A bunch of fixes
* Fix openspec schema
---------
Co-authored-by: Mohamed Bassem <me@mbassem.com>
