- Added ASSET_PREPROCESSING_JOB_TIMEOUT_SEC environment variable with default of 60 seconds (increased from hardcoded 30 seconds)
- Updated worker to use the configurable timeout from serverConfig
- Added documentation for the new configuration option
* feat: check import quota before importing bookmarks
Add quota validation before bookmark import to prevent users from
exceeding their bookmark limits. The implementation includes:
- New QuotaService.canImportBookmarks() method to check if user can import N bookmarks
- New tRPC checkImportQuota procedure for client-side quota validation
- Updated useBookmarkImport hook to parse files and check quota before import
- Added error banner in ImportExport component to display quota errors
- Optimized file parsing to avoid reading the file twice
The quota check displays remaining bookmarks and provides clear error
messages when the import would exceed the user's quota.
* fix
* some fixes
---------
Co-authored-by: Claude <noreply@anthropic.com>
Add a new search qualifier `is:broken` that allows users to filter bookmarks
with broken or failed links. This matches the functionality on the broken links
settings page, where a link is considered broken if:
- crawlStatus is "failure"
- crawlStatusCode is less than 200
- crawlStatusCode is greater than 299
The qualifier supports negation with `-is:broken` to find working links.
Changes:
- Add brokenLinks matcher type definition
- Update search query parser to handle is:broken qualifier
- Implement query execution logic for broken links filtering
- Add autocomplete support with translations
- Add parser tests
- Update search query language documentation
Co-authored-by: Claude <noreply@anthropic.com>
Previously, all RSS feeds were fetched at the top of each hour (minute 0),
which could cause load spikes. This change spreads feed fetches evenly
throughout the hour using a deterministic hash of the feed ID.
Each feed is assigned a target minute (0-59) based on its ID hash, ensuring
consistent scheduling across restarts while distributing the load evenly.
Co-authored-by: Claude <noreply@anthropic.com>
* feat: Add automated bookmark backup system
Implements a comprehensive automated backup feature for user bookmarks with the following capabilities:
Database Schema:
- Add backupSettings table to store user backup preferences (enabled, frequency, retention)
- Add backups table to track backup records with status and metadata
- Add BACKUP asset type for storing compressed backup files
- Add migration 0066_add_backup_tables.sql
Background Workers:
- Implement BackupSchedulingWorker cron job (runs daily at midnight UTC)
- Create BackupWorker to process individual backup jobs
- Deterministic scheduling spreads backup jobs across 24 hours based on user ID hash
- Support for daily and weekly backup frequencies
- Automated retention cleanup to delete old backups based on user settings
Export & Compression:
- Reuse existing export functionality for bookmark data
- Compress exports using Node.js built-in zlib (gzip level 9)
- Store compressed backups as assets with proper metadata
- Track backup size and bookmark count for statistics
tRPC API:
- backups.getSettings - Retrieve user backup configuration
- backups.updateSettings - Update backup preferences
- backups.list - List all user backups with metadata
- backups.get - Get specific backup details
- backups.delete - Delete a backup
- backups.download - Download backup file (base64 encoded)
- backups.triggerBackup - Manually trigger backup creation
UI Components:
- BackupSettings component with configuration form
- Enable/disable automatic backups toggle
- Frequency selection (daily/weekly)
- Retention period configuration (1-365 days)
- Backup list table with download and delete actions
- Manual backup trigger button
- Display backup stats (size, bookmark count, status)
- Added backups page to settings navigation
Technical Details:
- Uses Restate queue system for distributed job processing
- Implements idempotency keys to prevent duplicate backups
- Background worker concurrency: 2 jobs at a time
- 10-minute timeout for large backup exports
- Proper error handling and logging throughout
- Type-safe implementation with Zod schemas
* refactor: simplify backup settings and asset handling
- Move backup settings from separate table to user table columns
- Update BackupSettings model to use static methods with users table
- Remove download mutation in favor of direct asset links
- Implement proper quota checks using QuotaService.checkStorageQuota
- Update UI to use new property names and direct asset downloads
- Update shared types to match new schema
Key changes:
- backupSettingsTable removed, settings now in users table
- Backup downloads use direct /api/assets/{id} links
- Quota properly validated before creating backup assets
- Cleaner separation of concerns in tRPC models
* migration
* use zip instead of gzip
* fix drizzle
* fix settings
* streaming json
* remove more dead code
* add e2e tests
* return backup
* poll for backups
* more fixes
* more fixes
* fix test
* fix UI
* fix delete asset
* fix ui
* redirect for backup download
* cleanups
* fix idempotency
* fix tests
* add ratelimit
* add error handling for background backups
* i18n
* model changes
---------
Co-authored-by: Claude <noreply@anthropic.com>
* feat: lazy load tiktoken to reduce memory footprint
The js-tiktoken module loads a large encoding dictionary into memory
immediately on import. This change defers the loading of the encoding
until it's actually needed by using a lazy getter pattern.
This reduces memory usage for processes that import this module but
don't actually use the token encoding functions.
* fix: use createRequire for lazy tiktoken import in ES module
The previous implementation used bare require() which fails at runtime
in ES modules (ReferenceError: require is not defined). This fixes it
by using createRequire from Node's 'module' package, which creates a
require function that works in ES module contexts.
* refactor: convert tiktoken lazy loading to async dynamic imports
Changed from createRequire to async import() for lazy loading tiktoken,
making buildTextPrompt and buildSummaryPrompt async. This is cleaner for
ES modules and properly defers the large tiktoken encoding data until
it's actually needed.
Updated all callers to await these async functions:
- packages/trpc/routers/bookmarks.ts
- apps/workers/workers/inference/tagging.ts
- apps/workers/workers/inference/summarize.ts
- apps/web/components/settings/AISettings.tsx (converted to useEffect)
* feat: add untruncated prompt builders for UI previews
Added buildTextPromptUntruncated and buildSummaryPromptUntruncated
functions that don't require token counting or truncation. These are
synchronous and don't load tiktoken, making them perfect for UI
previews where exact token limits aren't needed.
Updated AISettings.tsx to use these untruncated versions, eliminating
the need for useEffect/useState and avoiding unnecessary tiktoken
loading in the browser.
* fix
* fix
---------
Co-authored-by: Claude <noreply@anthropic.com>
