Commit Graph

354 Commits

Author SHA1 Message Date
Carl Schwan d6f7805ae7 refactor: Improve log message
Co-authored-by: Josh <josh.t.richards@gmail.com>
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2026-06-09 10:45:48 +02:00
Robin Appelman ccb1d9afa7 fix: improve logging around failed chunked object store uploads
Signed-off-by: Robin Appelman <robin@icewind.nl>
2026-06-09 10:45:48 +02:00
Côme Chilliet 1ab09ec753 chore: Apply new coding standard to all files
The diff can be checked using: git diff --ignore-all-space --ignore-blank-lines
To see only the changes not related to blank lines.

Signed-off-by: Côme Chilliet <come.chilliet@nextcloud.com>
2026-06-01 13:46:39 +02:00
John Molakvoæ ee602dcfe4 fix: adjust wording in S3ConnectionTrait comment
Co-authored-by: Louis <louis@chmn.me>
Signed-off-by: John Molakvoæ <skjnldsv@users.noreply.github.com>
2026-05-12 09:07:20 +02:00
John Molakvoæ (skjnldsv) c26b1e4b07 fix(s3): add Content-MD5 header for DeleteObjects to fix AWS SDK v3.339.0+ compatibility
AWS SDK PHP v3.339.0+ introduced a breaking change requiring the Content-MD5
header for DeleteObjects operations. This causes 'MissingContentMD5' errors when
using S3-compatible services like MinIO.

Add middleware to automatically calculate and inject the Content-MD5 header on
all DeleteObjects requests. This is applied universally at the S3ConnectionTrait
level, fixing both external storage (AmazonS3) and core ObjectStore (S3) classes.

Fixes: https://github.com/aws/aws-sdk-php/issues/3068
Signed-off-by: John Molakvoæ (skjnldsv) <skjnldsv@protonmail.com>
2026-05-12 09:07:20 +02:00
Git'Fellow 3bbe0ee570 refactor: use strict operator
Signed-off-by: Git'Fellow <12234510+solracsf@users.noreply.github.com>
2026-04-30 00:51:29 +02:00
Josh d32cae3e68 Merge pull request #58840 from nextcloud/jtr/fix-ObjectStore-S3-doesObjectExistV2
fix(ObjectStore/S3): switch to more reliable doesObjectExistV2
2026-04-29 08:48:53 -04:00
Ferdinand Thiessen e0ba4d71b6 chore: add missing Override attribute to OC
Signed-off-by: Ferdinand Thiessen <opensource@fthiessen.de>
2026-04-28 21:29:27 +02:00
Josh 1989ae9865 chore(s3): fully resolve conflict in S3ObjectTrait
Signed-off-by: Josh <josh.t.richards@gmail.com>
2026-04-25 11:16:56 -04:00
copilot-swe-agent[bot] 617803fc10 Merge remote-tracking branch 'origin/master' into jtr/fix-ObjectStore-S3-doesObjectExistV2
# Conflicts:
#	lib/private/Files/ObjectStore/S3ObjectTrait.php

Co-authored-by: joshtrichards <1731941+joshtrichards@users.noreply.github.com>
2026-04-25 11:48:06 +00:00
Kostiantyn Miakshyn fcb10219d1 fix: Allow num_buckets to be equal as min_bucket
Signed-off-by: Kostiantyn Miakshyn <molodchick@gmail.com>
2026-04-19 18:28:12 +02:00
Stephen Cuppett cdaeed02b6 feat(objectstore): Add AWS SSE-KMS encryption support for S3 storage
Add support for Server-Side Encryption with AWS Key Management Service
(SSE-KMS) for S3 object storage. This allows Nextcloud to encrypt data
at rest in S3 using AWS-managed keys.

Key features:
- New config options: sse_kms_enabled and sse_kms_key_id
- Backward compatible with existing SSE-C (customer-provided keys)
- SSE-C takes precedence when both SSE-C and SSE-KMS are configured

Implementation details:
- Added getServerSideEncryptionParameters() method to centralize
  encryption parameter logic for both SSE-C and SSE-KMS
- Updated multipart uploads to use unified encryption parameters
- Added comprehensive PHPUnit tests for SSE-KMS scenarios
- Tested with AWS bucket and KMS keys in us-east-1 region

Co-Authored-By: Claude Sonnet 4.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Stephen Cuppett <steve@cuppett.com>
2026-04-16 13:21:53 -04:00
Josh 30bbb13f9a chore: at S3Exception to S3ObjectTrait
Signed-off-by: Josh <josh.t.richards@gmail.com>
2026-03-10 11:51:47 -04:00
Josh 7bcf684f81 fix(ObjectStore/S3): switch to more reliable doesObjectExistV2
The non-v2 version is deprecated, but more importantly the V2 implementation makes fewer assumptions about ambiguous or unexpected status codes. For example, a 403 would result in a false (object does not exist) in V1, but that's not necessarily what that means. V2 returns true/false on a much more narrow set of scenarios. And it throws for all others so they can be diagnosed properly.

Signed-off-by: Josh <josh.t.richards@gmail.com>
2026-03-10 11:12:57 -04:00
Robin Appelman cf124e7c75 fix: make objectstore copy consistent with changed local storage behavior
Signed-off-by: Robin Appelman <robin@icewind.nl>
2026-03-10 16:07:38 +01:00
Josh 6c1d648f59 chore: fixup/ revert cast in S3ObjectTrait
Signed-off-by: Josh <josh.t.richards@gmail.com>
2026-03-02 13:44:20 +01:00
Josh d5d34adcbb fix(S3ObjectTrait): cast $urn to a string until typing adding elsewhere
Signed-off-by: Josh <josh.t.richards@gmail.com>
2026-03-02 13:44:20 +01:00
Josh 9ce1ae8e61 fix(ObjectStorage): fix 0 folder handling in Primary Storage
Signed-off-by: Josh <josh.t.richards@gmail.com>
2026-03-02 13:44:20 +01:00
Carl Schwan 3979c493f9 refactor: Apply second batch of comments
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2026-02-06 13:52:51 +01:00
Carl Schwan 7b6078875b refactor: Run rector on lib/private
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2026-02-06 13:50:18 +01:00
Carl Schwan 0b8eae7789 perf(files): Optimize CacheEntry creation
Avoid many copy on writes and create array only once.

Signed-off-by: Carl Schwan <carlschwan@kde.org>
2026-01-15 11:47:41 +01:00
Robin Appelman afa23291d6 feat: don't gate perBucket object store configuration behind multibucket
a setup can have multiple bucket without having `multibucket` enabled trough things like per-groupfolder buckets

Signed-off-by: Robin Appelman <robin@icewind.nl>
2026-01-05 16:09:25 +01:00
Carl Schwan 2ea1bd4cdc feat(movie-preview): Use getDirectDownloadById for generating preview
Allow to speed-up considerably the creation of previews for movies
stored on S3.

Signed-off-by: Carl Schwan <carlschwan@kde.org>
2026-01-02 17:13:41 +01:00
Carl Schwan b6313f68d3 perf(s3): Expose pre-signed urls for S3
This is faster than going back to nextcloud to download the files.

This is an opt-in setting that can be enabled by setting
use_presigned_url in the object store config.

Additionally add support for the proxy config which is needed in a
docker setup. See https://github.com/juliusknorr/nextcloud-docker-dev/pull/431

Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2025-12-09 11:21:35 +01:00
Daniel Kesselberg f977a7fec6 fix(s3): make data integrity protections opt-in
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2025-12-02 21:03:26 +01:00
provokateurin 2ab29a69e0 feat(ObjectStore): Allow overriding arguments per bucket
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-12-01 16:06:05 +01:00
provokateurin 0c3a872908 feat(CertificateManager): Add option to specify the default certificates bundle path
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-12-01 10:47:23 +01:00
Kent Delante 96672e1c4d feat: emit an event when an S3 bucket is created
Signed-off-by: Kent Delante <kent.delante@proton.me>
2025-11-24 19:37:18 +08:00
Andy Scherzinger 9e2f07b7d4 Merge pull request #56395 from nextcloud/carl/cache-bucket-exists
perf(s3): Cache whether bucket exists
2025-11-24 12:13:27 +01:00
Carl Schwan a228ffa641 refactor(s3): Readability
Co-authored-by: Kate <26026535+provokateurin@users.noreply.github.com>
Signed-off-by: Carl Schwan <carl@carlschwan.eu>
2025-11-20 16:50:14 +01:00
nfebe 3030783785 feat(objectstore): add configurable S3 retry attempts
Add retriesMaxAttempts parameter to S3 objectstore configuration
to allow customization of AWS SDK retry behavior for handling
unreliable network conditions or proxy issues.

Defaults to 5 retries (AWS SDK default) if not specified.

Signed-off-by: nfebe <fenn25.fn@gmail.com>
2025-11-19 16:17:43 +01:00
Robin Appelman a7d130c598 Revert "fix(ObjectStoreStorage): Encode original-path in writeStream by @frabe1579"
This reverts commit 4b4b39e7ec.

Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-11-18 19:53:17 +01:00
Robin Appelman 8194be795d fix: encode s3 metadata as base64 if needed
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-11-18 16:06:57 +01:00
Carl Schwan 67d0307e66 perf(s3): Cache whether bucket exists
Otherwise, we call doesBucketExist all the time which does a network
request to the S3 server adding some non-trivial latency when creating a
S3 connection object.

Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2025-11-12 14:17:48 +01:00
Kate e3edf43e28 Merge pull request #56096 from nextcloud/fix/56077/s3-configure-checksum-defaults 2025-11-03 11:08:33 +01:00
Stephan Orbaugh ef21c79368 Merge pull request #55717 from DaanSelen/fix-issue-55665
fix(ObjectStoreStorage): fix s3 path encoding (issue causing 55665)
2025-11-03 10:58:53 +01:00
Fiehe Christoph b87add2711 fix(s3): expose request_checksum_calculation and response_checksum_validation
Fix https://github.com/nextcloud/server/issues/56077

This commit makes the configuration settings 'request_checksum_calculation' and 'response_checksum_validation' of the S3Client from the AWS SDK for PHP configurable.

Signed-off-by: Fiehe Christoph  <c.fiehe@eurodata.de>
Signed-off-by: Daniel Kesselberg <mail@danielkesselberg.de>
2025-10-30 11:57:32 +01:00
Daan Selen 4b4b39e7ec fix(ObjectStoreStorage): Encode original-path in writeStream by @frabe1579
Signed-off-by: Daan Selen <dselen@systemec.nl>
2025-10-15 11:53:39 +02:00
Thomas Citharel 5c8bdad828 fix(objectstore): add named parameter for like query
to avoid 'object::%' to be considered as a column
(`la colonne \\u00ab object::% \\u00bb n'existe pas` on PG)

Signed-off-by: Thomas Citharel <tcit@tcit.fr>
2025-10-13 16:18:18 +02:00
Louis Chmn 00ec57efd6 fix(S3): Only append streams if non-seekable
Later, when we rewind the stream in `writeMultiPart` during retry, both streams were rewinded, so the resulting stream was bigger than expected.

Inspired by https://github.com/aws/aws-sdk-php/blob/master/src/S3/ObjectUploader.php#L136-L146

Signed-off-by: Louis Chmn <louis@chmn.me>
2025-10-10 09:54:05 +02:00
Carl Schwan b0357663b9 perf(preview): Optimize migration and simplify DB layout
* Simplify migration by not moving the actual files and just updating
  the DB
* Don't store the storageid in the preview table as it is not needed
* Start adding tests

Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2025-10-06 13:37:15 +02:00
Carl Schwan 13c35c0f17 perf(preview): Migrate previews to the new optimized table
Signed-off-by: Carl Schwan <carl.schwan@nextcloud.com>
2025-10-06 13:37:15 +02:00
Carl Schwan 18fbacdd8d perf(preview): Split preview data to new table
The new oc_previews table is optimized for storing previews and should
decrease significantly the space taken by previews in the filecache
table.

This attend to reuse the IObjectStore abstraction over S3/Swift/Azure
but currently only support one single bucket configuration.

Signed-off-by: Carl Schwan <carl.schwan@nextclound.com>
2025-10-06 13:37:15 +02:00
yemkareems 200f9dc39f fix: small typo fix
Signed-off-by: yemkareems <yemkareems@gmail.com>
2025-09-29 09:12:38 +05:30
provokateurin 98081dfd42 feat(ObjectStore): Add option to limit the total size
Signed-off-by: provokateurin <kate@provokateurin.de>
2025-09-10 08:15:28 +02:00
Robin Appelman 0c1d5e8c54 fix: ensure all object store configuration have distict bucket names
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-09-05 18:45:32 +02:00
Robin Appelman 02f4a82088 fix: make bucket mapper work with new multi-object-store config
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-28 17:51:25 +02:00
Robin Appelman b3c53c7436 feat: allow object store configuration aliases for easier migrations
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-14 13:06:51 +02:00
Robin Appelman 2d4bba7b0c feat: add command to get user objectstore config mappings
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-14 13:06:51 +02:00
Robin Appelman 385dd36ff8 feat: allow configuring multiple objectstore configurations
Signed-off-by: Robin Appelman <robin@icewind.nl>
2025-08-14 13:06:51 +02:00