When oauth2_access_token_getter is not set (i.e. the user configures
oauth2_refresh_token_eval instead of oauth2_access_token_eval), the
first connection refreshes the token via oauth2_request_url and stores
it in self.oauth2_access_token. Subsequent connections found that
access_token_to_use was None and self.oauth2_access_token was not None,
but there was no branch to reuse the stored token -- so auth_string was
built with None as the Bearer token, causing authentication failure.
Add the missing branch to reuse the cached token when it has not
expired.
Reported-by: astroshark
When the IMAP server (e.g. Proton Mail Bridge) fails to respond to the
welcome greeting, imaplib2 raises a generic 'IMAP4 protocol error'.
Previously this either escaped as an unhandled exception or was marked
as ERROR.CRITICAL, causing offlineimap to stop syncing the account.
Classify all 'IMAP4 protocol error' exceptions as ERROR.REPO so the
existing retry loop in accounts.py handles them gracefully.
These methods are no longer called after the previous commit switched
message copying to single-threaded execution. Remove them from both
BaseFolder and IMAPFolder.
Multiple copy threads calling dstfolder.savemessage() and
statusfolder.savemessage()/deletemessage() concurrently modified
the same messagelist dict without any lock, which can cause
RuntimeError (dict changed size during iteration) in Python 3
and silent data corruption in the status cache.
Replace the threaded copy path with a direct sequential call to
copymessageto(). Folder-level parallelism (one thread per folder)
is preserved; only the per-message threading within a single folder
is removed.
When imapobj.select() raised an OfflineImapError with FOLDER or higher
severity, the connection was not released back to the pool. In the
FOLDER case the inner retry loop called acquireconnection() again,
leaking the previous connection and its semaphore slot. In the
re-raise case the connection was simply abandoned.
Add releaseconnection(imapobj, True) in both error paths so the
connection is always returned to the pool before retrying or
propagating the exception.
When the IMAP server does not support IDLE, noop(imapobj) was called
twice: once immediately in the else branch (releasing the connection
back to the pool), and again after stop_sig.wait(). The second call
operated on a connection that had already been returned to the pool
and could have been assigned to another thread, causing a double
semaphore release and potential pool corruption.
Remove the first noop(imapobj) call so the connection is only released
once, after the wait.
If ExitNotifyThread.start() raises (e.g. RuntimeError 'can't start new
thread' when the OS thread limit is reached), run() never executes and
the semaphore slot acquired in start() is never released. With
maxconnections=N, after N such failures the calling loop blocks forever
on the next start().
Wrap ExitNotifyThread.start() in a try/except so the semaphore is always
released on failure. The exception is re-raised so the caller can log
and handle it normally.
Commit 854f389 added a retry loop in acquireconnection() that reopens the
TCP connection when the socket is dead after an OfflineImapError from
__authn_helper(). The intent was to recover from transient network drops
during authentication (e.g. Protonmail Bridge closing the socket mid-TLS).
However, __authn_helper() also closes the socket when the *server*
actively rejects credentials (e.g. Gmail closes the connection after a
failed PLAIN attempt). In that case the retry logic would:
1. Open a new TCP connection to Gmail (consuming a server-side slot)
2. Fail authentication again with the same credentials
3. Repeat up to 3 times
4. Leave 3 half-open server-side connections → Gmail responds with
'[ALERT] Too many simultaneous connections. (Failure)'
Fix: distinguish the two cases via an 'auth_attempted' flag set on the
OfflineImapError raised by __authn_helper() when exc_stack is non-empty
(i.e. at least one auth method was tried and rejected). The retry is now
skipped when auth_attempted is True, so only genuine pre-auth network
failures (socket dead before any method was sent) trigger a retry.
connectionlock is intentionally released before TCP connection and
authentication (both are slow I/O operations that must not hold a lock).
However, close() can run during that window: it sets self.closing=True,
drains the semaphore via semaphorereset(), and logs out all connections
in assignedconnections + availableconnections — but a thread still in
the I/O phase has an imapobj that is NOT in either list yet.
When the I/O thread later acquires connectionlock to add its new imapobj
to assignedconnections, close() has already finished. The connection is
added to the pool without ever being logged out, leaking the TCP
connection.
Fix: re-check self.closing inside the final 'with self.connectionlock'
block, before appending to assignedconnections. If the server is
closing, call logout() on the new connection and raise so the outer
except block releases the semaphore (single release path — no double
free).
When oauth2_access_token_getter() returns None, the previous code
silently left access_token_to_use as None and continued, resulting in an
auth string of 'auth=Bearer None' sent to the server.
Fix:
- If the getter returns None, clear both oauth2_access_token and
oauth2_access_token_expires_at so no stale expiry is left behind,
and raise an OfflineImapError with a clear message.
- When XOAUTH2 authentication fails (server rejects the token),
invalidate the cached token so the next connection attempt calls
the getter again instead of reusing the rejected token for up to
600 seconds.
The connectionwait() method in IMAPServer was a no-op that introduced a TOCTOU race condition. Removing it eliminates therace without changing the effective concurrency limits. The waitforthread() method in IMAPFolder was calling connectionwait(), which is now removed, so it can also be simplified to a no-op.
- Add a lock to serialize OAuth2 token retrieval and caching to prevent
concurrent threads from invalidating each other's tokens.
- Remove the connectionwait() probe that introduced a TOCTOU race, as concurrency is already correctly enforced by the semaphore in acquireconnection().
This commit adds a try-except block around the IMAP ID command to catch any exceptions that may occur during its execution. If an exception is raised, it will log a warning message with the details of the error, improving the error reporting and allowing users to understand what went wrong without crashing the application.
Add a two-stage CI/CD workflow (publish.yml) that automatically builds
and publishes the package to PyPI.
The workflow consists of two jobs:
1. build: checks out the repository, sets up Python 3.x, installs the
'build' package, runs 'python -m build' to produce sdist and wheel
distributions, and uploads the resulting artifacts for the next job.
2. publish: downloads the artifacts produced by the build job and
publishes them to PyPI using the official pypa/gh-action-pypi-publish
action with OIDC Trusted Publishers authentication, so no API token
needs to be stored as a repository secret.
The workflow is triggered in two ways:
- Automatically, on any tag push matching 'v*':
git tag v8.0.0 && git push origin v8.0.0
- Manually, via the 'workflow_dispatch' event from the GitHub Actions UI.
Replace the bare open()/close() pattern with a context manager and
wrap I/O and encoding failures in OfflineImapError so the user gets
a clear, actionable message instead of a raw Python exception.
Before this change, a missing, unreadable, or non-UTF-8 password
file would surface as an unhandled FileNotFoundError, PermissionError
or UnicodeDecodeError with no indication of which repository was
affected.
The new error message includes the expanded file path and the
repository name:
Unable to read remotepassfile '/path/to/pass' for repository 'Name': ...
Closes: #248
This patch simplifies the return logic in the get_os_sslcertfile_searchpath function by removing the unnecessary try-finally block and directly checking the length of the location list after attempting to append hardcoded paths. If no valid paths are found, it returns None. Previously, using a finally with a return is a bug because it can mask exceptions and lead to unexpected behavior. This change makes the code cleaner and more straightforward.
platform.linux_distribution() was deprecated in Python 3.7 and
removed in Python 3.8. The try/except that fell back to
distro.linux_distribution() was therefore dead code in the try branch
on any supported Python version, making the distro package a de-facto
hard dependency already.
Replace the try/except with a direct top-level import of distro and
use distro.id() instead. distro.id() returns a normalised, lowercase,
hyphen-separated identifier (e.g. "ubuntu", "opensuse-leap", "rhel")
that does not need the .split()[0].lower() post-processing that the old
code applied to the human-readable distribution name.
Because distro.id() returns different identifiers than
linux_distribution()[0].split()[0].lower() did, the CA certificate
lookup table (__DEF_OS_LOCATIONS) is updated to match:
Old key New key(s) distro.id() value
─────────────────────────────────────────────────────────────────
linux-redhat → linux-rhel "rhel"
linux-suse → linux-sles, linux-sled "sles" / "sled"
linux-opensuse → linux-opensuse-leap, "opensuse-leap" /
linux-opensuse-tumbleweed "opensuse-tumbleweed"
Based on a patch from Adam Dinwoodie, as forwarded by Etienne Buira.
When utf_8_support is False (the default, standard RFC 3501 mode),
folder names received from the server are in Modified UTF-7 and must
be kept in that encoding internally. When utf_8_support is True,
names are decoded to UTF-8 for internal use.
Previous code decoded unconditionally in IMAPFolder.__init__, which
would corrupt non-ASCII names received as Modified UTF-7 when
utf_8_support is False. The inverse problem existed in
getfullIMAPname() and the three encode_mailbox_name() call sites in
IMAPRepository: they always converted UTF-8 → Modified UTF-7 before
sending to the server, which is wrong when utf_8_support is False
(names are already in Modified UTF-7 and must not be double-encoded).
Fix by applying the same conditional pattern consistently:
if account.utf_8_support:
name = imaputil.utf8_IMAP(name) # UTF-8 → Modified UTF-7
return imaputil.foldername_to_imapname(name)
This is applied in:
- IMAPFolder.__init__ (decode on receive)
- IMAPFolder.getfullIMAPname (encode before SELECT)
- IMAPRepository.getfolders (folderincludes SELECT)
- IMAPRepository.deletefolder
- IMAPRepository.makefolder_single
encode_mailbox_name() (which always assumed UTF-8 input) is removed
as it is no longer used anywhere.
Based on patch by Etienne Buira <etienne.buira@free.fr>
When `starttls = yes` is configured and the server does not advertise
STARTTLS in its capability list, the previous code silently returned
without attempting TLS, leaving the connection in cleartext.
RFC 2595 section 9 explicitly documents this attack vector:
"A man-in-the-middle attacker can remove STARTTLS from the
capability list or generate a failure response to the STARTTLS
command."
Silently skipping STARTTLS in that case makes offlineimap completely
vulnerable to such capability-stripping attacks: the attacker wins by
simply removing the STARTTLS keyword from the server greeting.
Fix: when STARTTLS is not advertised but is configured, emit a warning
and attempt STARTTLS anyway. If the server genuinely does not support
it, imaplib will raise an error and the existing error-handling path
raises OfflineImapError, aborting the connection. If the attempt
succeeds, TLS is established normally.
The resulting behaviour is:
- Server advertises STARTTLS → unchanged, works as before.
- Server does NOT advertise STARTTLS (possible MITM strip):
- Warning is logged to alert the user.
- STARTTLS is attempted regardless.
- If the server rejects it: OfflineImapError, connection aborted.
- If the server accepts it: TLS is established, sync continues.
Reported by: hartwork
References: RFC 2595 §9
This commit refactors the IMAP folder handling in the offlineimap repository to use the `encode_mailbox_name` function for encoding folder names. This change ensures that folder names are properly encoded when interacting with the IMAP server, improving compatibility and reliability when dealing with folder names that may contain special characters or non-ASCII characters. The `encode_mailbox_name` function combines UTF-8 encoding and quoting as needed, providing a consistent way to handle folder names across the codebase.
This commit updates the README.md file to specify that Python v3.6 or higher is required for offlineimap. It also clarifies the dependencies by marking which ones are required and which are optional, and adds new optional dependencies for SSL certificate testing.
This patch adds documentation for the `allow_nonstandard_capabilities` option in the `offlineimap.conf` configuration file, as well as in the main documentation file `offlineimap.txt`. This option controls how OfflineIMAP behaves when a server does not send a valid CAPABILITY response after a STARTTLS handshake, allowing for a tolerant fallback in such cases. The documentation includes warnings about the potential security implications of enabling this option.
This commit modifies the isusable method in the Blinkenlights class to properly check if the standard output is a terminal. The previous implementation had an incorrect condition that could lead to false positives. The new implementation checks only if sys.stdout is a terminal, which is the correct way to determine if curses can be used. Additionally, it includes a test to ensure that ncurses can start up without issues.
This patch corrects the method signature of the 'isusable' method in the 'Blinkenlights' class to include 'self' as the first parameter, which is necessary for instance methods in Python classes.
The previous implementation had a conditional check for Python versions to avoid calling `curses.initscr()` twice, which could cause issues in certain versions of Python. The refactored code removes the version check, because is always True (version is 3 now) and simply attempts to start ncurses, catching any exceptions that may occur. This simplifies the code and ensures that it works correctly across all supported Python versions without relying on specific version checks.
- Implement tolerance for non-standard IMAP servers that do not provide reliable CAPABILITY after STARTTLS.
- Introduce configuration option 'allow_nonstandard_capabilities' to enable fallback using pre-TLS capabilities.
Before this, authentication will fail after the access token has
expired, and if this is handled by an external program using
oauth2_access_token_eval, offlineimap3 will be able to renew the token
by itself. We should allow oauth2_access_token_eval to run in this case.
One might imagine that authentication needs to be reperformed for
example because of bad internet, and that the token will therefore be
refreshed unnecessarily. I believe most users of
oauth2_access_token_eval will use caching and that this will not be a
big problem.
Signed-off-by: Noa Torstensvik <noa@torstensvik.se>
We introduce a separate closing flag that's checked in
acquireconnection, decoupling the connectionlock from the semaphore
such that they can be acquired separately.
Signed-off-by: Michael Hohmuth <hohmuth@sax.de>
This commit addresses issues where certain IMAP servers (like Protonmail Bridge)
abruptly close the socket during the authentication phase.
Following a simplified approach:
1. Updated __authn_helper to initiate STARTTLS unconditionally at the
beginning of the loop.
2. Modified acquireconnection to catch OfflineImapError. If the socket is
detected as dead during authentication, it now logs a warning and retries
the entire connection process from scratch (up to 3 attempts).
3. This ensures each retry starts with a fresh connection and all configured
authentication methods available, avoiding permanent list modification.
The original Travis CI configuration was outdated, still targeting Python 2.7.
This commit updates the test matrix to include all supported Python 3 versions (3.6 through 3.13) to match the project's goals.
Additionally:
- Set dist: jammy (Ubuntu 22.04) for Linux builds to support modern Python versions.
- Updated the macOS (OSX) environment to use Python 3.12.0 for its tests.
Certain IMAP servers (like Protonmail Bridge) suffer from bugs where they hang and drop the connection when receiving specific authentication commands (such as `AUTHENTICATE PLAIN` immediately after STARTTLS). Previously, if the socket died during an authentication attempt, offlineimap would abort the entire synchronization process immediately.
This commit introduces `AuthMethodSocketDeadError` to handle this scenario gracefully. When an authentication mechanism kills the socket, it is temporarily removed from the list of available mechanisms for that session. Offlineimap then reconnects and retries the remaining authentication mechanisms (e.g., falling back from PLAIN to LOGIN), allowing synchronization to continue successfully without entering an infinite loop.
ç
When a user sets `starttls = yes`, offlineimap expects to negotiate a secure TLS tunnel. However, if a MITM attacker intercepts the connection and removes the `STARTTLS` capability from the server's greeting, the `if 'STARTTLS' in imapobj.capabilities` check fails silently. This causes offlineimap to skip TLS negotiation and proceed to send credentials in plaintext.
This commit introduces a strict check in `__start_tls()`: if the user requested STARTTLS but the server does not advertise it, offlineimap will immediately abort with a clear error message, protecting the user's password from being exposed. To connect insecurely, the user must explicitly set `starttls = no`.
When fetching emails with defects or malformed bytes (e.g., spam or broken
MIME boundaries), the Python 3 `email.parser` handles un-decodable bytes
by safely substituting them with the Unicode Replacement Character `\ufffd`
(using the `errors='replace'` handler by default).
However, a problem arises during the `_fetch_from_imap` sync process when
OfflineIMAP3 tests if the message can be serialized back to bytes via
`as_bytes()`. If the malformed text part was originally declared as
`us-ascii` (or left unspecified, defaulting to ASCII), Python attempts to
encode the `\ufffd` string back to bytes using the ASCII codec. Since `\ufffd`
is out of the ASCII range, this triggers a `UnicodeEncodeError`, causing
OfflineIMAP3 to raise an `OfflineImapError` and entirely skip syncing the
message.
This commit fixes the issue by catching the `UnicodeEncodeError` when
`as_bytes()` fails. It then walks through the message parts, identifies
the text payloads that cannot be encoded with their current charset, and
dynamically forces their charset to `utf-8`. This allows Python to safely
encode the `\ufffd` character (automatically applying base64/quoted-printable
transfer encoding if needed), successfully serializing the message so it
can be synced without crashing.
Closes#240Closes#229Closes#224Closes#160
Rodolfo García Peñas (kix)
Andreas Schacker
Noa Torstensvik
Michael Hohmuth
Derek Schrock