vim-mirror

mirror of https://github.com/vim/vim.git synced 2026-02-01 11:34:23 +01:00

Files

Yasuhiro Matsumoto 2b184d4b97 patch 9.1.2124: blob2str() does not handle UTF-16 encoding

Problem:  blob2str() does not handle UTF-16 encoding
          (Hirohito Higashi)
Solution: Refactor the code and fix remaining issues, see below
          (Yasuhiro Matsumoto).

blob2str() function did not properly handle UTF-16/UCS-2/UTF-32/UCS-4
encodings with endianness suffixes (e.g., utf-16le, utf-16be, ucs-2le).
The encoding name was canonicalized too aggressively, losing the
endianness information needed by iconv.

This change include few fixes:

- Preserve the raw encoding name with endianness suffix for iconv calls
- Normalize encoding names properly: "ucs2be" → "ucs-2be", "utf16le" →
  "utf-16le"
- For multi-byte encodings (UTF-16/32, UCS-2/4), convert the entire blob
  first, then split by newlines

convert_string() cannot handle UTF-16 because it uses string_convert()
which expects NUL-terminated strings. UTF-16 contains 0x00 bytes within
characters (e.g., "H" = 0x48 0x00), causing premature termination.
Therefore, for UTF-16/32 encodings, the fix uses string_convert_ext()
with an explicit input length to convert the entire blob at once.

The code appends two NUL bytes (ga_append(&blob_ga, NUL) twice) because
UTF-16 requires a 2-byte NUL terminator (0x00 0x00), not a single-byte
NUL.

- src/strings.c: Add from_encoding_raw to preserve endianness, special
  handling for UTF-16/32 and UCS-2/4
- src/mbyte.c: Fix convert_setup_ext() to use == ENC_UNICODE instead of
  & ENC_UNICODE. The bitwise AND was incorrectly treating UTF-16/UCS-2
  (which have ENC_UNICODE + ENC_2BYTE etc.) as UTF-8, causing iconv
  setup to be skipped.

fixes:  #19198
closes: #19246

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Yasuhiro Matsumoto <mattn.jp@gmail.com>
Signed-off-by: Christian Brabandt <cb@256bit.org>

2026-01-31 15:59:01 +00:00

arabic.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

autocmd.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

builtin.txt

patch 9.1.2124: blob2str() does not handle UTF-16 encoding

2026-01-31 15:59:01 +00:00

change.txt

runtime(doc): clarify the behaviour of 'fo-m'

2026-01-08 21:39:15 +00:00

channel.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

cmdline.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

debug.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

debugger.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

develop.txt

runtime(doc): Remove outdated agide.org link

2025-12-13 17:42:59 +01:00

diff.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

digraph.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

doctags.c

…

doctags.vim

…

editing.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

eval.txt

patch 9.1.2077: Vim9: type issue when appending item to a list

2026-01-10 18:14:13 +00:00

evim-fr.1

…

evim-fr.UTF-8.1

…

evim-it.1

…

evim-it.UTF-8.1

…

evim-ja.UTF-8.1

translation(jp): Update Japanese Translation for Vim 9.2 Release

2026-01-19 18:07:28 +00:00

evim-pl.1

…

evim-pl.UTF-8.1

…

evim-ru.1

…

evim-ru.UTF-8.1

…

evim-sv.1

translation(sv): Add Swedish translation

2026-01-26 19:40:37 +00:00

evim-sv.UTF-8.1

translation(sv): Add Swedish translation

2026-01-26 19:40:37 +00:00

evim-tr.1

…

evim-tr.UTF-8.1

…

evim.1

…

evim.man

…

farsi.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

filetype.txt

patch 9.1.1962: filetype: Erlang application resource files are not recognized

2025-12-07 19:12:33 +01:00

fold.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

ft_ada.txt

runtime(doc): Use correct option-name tags

2025-08-06 12:44:36 +02:00

ft_context.txt

runtime(doc): Tweak documentation style in {ft_context,terminal}.txt

2026-01-18 20:50:11 +00:00

ft_hare.txt

runtime(doc): Tweak documentation style more in options and ft_hare

2025-09-14 11:05:46 -04:00

ft_mp.txt

runtime: Update files for ConTeXt, METAFONT, and MetaPost.

2026-01-11 18:45:42 +00:00

ft_ps1.txt

…

ft_raku.txt

…

ft_rust.txt

runtime(doc): fix inconsistent indent in ft_rust.txt

2026-01-25 18:17:40 +00:00

ft_sql.txt

runtime(doc): remove dead links

2025-08-10 10:28:16 +02:00

gui_w32.txt

runtime(doc): Reformat example at :h gui-w32-fullscreen

2026-01-31 09:42:05 +00:00

gui_x11.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

gui.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

hangulin.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

hebrew.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

help.svx

translation(sv): Add Swedish translation

2026-01-26 19:40:37 +00:00

help.txt

patch 9.1.1893: ICCF charity will dissolve

2025-11-01 16:15:39 +00:00

help.txt.info

…

helphelp.txt

runtime(doc): clarify the help style a bit

2026-01-25 17:59:30 +00:00

howto.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

if_cscop.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

if_lua.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

if_mzsch.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

if_ole.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

if_perl.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

if_pyth.txt

runtime(doc): clarify vim.eval behavior with Vim special variables

2026-01-11 18:23:20 +00:00

if_ruby.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

if_sniff.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

if_tcl.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

indent.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

index.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

insert.txt

patch 9.1.2063: A few typos in the code and runtime files

2026-01-07 21:33:28 +00:00

intro.txt

patch 9.1.1933: completion: complete_match() is not useful

2025-11-27 21:28:05 +00:00

Make_all.mak

patch 9.1.1485: missing Wayland clipboard support

2025-06-27 21:10:35 +02:00

Make_mvc.mak

translation(sv): Add Swedish translation

2026-01-26 19:40:37 +00:00

Makefile

runtime(doc): non-portable sed regex in Makefile for pi_netrw.txt rule

2025-03-28 19:11:11 +01:00

makehtml.awk

…

maketags.awk

…

map.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

mbyte.txt

runtime(doc): use codepoint consistently

2025-12-17 23:24:57 +01:00

message.txt

patch 9.1.1965: q can accidentally start recording at more prompt

2025-12-09 13:27:00 +01:00

mlang.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

motion.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

netbeans.txt

runtime(doc): mark a-a-p website as dead

2026-01-25 15:23:07 +00:00

options.txt

patch 9.1.2114: modeless selection not copied to * register

2026-01-29 19:18:10 +00:00

os_390.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_amiga.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_beos.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_dos.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_haiku.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_mac.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_mint.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_msdos.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_os2.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_qnx.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_risc.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_unix.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_vms.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

os_win32.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

pattern.txt

runtime(doc): 'ignorecase' affects character classes in the old engine

2026-01-28 21:58:28 +00:00

pi_getscript.txt

runtime(doc): remove dead links

2025-08-10 10:28:16 +02:00

pi_gzip.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

pi_logipat.txt

…

pi_netrw.txt

runtime(doc): remove dead links

2025-08-10 10:28:16 +02:00

pi_paren.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

pi_spec.txt

…

pi_tar.txt

runtime(doc): remove dead links

2025-08-10 10:28:16 +02:00

pi_tutor.txt

runtime(doc): remove outdated Contribution section in pi_tutor

2025-05-19 19:34:44 +02:00

pi_vimball.txt

runtime(doc): Use correct option-name tags

2025-08-06 12:44:36 +02:00

pi_zip.txt

runtime(zip): add *.pkpass to list of zip extensions

2025-10-06 17:56:20 +00:00

popup.txt

runtime(doc): add termdebug tag, remove term "floating window"

2026-01-08 19:35:38 +00:00

print.txt

runtime(doc): use codepoint consistently

2025-12-17 23:24:57 +01:00

quickfix.txt

runtime(compiler): add pyright Python type checker

2025-12-27 14:07:21 +00:00

quickref.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

quotes.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

recover.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

remote.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

repeat.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

rileft.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

russian.txt

runtime(doc): Fix overlength lines in :help {russian,vietnamese}.txt

2026-01-25 18:05:43 +00:00

scroll.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

sign.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

spell.txt

runtime(doc): mark a-a-p website as dead

2026-01-25 15:23:07 +00:00

sponsor.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00

starting.txt

runtime(doc): clarify the behavior of CTRL-Z

2025-12-20 16:54:07 +00:00

syntax.txt

runtime(doc): fix missing code block marker in ft-python-syntax

2026-01-26 19:56:27 +00:00

tabpage.txt

runtime(doc): Clean up file header whitespace

2025-11-09 19:23:50 +00:00