From: "nobu (Nobuyoshi Nakada) via ruby-core" Date: 2024-06-01T10:40:17+00:00 Subject: [ruby-core:118134] [Ruby master Bug#20517] `Ripper.tokenize('"\\M-あ"')` separates encoding valid string to encoding invalid string. Issue #20517 has been updated by nobu (Nobuyoshi Nakada). https://github.com/ruby/ruby/pull/10893 ---------------------------------------- Bug #20517: `Ripper.tokenize('"\\M-���"')` separates encoding valid string to encoding invalid string. https://bugs.ruby-lang.org/issues/20517#change-108573 * Author: tompng (tomoya ishida) * Status: Open * ruby -v: ruby 3.4.0dev (2024-04-11T08:57:52Z master e7f8db9079) [x86_64-linux] * Backport: 3.1: REQUIRED, 3.2: REQUIRED, 3.3: REQUIRED ---------------------------------------- ~~~ruby Ripper.tokenize '"\\M-���"' => ["\"", "\\M-\xE3", "\x81", "\x82", "\""] ~~~ I expect all tokens to be valid_encoding if the source string is valid_encoding. Similar to https://bugs.ruby-lang.org/issues/20030 -- https://bugs.ruby-lang.org/ ______________________________________________ ruby-core mailing list -- ruby-core@ml.ruby-lang.org To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/