From: "tompng (tomoya ishida) via ruby-core" <ruby-core@...>
Date: 2024-06-01T03:37:46+00:00
Subject: [ruby-core:118132]  [Ruby master Bug#20517] `Ripper.tokenize('"\\M-あ"')` separates encoding valid string to encoding invalid string.

Issue #20517 has been reported by tompng (tomoya ishida).

----------------------------------------
Bug #20517: `Ripper.tokenize('"\\M-���"')` separates encoding valid string to encoding invalid string.
https://bugs.ruby-lang.org/issues/20517

* Author: tompng (tomoya ishida)
* Status: Open
* ruby -v: ruby 3.4.0dev (2024-04-11T08:57:52Z master e7f8db9079) [x86_64-linux]
* Backport: 3.1: UNKNOWN, 3.2: UNKNOWN, 3.3: UNKNOWN
----------------------------------------
~~~ruby
Ripper.tokenize '"\\M-���"'
=> ["\"", "\\M-\xE3", "\x81", "\x82", "\""]
~~~
I expect all tokens to be valid_encoding if the source string is valid_encoding.
Similar to https://bugs.ruby-lang.org/issues/20030



-- 
https://bugs.ruby-lang.org/
 ______________________________________________
 ruby-core mailing list -- ruby-core@ml.ruby-lang.org
 To unsubscribe send an email to ruby-core-leave@ml.ruby-lang.org
 ruby-core info -- https://ml.ruby-lang.org/mailman3/postorius/lists/ruby-core.ml.ruby-lang.org/