summaryrefslogtreecommitdiff
path: root/encoding.c
AgeCommit message (Collapse)Author
2024-11-13string.c: Directly create strings with the correct encodingJean Boussier
While profiling msgpack-ruby I noticed a very substantial amout of time spent in `rb_enc_associate_index`, called by `rb_utf8_str_new`. On that benchmark, `rb_utf8_str_new` is 33% of the total runtime, in big part because it cause GC to trigger often, but even then `5.3%` of the total runtime is spent in `rb_enc_associate_index` called by `rb_utf8_str_new`. After closer inspection, it appears that it's performing a lot of safety check we can assert we don't need, and other extra useless operations, because strings are first created and filled as ASCII-8BIT and then later reassociated to the desired encoding. By directly allocating the string with the right encoding, it allow to skip a lot of duplicated and useless operations. After this change, the time spent in `rb_utf8_str_new` is down to `28.4%` of total runtime, and most of that is GC. Notes: Merged: https://github.com/ruby/ruby/pull/12076
2024-10-05Move common code to `enc_compatible_latter`Nobuyoshi Nakada
2024-06-27Fix corruption of internal encoding stringPeter Zhu
[Bug #20598] Just like [Bug #20595], Encoding#name_list and Encoding#aliases can have their strings corrupted when Encoding.default_internal is set to nil. Co-authored-by: Matthew Valentine-House <[email protected]>
2024-06-27Fix corruption of encoding name stringPeter Zhu
[Bug #20595] enc_set_default_encoding will free the C string if the encoding is nil, but the C string can be used by the encoding name string. This will cause the encoding name string to be corrupted. Consider the following code: Encoding.default_internal = Encoding::ASCII_8BIT names = Encoding.default_internal.names p names Encoding.default_internal = nil p names It outputs: ["ASCII-8BIT", "BINARY", "internal"] ["ASCII-8BIT", "BINARY", "\x00\x00\x00\x00\x00\x00\x00\x00"] Co-authored-by: Matthew Valentine-House <[email protected]>
2024-04-18Add a hint of `ASCII-8BIT` being `BINARY`Jean Boussier
[Feature #18576] Since outright renaming `ASCII-8BIT` is deemed to backward incompatible, the next best thing would be to only change its `#inspect`, particularly in exception messages.
2024-03-06Refactor VM root modulesJean Boussier
This `st_table` is used to both mark and pin classes defined from the C API. But `vm->mark_object_ary` already does both much more efficiently. Currently a Ruby process starts with 252 rooted classes, which uses `7224B` in an `st_table` or `2016B` in an `RArray`. So a baseline of 5kB saved, but since `mark_object_ary` is preallocated with `1024` slots but only use `405` of them, it's a net `7kB` save. `vm->mark_object_ary` is also being refactored. Prior to this changes, `mark_object_ary` was a regular `RArray`, but since this allows for references to be moved, it was marked a second time from `rb_vm_mark()` to pin these objects. This has the detrimental effect of marking these references on every minors even though it's a mostly append only list. But using a custom TypedData we can save from having to mark all the references on minor GC runs. Addtionally, immediate values are now ignored and not appended to `vm->mark_object_ary` as it's just wasted space.
2024-01-03Fix memory leak in setting encodingsPeter Zhu
There is a memory leak in Encoding.default_external= and Encoding.default_internal= because the duplicated name is not freed when overwriting. 10.times do 1_000_000.times do Encoding.default_internal = nil end puts `ps -o rss= -p #{$$}` end Before: 25664 41504 57360 73232 89168 105056 120944 136816 152720 168576 After: 9648 9648 9648 9680 9680 9680 9680 9680 9680 9680
2023-12-07Free everything at shutdownAdam Hess
when the RUBY_FREE_ON_SHUTDOWN environment variable is set, manually free memory at shutdown. Co-authored-by: Nobuyoshi Nakada <[email protected]> Co-authored-by: Peter Zhu <[email protected]>
2023-02-07Mark Encoding as Write Barrier protectedJean Boussier
It doesn't even have a mark function. It's only about a hundred objects, but not reason to scan them every time. Notes: Merged: https://github.com/ruby/ruby/pull/7254
2023-01-11Remove Encoding#replicateBenoit Daloze
Notes: Merged: https://github.com/ruby/ruby/pull/7079
2022-12-16surpress warningKoichi Sasada
now `enc_table->list` is not a pointer.
2022-12-16fixed encoding tableKoichi Sasada
This reduces global lock acquiring for reading. https://bugs.ruby-lang.org/issues/18949 Notes: Merged: https://github.com/ruby/ruby/pull/6935
2022-09-12Remove get_actual_encoding() and the dynamic endian detection for dummy ↵Benoit Daloze
UTF-16/UTF-32 * And simplify callers of get_actual_encoding(). * See [Feature #18949]. * See https://github.com/ruby/ruby/pull/6322#issuecomment-1242758474
2022-09-10Deprecate Encoding#replicateBenoit Daloze
* See [Feature #18949].
2022-07-21Expand tabs [ci skip]Takashi Kokubun
[Misc #18891] Notes: Merged: https://github.com/ruby/ruby/pull/6094
2022-07-19Rename ENCINDEX_ASCII to ENCINDEX_ASCII_8BITJean Boussier
Otherwise it's way too easy to confuse it with US_ASCII. Notes: Merged: https://github.com/ruby/ruby/pull/6127
2022-04-01[DOC] Main doc for encodings moved from encoding.c to doc/encodings.rdoc (#5748)Burdette Lamar
Main doc for encodings moved from encoding.c to doc/encodings.rdoc Notes: Merged-By: BurdetteLamar <[email protected]>
2021-10-24suppress warnings for probable NULL dererefencesNobuyoshi Nakada
Notes: Merged: https://github.com/ruby/ruby/pull/5015
2021-10-05include/ruby/encoding.h: convert macros into inline functions卜部昌平
Less macros == huge win. Notes: Merged: https://github.com/ruby/ruby/pull/4909
2021-10-02Make encoding loading not issue warningJeremy Evans
Instead of relying on setting an unsetting ruby_verbose, which is not thread-safe, restructure require_internal and load_lock to accept a warn argument for whether to warn, and add rb_require_internal_silent to require without warnings. Use rb_require_internal_silent when loading encoding. Note this does not modify ruby_debug and errinfo handling, those remain thread-unsafe. Also silent requires when loading transcoders. Notes: Merged: https://github.com/ruby/ruby/pull/4887
2021-08-22Add rb_encoding_check functionS-H-GAMELINKS
Notes: Merged: https://github.com/ruby/ruby/pull/4759
2021-08-02Using RBOOL macroS.H
Notes: Merged: https://github.com/ruby/ruby/pull/4695 Merged-By: nobu <[email protected]>
2021-03-22rb_enc_interned_str: handle autoloaded encodingsJean Boussier
If called with an autoloaded encoding that was not yet initialized, `rb_enc_interned_str` would crash with a NULL pointer exception. See: https://github.com/ruby/ruby/pull/4119#issuecomment-800189841 Notes: Merged: https://github.com/ruby/ruby/pull/4290
2020-12-17No sync on ASCII/US_ASCCII/UTF-8Koichi Sasada
rb_enc_from_index(index) doesn't need locking if index specify ASCII/US_ASCCII/UTF-8. rb_enc_from_index() is called frequently so it has impact. user system total real r_parallel/miniruby 174 0.000209 0.000000 5.559872 ( 1.811501) r_parallel/master_mini 175 0.000238 0.000000 12.664707 ( 3.523641) (repeat x1000 `s.split(/,/)` where s = '0,,' * 1000) Notes: Merged: https://github.com/ruby/ruby/pull/3916
2020-12-08Set default for Encoding.default_external to UTF-8 on Windows (#2877)Lars Kanis
* Use UTF-8 as default for Encoding.default_external on Windows * Document UTF-8 change on Windows to Encoding.default_external fix https://bugs.ruby-lang.org/issues/16604 Notes: Merged-By: nurse <[email protected]>
2020-11-18fix public interfaceKoichi Sasada
To make some kind of Ractor related extensions, some functions should be exposed. * include/ruby/thread_native.h * rb_native_mutex_* * rb_native_cond_* * include/ruby/ractor.h * RB_OBJ_SHAREABLE_P(obj) * rb_ractor_shareable_p(obj) * rb_ractor_std*() * rb_cRactor and rm ractor_pub.h and rename srcdir/ractor.h to srcdir/ractor_core.h (to avoid conflict with include/ruby/ractor.h) Notes: Merged: https://github.com/ruby/ruby/pull/3775
2020-10-21Don't redefine #rb_intern over and over againStefan Stüben
Notes: Merged: https://github.com/ruby/ruby/pull/3589
2020-10-20Revert "reduce lock for encoding"Koichi Sasada
This reverts commit de17e2dea137bc5ba9f00e3acec32792d0dbb2eb. This patch can introduce race condition because of conflicting read/write access for enc_table::default_list. Maybe we need to freeze default_list at the end of Init_encdb() in enc/encdb.c.
2020-10-19reduce lock for encodingKoichi Sasada
To reduce the number of locking for encoding manipulation, enc_table::list is splited to ::default_list and ::additional_list. ::default_list is pre-allocated and no need locking to access to the ::default_list. If additional encoding space is needed, use ::additional_list and this list need to use locking. However, most of case, ::default_list is enough. Notes: Merged: https://github.com/ruby/ruby/pull/3664
2020-10-15Check encoding name to replicateNobuyoshi Nakada
https://hackerone.com/reports/954433
2020-10-14freeze Encoding objectsKoichi Sasada
Encoding objects can be accessed in multi-ractors and there is no state to mutate. So we can mark it as frozen and shareable. [Bug #17188] Notes: Merged: https://github.com/ruby/ruby/pull/3654
2020-10-14sync enc_table and rb_encoding_listKoichi Sasada
enc_table which manages Encoding information. rb_encoding_list also manages Encoding objects. Both are accessed/modified by ractors simultaneously so that they should be synchronized. For enc_table, this patch introduced GLOBAL_ENC_TABLE_ENTER/LEAVE/EVAL to access this table with VM lock. To make shortcut, three new global variables global_enc_ascii, global_enc_utf_8, global_enc_us_ascii are also introduced. For rb_encoding_list, we split it to rb_default_encoding_list (256 entries) and rb_additional_encoding_list. rb_default_encoding_list is fixed sized Array so we don't need to synchronized (and most of apps only needs it). To manage 257 or more encoding objects, they are stored into rb_additional_encoding_list. To access rb_additional_encoding_list., VM lock is needed. Notes: Merged: https://github.com/ruby/ruby/pull/3654
2020-05-16Fallback to built-in UTF-8 for minirubyNobuyoshi Nakada
Source code encoding is defaulted to UTF-8 now too.
2020-05-11sed -i 's|ruby/impl|ruby/internal|'卜部昌平
To fix build failures. Notes: Merged: https://github.com/ruby/ruby/pull/3079
2020-05-11sed -i s|ruby/3|ruby/impl|g卜部昌平
This shall fix compile errors. Notes: Merged: https://github.com/ruby/ruby/pull/3079
2020-05-11Added more NORETURN declarationsNobuyoshi Nakada
2020-04-08Merge pull request #2991 from shyouhei/ruby.h卜部昌平
Split ruby.h Notes: Merged-By: shyouhei <[email protected]>
2020-02-13Get rid of warnings/exceptions at cleanupNobuyoshi Nakada
After the encoding index instance variable is removed when all instance variables are removed in `obj_free`, then `rb_str_free` causes uninitialized instance variable warning and nil-to-integer conversion exception. Both cases result in object allocation during GC, and crashes.
2020-02-10rb_enc_str_asciionly_p expects T_STRING卜部昌平
This `str2` variable can be non-string (regexp etc.) but the previous code passed it directly to rb_enc_str_asciionly_p(), which expects its argument be a string. Let's enforce that constraint.
2020-02-07more on NULL versus functions.卜部昌平
Function pointers are not void*. See also ce4ea956d24eab5089a143bba38126f2b11b55b6 8427fca49bd85205f5a8766292dd893f003c0e48
2020-02-03Fix description of Encoding.default_(in|ex)ternalLars Kanis
Data written to files is not transcoded per default, but only when default_internal is set. The default for default_internal is nil and doesn't depend on the source file encoding. Notes: Merged: https://github.com/ruby/ruby/pull/2878
2019-12-26decouple internal.h headers卜部昌平
Saves comitters' daily life by avoid #include-ing everything from internal.h to make each file do so instead. This would significantly speed up incremental builds. We take the following inclusion order in this changeset: 1. "ruby/config.h", where _GNU_SOURCE is defined (must be the very first thing among everything). 2. RUBY_EXTCONF_H if any. 3. Standard C headers, sorted alphabetically. 4. Other system headers, maybe guarded by #ifdef 5. Everything else, sorted alphabetically. Exceptions are those win32-related headers, which tend not be self- containing (headers have inclusion order dependencies). Notes: Merged: https://github.com/ruby/ruby/pull/2711
2019-12-22[Doc] Fix sample in Encoding#namesQuestionDriven
Notes: Merged: https://github.com/ruby/ruby/pull/2760
2019-12-22[Doc] Fix wrong example in Encoding.aliasesQuestionDriven
Notes: Merged: https://github.com/ruby/ruby/pull/2760
2019-11-18Deprecate taint/trust and related methods, and make the methods no-opsJeremy Evans
This removes the related tests, and puts the related specs behind version guards. This affects all code in lib, including some libraries that may want to support older versions of Ruby. Notes: Merged: https://github.com/ruby/ruby/pull/2476
2019-11-18Warn on access/modify of $SAFE, and remove effects of modifying $SAFEJeremy Evans
This removes the security features added by $SAFE = 1, and warns for access or modification of $SAFE from Ruby-level, as well as warning when calling all public C functions related to $SAFE. This modifies some internal functions that took a safe level argument to no longer take the argument. rb_require_safe now warns, rb_require_string has been added as a version that takes a VALUE and does not warn. One public C function that still takes a safe level argument and that this doesn't warn for is rb_eval_cmd. We may want to consider adding an alternative method that does not take a safe level argument, and warn for rb_eval_cmd. Notes: Merged: https://github.com/ruby/ruby/pull/2476
2019-11-05Moved Init_encoding from wrong place [Bug #16292]Nobuyoshi Nakada
2019-10-09avoid overflow in integer multiplication卜部昌平
This changeset basically replaces `ruby_xmalloc(x * y)` into `ruby_xmalloc2(x, y)`. Some convenient functions are also provided for instance `rb_xmalloc_mul_add(x, y, z)` which allocates x * y + z byes. Notes: Merged: https://github.com/ruby/ruby/pull/2540
2019-08-04Better wording for __ENCODING__Lars Kanis
"locale encoding" is misleading since it doesn't mean Encoding.find("locale") but the encoding used to interpret the script file. It's therefore better to call it "script encoding" as in the paragraphs above. Closes: https://github.com/ruby/ruby/pull/1611
2019-07-23Let the index boundary check in rb_enc_from_index be flagged as unlikelyLourens Naudé
[Misc #15806] Closes: https://github.com/ruby/ruby/pull/2128
generated by cgit v1.2.3 (git 2.25.1) at 2025-05-09 15:47:41 +0000