数据解析速度慢？让simdjson告诉你什么是真正的飞快,每秒解析十亿字节的JSON！这个开源项目基于SIMD指令，提供了极速的...

[](https://bugs.chromium.org/p/oss-fuzz/issues/list?sort=-opened&can=1&q=proj:simdjson) [![][license img]][license] [![][licensemit img]][licensemit] [](https://simdjson.github.io/simdjson/) simdjson : Parsing gigabytes of JSON per second =============================================== <img src="images/logo.png" width="10%" style="float: right"> JSON is everywhere on the Internet. Servers spend a *lot* of time parsing it. We need a fresh approach. The simdjson library uses commonly available SIMD instructions and microparallel algorithms to parse JSON 4x faster than RapidJSON and 25x faster than JSON for Modern C++. * **Fast:** Over 4x faster than commonly used production-grade JSON parsers. * **Record Breaking Features:** Minify JSON at 6 GB/s, validate UTF-8 at 13 GB/s, NDJSON at 3.5 GB/s. * **Easy:** First-class, easy to use and carefully documented APIs. * **Strict:** Full JSON and UTF-8 validation, lossless parsing. Performance with no compromises. * **Automatic:** Selects a CPU-tailored parser at runtime. No configuration needed. * **Reliable:** From memory allocation to error handling, simdjson's design avoids surprises. * **Peer Reviewed:** Our research appears in venues like VLDB Journal, Software: Practice and Experience. This library is part of the [Awesome Modern C++](https://awesomecpp.com) list. Table of Contents ----------------- * [Real-world usage](#real-world-usage) * [Quick Start](#quick-start) * [Documentation](#documentation) * [Godbolt](#godbolt) * [Performance results](#performance-results) * [Packages](#packages) * [Bindings and Ports of simdjson](#bindings-and-ports-of-simdjson) * [About simdjson](#about-simdjson) * [Funding](#funding) * [Contributing to simdjson](#contributing-to-simdjson) * [License](#license) Real-world usage ---------------- - [Node.js](https://nodejs.org/) - [ClickHouse](https://github.com/ClickHouse/ClickHouse) - [Meta Velox](https://velox-lib.io) - [Google Pax](https://github.com/google/paxml) - [milvus](https://github.com/milvus-io/milvus) - [QuestDB](https://questdb.io/blog/questdb-release-8-0-3/) - [Clang Build Analyzer](https://github.com/aras-p/ClangBuildAnalyzer) - [Shopify HeapProfiler](https://github.com/Shopify/heap-profiler) - [StarRocks](https://github.com/StarRocks/starrocks) - [Microsoft FishStore](https://github.com/microsoft/FishStore) - [Intel PCM](https://github.com/intel/pcm) - [WatermelonDB](https://github.com/Nozbe/WatermelonDB) - [Apache Doris](https://github.com/apache/doris) - [Dgraph](https://github.com/dgraph-io/dgraph) - [UJRPC](https://github.com/unum-cloud/ujrpc) - [fastgltf](https://github.com/spnda/fastgltf) - [vast](https://github.com/tenzir/vast) - [ada-url](https://github.com/ada-url/ada) - [fastgron](https://github.com/adamritter/fastgron) - [WasmEdge](https://wasmedge.org) - [RonDB](https://github.com/logicalclocks/rondb) If you are planning to use simdjson in a product, please work from one of our releases. Quick Start ----------- The simdjson library is easily consumable with a single .h and .cpp file. 0. Prerequisites: `g++` (version 7 or better) or `clang++` (version 6 or better), and a 64-bit system with a command-line shell (e.g., Linux, macOS, freeBSD). We also support programming environments like Visual Studio and Xcode, but different steps are needed. Users of clang++ may need to specify the C++ version (e.g., `c++ -std=c++17`) since clang++ tends to default on C++98. 1. Pull [simdjson.h](singleheader/simdjson.h) and [simdjson.cpp](singleheader/simdjson.cpp) into a directory, along with the sample file [twitter.json](jsonexamples/twitter.json). You can download them with the `wget` utility: ``` wget https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.h https://raw.githubusercontent.com/simdjson/simdjson/master/singleheader/simdjson.cpp https://raw.githubusercontent.com/simdjson/simdjson/master/jsonexamples/twitter.json ``` 2. Create `quickstart.cpp`: ```c++ #include <iostream> #include "simdjson.h" using namespace simdjson; int main(void) { ondemand::parser parser; padded_string json = padded_string::load("twitter.json"); ondemand::document tweets = parser.iterate(json); std::cout << uint64_t(tweets["search_metadata"]["count"]) << " results." << std::endl; } ``` 3. `c++ -o quickstart quickstart.cpp simdjson.cpp` 4. `./quickstart` ``` 100 results. ``` Documentation ------------- Usage documentation is available: * [Basics](doc/basics.md) is an overview of how to use simdjson and its APIs. * [Performance](doc/performance.md) shows some more advanced scenarios and how to tune for them. * [Implementation Selection](doc/implementation-selection.md) describes runtime CPU detection and how you can work with it. * [API](https://simdjson.github.io/simdjson/) contains the automatically generated API documentation. Godbolt ------------- Some users may want to browse code along with the compiled assembly. You want to check out the following lists of examples: * [simdjson examples with errors handled through exceptions](https://godbolt.org/z/7G5qE4sr9) * [simdjson examples with errors without exceptions](https://godbolt.org/z/e9dWb9E4v) Performance results ------------------- The simdjson library uses three-quarters less instructions than state-of-the-art parser [RapidJSON](https://rapidjson.org). To our knowledge, simdjson is the first fully-validating JSON parser to run at [gigabytes per second](https://en.wikipedia.org/wiki/Gigabyte) (GB/s) on commodity processors. It can parse millions of JSON documents per second on a single core. The following figure represents parsing speed in GB/s for parsing various files on an Intel Skylake processor (3.4 GHz) using the GNU GCC 10 compiler (with the -O3 flag). We compare against the best and fastest C++ libraries on benchmarks that load and process the data. The simdjson library offers full unicode ([UTF-8](https://en.wikipedia.org/wiki/UTF-8)) validation and exact number parsing. <img src="doc/rome.png" width="60%"> The simdjson library offers high speed whether it processes tiny files (e.g., 300 bytes) or larger files (e.g., 3MB). The following plot presents parsing speed for [synthetic files over various sizes generated with a script](https://github.com/simdjson/simdjson_experiments_vldb2019/blob/master/experiments/growing/gen.py) on a 3.4 GHz Skylake processor (GNU GCC 9, -O3). <img src="doc/growing.png" width="60%"> [All our experiments are reproducible](https://github.com/simdjson/simdjson_experiments_vldb2019). For NDJSON files, we can exceed 3 GB/s with [our multithreaded parsing functions](https://github.com/simdjson/simdjson/blob/master/doc/parse_many.md). Packages ------------------------------ [](https://repology.org/project/simdjson/versions) Bindings and Ports of simdjson ------------------------------ We distinguish between "bindings" (which just wrap the C++ code) and a port to another programming language (which reimplements everything). - [ZippyJSON](https://github.com/michaeleisel/zippyjson): Swift bindings for the simdjson project. - [libpy_simdjson](https://github.com/gerrymanoim/libpy_simdjson/): high-speed Python bindings for simdjson using [libpy](https://github.com/quantopian/libpy). - [pysimdjson](https://github.com/TkTech/pysimdjson): Python bindings for the simdjson project. - [cysimdjson](https://github.com/TeskaLabs/cysimdjson): high-speed Python bindings for the simdjson project. - [simdjson-rs](https://github.com/simd-lite): Rust port. - [simdjson-rust](https://github.com/SunDoge/simdjson-rust): Rust wrapper (bindings). - [SimdJsonSharp](https://github.com/EgorBo/SimdJsonSharp): C# version for .NET Core (bindings and full port). - [simdjson