This patch adds managed-memory-based aggregation to Spark SQL / DataFrames. Instead of working with Java objects, this new aggregation path uses sun.misc.Unsafe to manipulate raw memory. This reduces the memory footprint for aggregations, resulting in fewer spills, OutOfMemoryErrors, and garbage collection pauses. As a result, this allows for higher memory utilization. It can also result in better
![[SPARK-7076][SPARK-7077][SPARK-7080][SQL] Use managed memory for aggregations by JoshRosen · Pull Request #5725 · apache/spark](https://cdn-ak-scissors.b.st-hatena.com/image/square/6e623786961cd05f3663bf734989458210997fe7/height=288;version=1;width=512/https%3A%2F%2F2.zoppoz.workers.dev%3A443%2Fhttps%2Fopengraph.githubassets.com%2Ff9a33e016066fde03a4c0fd4c5099d7c44c8e1d9503dc74b251c703a36cce8cb%2Fapache%2Fspark%2Fpull%2F5725)
This patch adds a cache-friendly external sorter which operates on serialized bytes and uses this sorter to implement a new sort operator for Spark SQL and DataFrames. Overview of the new sorter The new sorter design is inspired by Alphasort and implements a key-prefix optimization in order to improve the cache friendliness of the sort. In naive sort implementations, the sorting algorithm operates
![[SPARK-7078] [SPARK-7079] Binary processing sort for Spark SQL by JoshRosen · Pull Request #6444 · apache/spark](https://cdn-ak-scissors.b.st-hatena.com/image/square/ae8ae3b51b8f47b21440347f84e94f7339b715d8/height=288;version=1;width=512/https%3A%2F%2F2.zoppoz.workers.dev%3A443%2Fhttps%2Fopengraph.githubassets.com%2F38dd3170cd95cc1dea643446e7ef8e77ec33c8557dc0df8e66a10070390df1a3%2Fapache%2Fspark%2Fpull%2F6444)
1
リリース、障害情報などのサービスのお知らせ
最新の人気エントリーの配信
処理を実行中です
j次のブックマーク
k前のブックマーク
lあとで読む
eコメント一覧を開く
oページを開く
