General Dev>In The News

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

CommunityNews

CommunityNews

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

Using NumPy to replace Pandas GroupBy-Apply pattern for performance.
If you use PySpark a lot you would know that the DataFrame API is great. However there are times when it is not sufficient because it does not cover every single piece of functionality we may want. This is where the Pandas UDF functionality comes in. The nice thing about the Pandas UDF functionality is that it uses Arrow for data transfer between Spark and Pandas which minimizes serialization-deserialization costs. I have a slight preference for Pandas Function API over Pandas UDF but now let’s get to the meat of the post which is about speeding up the Pandas GroupBy-Apply pattern by using NumPy instead.

Read in full here:

This thread was posted by one of our members via one of our news source trackers.

View thread on forum
#performance#pandas
0 442 0

Popular General Dev topics Top

First poster: bot
General Dev>In The News

Hush Keyboards with Hushboard
Hush Keyboards with Hushboard. Yesterday while surfing the ASCII highways of IRC (yes, IRC) a URL linking to a MacOS application scrolle...
popey.com
/keyboards
0 1050 0
New
CommunityNews
General Dev>In The News

LiveKit – open-source, high performance WebRTC infrastructure
GitHub - livekit/livekit: Scalable, high-performance WebRTC SFU. SDKs in JavaScript, React, React Native, Flutter, Swift, Kotlin, Unity/C...
github.com
#infrastructure#performance#webrtc
1 1103 1
New
First poster: bot
General Dev>In The News

The cheapest flash microcontroller you can buy is actually an Arm Cortex-M0+
The cheapest flash microcontroller you can buy is actually an Arm Cortex-M0+ - Jay Carlson. Puya’s 10-cent PY32 series is complicating t...
jaycarlson.net
#arm
0 1022 0
New
First poster: bot
General Dev>In The News

ChatML: ChatGPT API expects a structured format, called Chat Markup Language
openai-python/chatml.md at main · openai/openai-python. The OpenAI Python library provides convenient access to the OpenAI API from appl...
github.com
#chat#api#chatgpt
0 935 0
New
First poster: bot
General Dev>In The News

When Zig is safer and faster than Rust
When Zig is safer and faster than Rust. There are endless debates online about Rust vs. Zig, this post explores a side of the argument I...
zackoverflow.dev
/rust/zig
0 929 0
New
First poster: bot
General Dev>In The News

Zig now has built-in HTTP server and client in std
zig/http.zig at 7cf2cbb33ef34c1d211135f56d30fe23b6cacd42 · ziglang/zig. General-purpose programming language and toolchain for maintaini...
github.com
/zig#http
0 2595 0
New
CommunityNews
General Dev>In The News

Apple Patents Suggest Future AirPods Could Monitor Biosignals and Brain Activity
Apple Patents Suggest Future AirPods Could Monitor Biosignals & Brain Activity - AppleMagazine. The US Patent & Trademark Office...
applemagazine.com
#apple#monitor
0 916 0
New
First poster: AstonJ
General Dev>In The News

Jan: An open source alternative to ChatGPT that runs on the desktop
Jan | Rethink the Computer. Jan turns your computer into an AI machine by running LLMs locally on your computer. It’s a privacy-focus, l...
jan.ai
#desktop#chatgpt
4 712 4
New
First poster: alvinkatojr
General Dev>In The News

Distributed Systems Programming Has Stalled
Over the last decade, we’ve seen great advancements in distributed systems, but the way we program them has seen few fundamental improvem...
shadaj.me
#programming
4 163 2
New
CommunityNews
General Dev>In The News

Self-Hosting a Firefox Sync Server
After switching from Firefox to LibreWolf, I became interested in the idea of self-hosting my own Firefox Sync server. Although I had see...
blog.diego.dev
#hosting#firefox
0 145 0
New

Other popular topics Top

AstonJ
General Dev>Hardware

Which keyboard do you have?
If it’s a mechanical keyboard, which switches do you have? Would you recommend it? Why? What will your next keyboard be? Pics always w...
#hardware/keyboards#sticky#mechanical-keyboards
144 8358 50
New
wolf4earth
Science/Tech>Health & Diet

How do you keep fit and healthy?
@AstonJ prompted me to open this topic after I mentioned in the lockdown thread how I started to do a lot more for my fitness. https://f...
#fitness#heath
193 4376 82
New
axelson
Cross Platform>Libraries/Tools

Obsidian – a cross platform app to help you create a knowledge base with Markdown files
I’ve been really enjoying obsidian.md: It is very snappy (even though it is based on Electron). I love that it is all local by defaul...
#apps#macos#windows#linux/markdown#obsidian
38 3905 19
New
DevotionGeo
Backend>Chat

Would you use Erlang now when there is Elixir?
Why, if your answer is yes?
/elixir/erlang
167 4267 52
New
PragmaticBookshelf
Community>In The Spotlight

Spotlight: Dmitry Zinoviev (Author) Interview and AMA!
Author Spotlight Dmitry Zinoviev @aqsaqal Today we’re putting our spotlight on Dmitry Zinoviev, author of Data Science Essentials in ...
#author-spotlight/python/book-complex-network-analysis-in-python/book-data-science-essentials-in-python/book-resourceful-code-reuse/book-pythonic-programming
33 4660 14
New
First poster: CommunityNews
AI>In The News

How to fix the eyes in AI-generated images
aidemos.info
0 2869 0
New
PragmaticBookshelf
Community>In The Spotlight

Spotlight: David Bryant Copeland (Author) Interview and AMA!
Author Spotlight: David Bryant Copeland @davetron5000 We’re so happy to bring you another Author Spotlight, a series where we sit dow...
#author-spotlight/book-agile-web-development-with-rails-6/book-rails-angular-postgres-and-bootstrap-second-edition/book-sustainable-web-development-with-ruby-on-rails
19 3303 15
New
PragmaticBookshelf
Community>In The Spotlight

Spotlight: Noel Rappin (Author)
A Ruby-Centric Chat with Noel Rappin @noelrappin Once you start noodling around with Ruby you quickly figure out, as Noel Rappi...
#author-spotlight/ruby/rails/book-programming-ruby-3-2-5th-edition
41 2642 20
New
AnfaengerAlex
Android>Questions

Unresolved Reference to android in build.gradle.kts – Beginner Issue
Hello, I’m a beginner in Android development and I’m facing an issue with my project setup. In my build.gradle.kts file, I have the foll...
#binding
0 1502 2
New
AstonJ
AI>In The News

DeepSeek (671B) running on a cluster of 8 Mac Mini Pros with 64GB RAM each
This is cool! DEEPSEEK-V3 ON M4 MAC: BLAZING FAST INFERENCE ON APPLE SILICON We just witnessed something incredible: the largest open-s...
#ai#macs/deepseek
0 2930 1
New
Back to top

General Dev>In The News

General Dev In The News ❯

Latest on Devtalk

Devtalk ❯

We ❤️ helpful members!

We reward our most helpful members via our MOTM scheme - by giving away a whopping 25 books per year!

Member of the Month

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Popular Portals

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that'll move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement See our Roadmap