General Dev>In The News

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

CommunityNews

CommunityNews

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

Using NumPy to replace Pandas GroupBy-Apply pattern for performance.
If you use PySpark a lot you would know that the DataFrame API is great. However there are times when it is not sufficient because it does not cover every single piece of functionality we may want. This is where the Pandas UDF functionality comes in. The nice thing about the Pandas UDF functionality is that it uses Arrow for data transfer between Spark and Pandas which minimizes serialization-deserialization costs. I have a slight preference for Pandas Function API over Pandas UDF but now let’s get to the meat of the post which is about speeding up the Pandas GroupBy-Apply pattern by using NumPy instead.

Read in full here:

This thread was posted by one of our members via one of our news source trackers.

View thread on forum
#performance#pandas
0 640 0

Where Next?

View thread on forum
performance
pandas
Home General Dev>In The News
#performance#pandas
0 640 0
Last post

Popular General Dev topics Top

First poster: bot
General Dev>In The News

Are top websites using WebGL for fingerprinting?
Site Fingerprinting google.com Yes youtube.com Yes Amazon.com Yes Yahoo.com Yes Zoom.us No Facebook.com Yes Reddit.com Ye...
jonatron.github.io
#webgl
0 1743 0
New
First poster: bot
General Dev>In The News

Logitech Pop Keys review: Reliable wireless mechanical keyboard with a divisive style
Mediocre typing feel overshadows reliable wireless and fantastic battery life.
arstechnica.com
/keyboards#review
0 1557 0
New
First poster: bot
General Dev>In The News

Kyria Build, Part 1: A wireless ergonomic keyboard
It has some interesting features: It’s entirely wireless (the left half speaks Bluetooth to the right half, and the right half speaks B...
ianthehenry.com
#ergonomic#keyboard
0 1719 0
New
CommunityNews
General Dev>In The News

LiveKit – open-source, high performance WebRTC infrastructure
GitHub - livekit/livekit: Scalable, high-performance WebRTC SFU. SDKs in JavaScript, React, React Native, Flutter, Swift, Kotlin, Unity/C...
github.com
#infrastructure#performance#webrtc
1 1773 1
New
First poster: bot
General Dev>In The News

Fish shell to be rewritten in Rust
Rewrite it in Rust by ridiculousfish · Pull Request #9512 · fish-shell/fish-shell. (Sorry for the meme; also this is obligatory.) I thi...
github.com
/rust#shell#fish
0 1424 0
New
First poster: bot
General Dev>In The News

When Zig is safer and faster than Rust
When Zig is safer and faster than Rust. There are endless debates online about Rust vs. Zig, this post explores a side of the argument I...
zackoverflow.dev
/rust/zig
0 1459 0
New
First poster: KnowledgeIsPower
General Dev>In The News

Building a Slack alternative with Rust/Tauri
Building a Slack/Discord alternative with Tauri/Rust linen <span class="hashtag-icon-placeholder"></span>blog. Introduction My name is K...
linen.dev
/rust#slack#tauri
1 1425 1
New
First poster: fullstackplus
General Dev>In The News

Why Python is terrible
Why Python is terrible… Nice language, but unsuitable for most professional purposes
josvisser.substack.com
/python
8 1314 6
New
CommunityNews
General Dev>In The News

SLUM: The Shadow Library Uptime Monitor
SLUM: The Shadow Library Uptime Monitor. This dashboard tracks the availability of popular shadow libraries in real time from a US-based...
open-slum.org
#library#monitor
0 2891 0
New
First poster: andrea
General Dev>In The News

Everything I know about good API design
Most of what modern software engineers do involves APIs: public interfaces for communicating with a program, like this one from Twilio. I...
seangoedecke.com
#api#design
1 1534 1
New

Other popular topics Top

Devtalk
General Dev>Dev Chat

HELLO WORLD (Introductions thread!)
Hello Devtalk World! Please let us know a little about who you are and where you’re from :nerd_face:
#community
481 6762 116
New
AstonJ
General Dev>Hardware

Which keyboard do you have?
If it’s a mechanical keyboard, which switches do you have? Would you recommend it? Why? What will your next keyboard be? Pics always w...
#hardware/keyboards#sticky#mechanical-keyboards
144 9115 50
New
Rainer
Community>Journals

Programming Erlang Book Club
My first contact with Erlang was about 2 years ago when I used RabbitMQ, which is written in Erlang, for my job. This made me curious and...
/erlang/book-programming-erlang-2nd-edition#book-club
195 6815 95
New
AstonJ
General Dev>Hardware

Planck vs Preonic vs Subatomic (Keyboards)
I ended up cancelling my Moonlander order as I think it’s just going to be a bit too bulky for me. I think the Planck and the Preonic (o...
/keyboards#mechanical-keyboards#ortholinear#planck#preonic
105 17596 47
New
AstonJ
General Dev>Hardware

BIIP MT3 Extended 2048 Custom Keycap Set (Drop)
This looks like a stunning keycap set :orange_heart: A LEGENDARY KEYBOARD LIVES ON When you bought an Apple Macintosh computer in the e...
/keyboards#apple#keycaps#mechanical-keyboards
14 6713 7
New
PragmaticBookshelf
Backend>Learning Resources

Programming WebRTC
Use WebRTC to build web applications that stream media and data in real time directly from one user to another, all in the browser. ...
pragprog.com
#pragprog#published-book/js#webrtc/book-programming-webrtc
27 6969 6
New
AstonJ
macOS>Chat

How to block any website on Mac using Little Snitch
If you want a quick and easy way to block any website on your Mac using Little Snitch simply… File &gt; New Rule: And select Deny, O...
#macos#how-to#littlesnitch
5 11227 3
New
PragmaticBookshelf
Backend>Learning Resources

Ash Framework
Explore the power of Ash Framework by modeling and building the domain for a real-world web application. Rebecca Le @sevenseacat and ...
pragprog.com
#pragprog/elixir#published-book/ash/book-ash-framework
15 7555 9
New
AstonJ
AI>Chat

How to: Run DeepSeek on Mac, Windows, and Linux!
This is a very quick guide, you just need to: Download LM Studio: https://lmstudio.ai/ Click on search Type DeepSeek, then select the o...
#macs/deepseek#guides#lm-studio
14 9328 10
New
PragmaticBookshelf
Backend>Learning Resources

Advanced Functional Programming with Elixir
Use advanced functional programming principles, practical Domain-Driven Design techniques, and production-ready Elixir code to build scal...
joekoski.com
#pragprog/elixir#published-book#functional-programming/book-advanced-functional-programming-with-elixir
43 4989 22
New
Back to top

Sponsor Spotlight

Manning

Develop your skills with books, videos, and courses.

General Dev>In The News

General Dev In The News ❯

Latest on Devtalk

Devtalk ❯

We ❤️ helpful members!

We reward our most helpful members via our MOTM scheme - by giving away a whopping 25 books per year!

Member of the Month

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

EEF

Supporting innovation across the BEAM ecosystem.

coders51

We build reliable cloud platforms for business-critical systems.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

Categories:

Sub Categories:

Popular Portals

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

EEF

Supporting innovation across the BEAM ecosystem.

coders51

We build reliable cloud platforms for business-critical systems.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that move you from confusion to "Aha, now I get it!"

View all our sponsors ❯

We're in Beta

About us Mission Statement See our Roadmap