General Dev>In The News

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

CommunityNews

CommunityNews

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

Using NumPy to replace Pandas GroupBy-Apply pattern for performance.
If you use PySpark a lot you would know that the DataFrame API is great. However there are times when it is not sufficient because it does not cover every single piece of functionality we may want. This is where the Pandas UDF functionality comes in. The nice thing about the Pandas UDF functionality is that it uses Arrow for data transfer between Spark and Pandas which minimizes serialization-deserialization costs. I have a slight preference for Pandas Function API over Pandas UDF but now let’s get to the meat of the post which is about speeding up the Pandas GroupBy-Apply pattern by using NumPy instead.

Read in full here:

This thread was posted by one of our members via one of our news source trackers.

View thread on forum
#performance#pandas
0 640 0

Where Next?

View thread on forum
performance
pandas
Home General Dev>In The News
#performance#pandas
0 640 0
Last post

Popular General Dev topics Top

First poster: AstonJ
General Dev>In The News

Permission.site (online tool to check browser permissions)
https://permission.site/ This thread was posted by one of our members via one of our news source trackers.
permission.site
22 1763 8
New
First poster: AstonJ
General Dev>In The News

Wearable Microphone Jamming
We engineered a wearable microphone jammer that is capable of disabling microphones in its user’s surroundings, including hidden micropho...
sandlab.cs.uchicago.edu
/security
7 1834 3
New
First poster: malloryerik
General Dev>In The News

Doom-emacs: An Emacs framework
GitHub - hlissner/doom-emacs: An Emacs framework for the stubborn martian hacker. An Emacs framework for the stubborn martian hacker - G...
github.com
/emacs#doom-emacs
55 3346 16
New
First poster: bot
General Dev>In The News

Introducing the ChatGPT/LLM error tracker
Large Language Models like ChatGPT say The Darnedest Things. The Errors They MakeWhy We Need to Document Them, and What We Have Decided ...
garymarcus.substack.com
#error#chatgpt
0 3521 0
New
First poster: bot
General Dev>In The News

Writing a Python SQL engine from scratch
sqlglot/python_sql_engine.md at main · tobymao/sqlglot. Python SQL Parser and Transpiler. Contribute to tobymao/sqlglot development by c...
github.com
/python#sql#writing
0 1647 0
New
First poster: bot
General Dev>In The News

A reason why Mac speakers sound better and louder than most
Hector Martin (@marcan@treehouse.systems). Attached: 1 image For those wondering why the hell we need all this safety system stuff for...
social.treehouse.systems
0 1576 0
New
First poster: AstonJ
General Dev>In The News

Stats: macOS system monitor in your menu bar
GitHub - exelban/stats: macOS system monitor in your menu bar. macOS system monitor in your menu bar. Contribute to exelban/stats develo...
github.com
#macos#github#monitor
1 537 1
New
CommunityNews
General Dev>In The News

Deepseek - starting this week we'll open-source 5 repos
We’re a tiny team @deepseek-ai pushing our limits in AGI exploration. Starting this week , Feb 24, 2025 we’ll open-source 5 repos – one ...
github.com
#github/deepseek
1 1227 0
New
CommunityNews
General Dev>In The News

GitSyncPad - Effortless Git Version Control
GitSyncPad is an innovative micro keypad designed for effortless Git version control. Execute commands like git add, git commit, and git ...
gitsyncpad.xyz
#git
0 616 0
New
CommunityNews
General Dev>In The News

TrueSkill 2: An improved Bayesian skill rating system - Microsoft Research
Online multiplayer games, such as Gears of War and Halo, use skill-based matchmaking to give players fair and enjoyable matches. They dep...
microsoft.com
#microsoft
0 1031 0
New

Other popular topics Top

AstonJ
General Dev>Hardware

Which keyboard do you have?
If it’s a mechanical keyboard, which switches do you have? Would you recommend it? Why? What will your next keyboard be? Pics always w...
#hardware/keyboards#sticky#mechanical-keyboards
144 9115 50
New
AstonJ
General Dev>Hardware

Poll: Which keyboard layout do you use?
poll poll Be sure to check out @Dusty’s article posted here: An Introduction to Alternative Keyboard Layouts It’s one of the best write-...
colemakmods.github.io
#polls/keyboards
10 6048 11
New
AstonJ
General Dev>Hardware

Custom keyboard keycaps
There’s a whole world of custom keycaps out there that I didn’t know existed! Check out all of our Keycaps threads here: https://forum....
#hardware/keyboards#keycaps#mechanical-keyboards
15 11086 19
New
dimitarvp
macOS>Chat

My thoughts on macOS vs Linux
Small essay with thoughts on macOS vs. Linux: I know @Exadra37 is just waiting around the corner to scream at me “I TOLD YOU SO!!!” but I...
#macos#linux
166 9730 69
New
AstonJ
General Dev>Dev Chat

Languages Without Garbage Collection
Continuing the discussion from Thinking about learning Crystal, let’s discuss - I was wondering which languages don’t GC - maybe we can c...
#garbage-collection
21 5575 7
New
PragmaticBookshelf
Backend>Learning Resources

Python Testing with pytest, Second Edition
Create efficient, elegant software tests in pytest, Python's most powerful testing framework. Brian Okken @brianokken Edited by Kat...
pragprog.com
#pragprog/python#published-book/book-python-testing-with-pytest-second-edition
16 7461 4
New
PragmaticBookshelf
Backend>Learning Resources

Agile Web Development with Rails 7
Rails 7 completely redefines what it means to produce fantastic user experiences and provides a way to achieve all the benefits of single...
pragprog.com
#pragprog#web-development/ruby/rails#published-book/book-agile-web-development-with-rails-7
32 6600 9
New
PragmaticBookshelf
Community>In The Spotlight

Spotlight: Rebecca Skinner (Author) Interview and AMA!
Author Spotlight Rebecca Skinner @RebeccaSkinner Welcome to our latest author spotlight, where we sit down with Rebecca Skinner, auth...
#author-spotlight/haskell/book-effective-haskell
106 11719 28
New
First poster: CommunityNews
AI>In The News

How to fix the eyes in AI-generated images
aidemos.info
0 4508 0
New
PragmaticBookshelf
Backend>Learning Resources

Programming Ruby 3.2 (5th Edition)
Programming Ruby is the most complete book on Ruby, covering both the language itself and the standard library as well as commonly used t...
twitter.com
#pragprog/ruby/rails#published-book/book-programming-ruby-3-2-5th-edition
28 8550 13
New
Back to top

Sponsor Spotlight

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

General Dev>In The News

General Dev In The News ❯

Latest on Devtalk

Devtalk ❯

We ❤️ helpful members!

We reward our most helpful members via our MOTM scheme - by giving away a whopping 25 books per year!

Member of the Month

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

EEF

Supporting innovation across the BEAM ecosystem.

coders51

We build reliable cloud platforms for business-critical systems.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that move you from confusion to "Aha, now I get it!"

Linux New Media

Publishing Linux Magazine, ADMIN, and MakerSpace for 25+ years.

View all our sponsors ❯

Categories:

Sub Categories:

Popular Portals

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

EEF

Supporting innovation across the BEAM ecosystem.

coders51

We build reliable cloud platforms for business-critical systems.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that move you from confusion to "Aha, now I get it!"

Linux New Media

Publishing Linux Magazine, ADMIN, and MakerSpace for 25+ years.

View all our sponsors ❯

We're in Beta

About us Mission Statement See our Roadmap