General Dev>In The News

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

CommunityNews

CommunityNews

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

Using NumPy to replace Pandas GroupBy-Apply pattern for performance.
If you use PySpark a lot you would know that the DataFrame API is great. However there are times when it is not sufficient because it does not cover every single piece of functionality we may want. This is where the Pandas UDF functionality comes in. The nice thing about the Pandas UDF functionality is that it uses Arrow for data transfer between Spark and Pandas which minimizes serialization-deserialization costs. I have a slight preference for Pandas Function API over Pandas UDF but now let’s get to the meat of the post which is about speeding up the Pandas GroupBy-Apply pattern by using NumPy instead.

Read in full here:

This thread was posted by one of our members via one of our news source trackers.

View thread on forum
#performance#pandas
0 640 0

Where Next?

View thread on forum
performance
pandas
Home General Dev>In The News
#performance#pandas
0 640 0
Last post

Popular General Dev topics Top

First poster: malloryerik
General Dev>In The News

Doom-emacs: An Emacs framework
GitHub - hlissner/doom-emacs: An Emacs framework for the stubborn martian hacker. An Emacs framework for the stubborn martian hacker - G...
github.com
/emacs#doom-emacs
55 3346 16
New
First poster: mindriot
General Dev>In The News

LG 28-inch 16:18 DualUp Monitor
LG 28-inch 16:18 DualUp Monitor with Ergo Stand and USB Type-C™ (28MQ780-B) | LG USA. Shop LG 28MQ780-B on the official LG.com website ...
lg.com
12 2239 12
New
First poster: bot
General Dev>In The News

Writing a Python SQL engine from scratch
sqlglot/python_sql_engine.md at main · tobymao/sqlglot. Python SQL Parser and Transpiler. Contribute to tobymao/sqlglot development by c...
github.com
/python#sql#writing
0 1647 0
New
First poster: bot
General Dev>In The News

Jailbreak Chat: A collection of ChatGPT jailbreaks
Jailbreak Chat. Collection of ChatGPT jailbreak prompts
jailbreakchat.com
#chat#chatgpt
0 1741 0
New
First poster: dyowee
General Dev>In The News

DreamBerd is a perfect programming language
GitHub - TodePond/DreamBerd: perfect programming language. perfect programming language. Contribute to TodePond/DreamBerd development by...
github.com
#programming
3 1734 2
New
CommunityNews
General Dev>In The News

Testing Intel’s Arc A770 GPU for Deep Learning
Christian Mills - Testing Intel’s Arc A770 GPU for Deep Learning Pt. 2. This post covers my experience training image classification mod...
christianjmills.com
#testing#learning#intel
0 1940 0
New
CommunityNews
General Dev>In The News

Perplexica: Open-Source Perplexity Alternative
GitHub - ItzCrazyKns/Perplexica: Perplexica is an AI-powered search engine. It is an Open source alternative to Perplexity AI. Perplexic...
github.com
0 1794 0
New
CommunityNews
General Dev>In The News

Deepseek - starting this week we'll open-source 5 repos
We’re a tiny team @deepseek-ai pushing our limits in AGI exploration. Starting this week , Feb 24, 2025 we’ll open-source 5 repos – one ...
github.com
#github/deepseek
1 1227 0
New
First poster: alvinkatojr
General Dev>In The News

Distributed Systems Programming Has Stalled
Over the last decade, we’ve seen great advancements in distributed systems, but the way we program them has seen few fundamental improvem...
shadaj.me
#programming
4 981 2
New
CommunityNews
General Dev>In The News

Self-Hosting a Firefox Sync Server
After switching from Firefox to LibreWolf, I became interested in the idea of self-hosting my own Firefox Sync server. Although I had see...
blog.diego.dev
#hosting#firefox
0 1154 0
New

Other popular topics Top

PragmaticBookshelf
Backend>Learning Resources

Programming Machine Learning
Machine learning can be intimidating, with its reliance on math and algorithms that most programmers don't encounter in their regular wor...
pragprog.com
#pragprog#ai/python#published-book/book-programming-machine-learning#math#algorithms
6 5350 3
New
dasdom
General Dev>Dev Chat

Standing Desks
No chair. I have a standing desk. This post was split into a dedicated thread from our thread about chairs :slight_smile:
#workspace#opinions
177 9886 77
New
AstonJ
General Dev>Hardware

Custom keyboard keycaps
There’s a whole world of custom keycaps out there that I didn’t know existed! Check out all of our Keycaps threads here: https://forum....
#hardware/keyboards#keycaps#mechanical-keyboards
15 11086 19
New
AstonJ
General Dev>Hardware

Keyboard thock (sound)
I’ve been hearing quite a lot of comments relating to the sound of a keyboard, with one of the most desirable of these called ‘thock’, he...
/keyboards#mechanical-keyboards
14 11197 8
New
Margaret
General Dev>Dev Chat

PragProg’s Medium Posts
Hello everyone! This thread is to tell you about what authors from The Pragmatic Bookshelf are writing on Medium.
#pragprog#blog-post
1147 29994 760
New
Help
Game Dev>Questions

Can I use Java to program a game for Nintendo switch?
I am trying to crate a game for the Nintendo switch, I wanted to use Java as I am comfortable with that programming language. Can you use...
/java#nintendo
8 4771 3
New
PragmaticBookshelf
Community>In The Spotlight

Spotlight: Erin Dees (Author) Interview and AMA!
Author Spotlight Erin Dees @undees Welcome to our new author spotlight! We had the pleasure of chatting with Erin Dees, co-author of ...
#author-spotlight/ruby/rails/book-effective-testing-with-rspec-3/book-seven-more-languages-in-seven-weeks/book-cucumber-recipes#rspec
24 4247 11
New
PragmaticBookshelf
Community>In The Spotlight

Spotlight: VM Brasseur (Author) Interview and AMA!
Author Spotlight: VM Brasseur @vmbrasseur We have a treat for you today! We turn the spotlight onto Open Source as we sit down with V...
#author-spotlight/book-forge-your-future-with-open-source
16 5051 11
New
AnfaengerAlex
Android>Questions

Unresolved Reference to android in build.gradle.kts – Beginner Issue
Hello, I’m a beginner in Android development and I’m facing an issue with my project setup. In my build.gradle.kts file, I have the foll...
#binding
0 7460 2
New
PragmaticBookshelf
Backend>Learning Resources

Advanced Functional Programming with Elixir
Use advanced functional programming principles, practical Domain-Driven Design techniques, and production-ready Elixir code to build scal...
joekoski.com
#pragprog/elixir#published-book#functional-programming/book-advanced-functional-programming-with-elixir
43 4989 22
New
Back to top

Sponsor Spotlight

coders51

We build reliable cloud platforms for business-critical systems.

General Dev>In The News

General Dev In The News ❯

Latest on Devtalk

Devtalk ❯

We ❤️ helpful members!

We reward our most helpful members via our MOTM scheme - by giving away a whopping 25 books per year!

Member of the Month

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

EEF

Supporting innovation across the BEAM ecosystem.

coders51

We build reliable cloud platforms for business-critical systems.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that move you from confusion to "Aha, now I get it!"

Linux New Media

Publishing Linux Magazine, ADMIN, and MakerSpace for 25+ years.

View all our sponsors ❯

Categories:

Sub Categories:

Popular Portals

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

EEF

Supporting innovation across the BEAM ecosystem.

coders51

We build reliable cloud platforms for business-critical systems.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that move you from confusion to "Aha, now I get it!"

Linux New Media

Publishing Linux Magazine, ADMIN, and MakerSpace for 25+ years.

View all our sponsors ❯

We're in Beta

About us Mission Statement See our Roadmap