General Dev>In The News

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

CommunityNews

CommunityNews

Using NumPy to replace Pandas GroupBy-Apply pattern for performance

Using NumPy to replace Pandas GroupBy-Apply pattern for performance.
If you use PySpark a lot you would know that the DataFrame API is great. However there are times when it is not sufficient because it does not cover every single piece of functionality we may want. This is where the Pandas UDF functionality comes in. The nice thing about the Pandas UDF functionality is that it uses Arrow for data transfer between Spark and Pandas which minimizes serialization-deserialization costs. I have a slight preference for Pandas Function API over Pandas UDF but now let’s get to the meat of the post which is about speeding up the Pandas GroupBy-Apply pattern by using NumPy instead.

Read in full here:

This thread was posted by one of our members via one of our news source trackers.

View thread on forum
#performance#pandas
0 640 0

Where Next?

View thread on forum
performance
pandas
Home General Dev>In The News
#performance#pandas
0 640 0
Last post

Popular General Dev topics Top

First poster: bot
General Dev>In The News

A reason why Mac speakers sound better and louder than most
Hector Martin (@marcan@treehouse.systems). Attached: 1 image For those wondering why the hell we need all this safety system stuff for...
social.treehouse.systems
0 1576 0
New
First poster: bot
General Dev>In The News

When Zig is safer and faster than Rust
When Zig is safer and faster than Rust. There are endless debates online about Rust vs. Zig, this post explores a side of the argument I...
zackoverflow.dev
/rust/zig
0 1459 0
New
First poster: bot
General Dev>In The News

ONNX Runtime merges WebGPU backend
[js/web] WebGPU backend via JSEP by fs-eire · Pull Request #14579 · microsoft/onnxruntime. Description This change introduced the follo...
github.com
#backend
0 1372 0
New
CommunityNews
General Dev>In The News

The Definitive PHP 7.2, 7.3, 7.4, 8.0, and 8.1 Benchmarks (2023)
The Definitive PHP 7.2, 7.3, 7.4, 8.0, and 8.1 Benchmarks (2023). We tested the performance of 14 PHP platforms (WordPress, Drupal, Lara...
kinsta.com
/php
0 1888 0
New
CommunityNews
General Dev>In The News

X can’t stop spread of explicit, fake AI Taylor Swift images
Will Swifties’ war on AI fakes spark a deepfake porn reckoning?
arstechnica.com
/swift
0 8379 0
New
CommunityNews
General Dev>In The News

Perplexica: Open-Source Perplexity Alternative
GitHub - ItzCrazyKns/Perplexica: Perplexica is an AI-powered search engine. It is an Open source alternative to Perplexity AI. Perplexic...
github.com
0 1794 0
New
CommunityNews
General Dev>In The News

Tantivy – full-text search engine library inspired by Apache Lucene
GitHub - quickwit-oss/tantivy: Tantivy is a full-text search engine library inspired by Apache Lucene and written in Rust. Tantivy is a ...
github.com
#lucene
0 451 0
New
First poster: DevotionGeo
General Dev>In The News

To avoid being replaced by LLMs, do what they can't
To avoid being replaced by LLMs, do what they can’t. What LLM’s can’t do yet
seangoedecke.com
18 1109 7
New
First poster: joeb
General Dev>In The News

JavaScript Fatigue Strikes Back
The new frameworks will continue until morale improves.
allenpike.com
/js
6 869 5
New
CommunityNews
General Dev>In The News

Should managers still code?
Ah, the eternal question, straight from the mailbag.
theengineeringmanager.substack.com
#code
0 893 0
New

Other popular topics Top

PragmaticBookshelf
General Dev>Learning Resources

A Common-Sense Guide to Data Structures and Algorithms, Second Edition
Algorithms and data structures are much more than abstract concepts. Mastering them enables you to write code that runs faster and more e...
pragprog.com
#pragprog/python/ruby#published-book/book-a-common-sense-guide-to-data-structures-and-algorithms-second-edition#math#algorithms/js
19 6022 5
New
PragmaticBookshelf
General Dev>Learning Resources

The Pragmatic Programmer, 20th Anniversary Edition
Andy and Dave wrote this influential, classic book to help their clients create better software and rediscover the joy of coding. Almost ...
pragprog.com
#pragprog#published-book/book-the-pragmatic-programmer-20th-anniversary-edition
4 4782 0
New
PragmaticBookshelf
Backend>Learning Resources

Seven Languages in Seven Weeks
Ruby, Io, Prolog, Scala, Erlang, Clojure, Haskell. With Seven Languages in Seven Weeks, by Bruce A. Tate, you’ll go beyond the syntax—and...
pragprog.com
#pragprog/clojure/erlang/haskell/prolog/ruby/scala#published-book/book-seven-languages-in-seven-weeks
5 5730 1
New
PragmaticBookshelf
Backend>Learning Resources

Testing Elixir
Write Elixir tests that you can be proud of. Dive into Elixir’s test philosophy and gain mastery over the terminology and concepts that u...
pragprog.com
#pragprog/elixir#published-book/book-testing-elixir
33 5004 8
New
Maartz
General Dev>Dev Chat

Roc Language - a new purely functional programming language built for speed and ergonomics
Hi folks, I don’t know if I saw this here but, here’s a new programming language, called Roc Reminds me a bit of Elm and thus Haskell. ...
#programminguages#functional-programming
49 5164 14
New
Help
Game Dev>Questions

Can I use Java to program a game for Nintendo switch?
I am trying to crate a game for the Nintendo switch, I wanted to use Java as I am comfortable with that programming language. Can you use...
/java#nintendo
8 4771 3
New
PragmaticBookshelf
Community>In The Spotlight

Spotlight: Mike Riley (Author) Interview and AMA!
Author Spotlight Mike Riley @mriley This month, we turn the spotlight on Mike Riley, author of Portable Python Projects. Mike’s book ...
#author-spotlight/python#iot/book-portable-python-projects#internet-of-things
62 7035 19
New
AnfaengerAlex
Android>Questions

Unresolved Reference to android in build.gradle.kts – Beginner Issue
Hello, I’m a beginner in Android development and I’m facing an issue with my project setup. In my build.gradle.kts file, I have the foll...
#binding
0 7460 2
New
PragmaticBookshelf
Backend>Learning Resources

Ash Framework
Explore the power of Ash Framework by modeling and building the domain for a real-world web application. Rebecca Le @sevenseacat and ...
pragprog.com
#pragprog/elixir#published-book/ash/book-ash-framework
15 7555 9
New
xiji2646-netizen
AI>Chat

Has anyone tried the Karpathy CLAUDE.md rules? (97.8k stars)
There’s a GitHub repo at forrestchang/andrej-karpathy-skills that’s sitting at 97.8k stars. It’s a single CLAUDE.md file with four behavi...
#claude
0 37 1
New
Back to top

Sponsor Spotlight

AppSignal

Catch errors, track performance, monitor hosts and more.

General Dev>In The News

General Dev In The News ❯

Latest on Devtalk

Devtalk ❯

We ❤️ helpful members!

We reward our most helpful members via our MOTM scheme - by giving away a whopping 25 books per year!

Member of the Month

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

EEF

Supporting innovation across the BEAM ecosystem.

coders51

We build reliable cloud platforms for business-critical systems.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that move you from confusion to "Aha, now I get it!"

Linux New Media

Publishing Linux Magazine, ADMIN, and MakerSpace for 25+ years.

View all our sponsors ❯

Categories:

Sub Categories:

Popular Portals

Devtalk Sponsors

Pragmatic Bookshelf

Practical resources that improve the lives of professional developers.

Erlang Solutions

We build trusted fault-tolerant systems that scale to billions of users.

Honeybadger

Real-time error tracking, performance insights, and observability for devs.

EEF

Supporting innovation across the BEAM ecosystem.

coders51

We build reliable cloud platforms for business-critical systems.

Manning Publications

Develop your skills with books, videos, and courses.

AppSignal

Catch errors, track performance, monitor hosts and more.

WyeWorks

Enabling companies to succeed by building software people love.

Pragmatic Studio

Courses that move you from confusion to "Aha, now I get it!"

Linux New Media

Publishing Linux Magazine, ADMIN, and MakerSpace for 25+ years.

View all our sponsors ❯

We're in Beta

About us Mission Statement See our Roadmap