CommunityNews

CommunityNews

Thank you, AI - End of an era for me. No more self-hosted git

Ok, it is over. End of an era for me. No more self-hosted git. I had a public git server running since 2011, and a public cvs server before that. AI scrapers have hammered the poor, little server to death by flooding the cgit frontend with tons of pointless² requests. Actually a few months ago already. Now I finally decided to not try rebuild the server, be it with or without cgit web frontend. I don’t feel like taking up the fight with the scrapers in my spare time, I leave that to people who are in a better position to do so. Most repositories had mirrors on one or two of the large gitforges already. Those are the primary repositories now. Go look at gitlab and github. Last week I’ve fixed all (I hope) dangeling links to the cgit repsitories to point to the forges instead. Now I’m down to one self-hosted service, which is the webserver hosting mainly this blog and a few more little things. In 2018 I’ve migrated the blog from wordpress to jekyll, so it is all static pages. Taking this out by AI scrapers overloading the machine should be next to impossible, and so far this has hold up. Nevertheless AI scrapers already managed to trigger one outage. Apparently millions of 404 answers where not enough to convince the bots that there is no cgit service (any more). Apache had no problems to deliver those, but the logs have filled up the disk so fast that logrotate didn’t manage to keep things under control with the default configuration. Fixed config. Knook wood. ¹ Title inspired by the 2025 edition of Security Nightmares. Fun watching if you speak german. ² Most inefficient way to get the complete repo. Just clone it, ok?

Read in full here:

Most Liked

alvinkatojr

alvinkatojr

Hmm, I believe that there are multiple solutions to this mindless scraping by AI, including open source options like Anubis. The author should have done a little more research before calling it quits..

toon159

toon159

I heard Cloudflare could prevent this easily.

alvinkatojr

alvinkatojr

Yes. They launched a service that feeds gibberish to scrapers similar to Anubus. The author should have looked into this before quitting.

Where Next?

Popular Ai topics Top

First poster: bot
NVIDIA Doubles Down: Announces A100 80GB GPU, Supercharging World’s Most Powerful GPU for AI Supercomputing. SC20—NVIDIA today unveiled ...
New
CommunityNews
DeepMind’s New AI With a Memory Outperforms Algorithms 25 Times Its Size. DeepMind’s model, with just 7 billion parameters, outperformed...
New
First poster: bot
An ancient language has defied decryption for 100 years. Can AI crack the code?. Machine learning can translate between two known langua...
New
First poster: bot
AI Wrote and Performed a Jerry Seinfeld Routine!. I used GPT-3 to write a Jerry Seinfeld stand-up routine about cats - and then used Dee...
New
First poster: bot
AlphaTensor discovers better algorithms for matrix math, inspiring another improvement from afar.
New
First poster: bot
Chri Besenbruch, CEO of Deep Render, sees many problems with the way video compression standards are developed today. He thinks they aren...
New
New
First poster: chris.johan
Stop vibe-coding blindly! Why reading AI-generated code is crucial in 2025. Avoid security flaws, architectural decay, and knowledge loss...
New
CommunityNews
Reproducibility is a bedrock of scientific progress. However, it’s remarkably difficult to get reproducible results out of large language...
New
First poster: jkdiaz
TechCrunch spoke to experienced coders about their time using AI-generated code about what they see as the future of vibe coding.
New

Other popular topics Top

ohm
Which, if any, games do you play? On what platform? I just bought (and completed) Minecraft Dungeons for my Nintendo Switch. Other than ...
New
brentjanderson
Bought the Moonlander mechanical keyboard. Cherry Brown MX switches. Arms and wrists have been hurting enough that it’s time I did someth...
New
Rainer
My first contact with Erlang was about 2 years ago when I used RabbitMQ, which is written in Erlang, for my job. This made me curious and...
New
AstonJ
I ended up cancelling my Moonlander order as I think it’s just going to be a bit too bulky for me. I think the Planck and the Preonic (o...
New
AstonJ
Do the test and post your score :nerd_face: :keyboard: If possible, please add info such as the keyboard you’re using, the layout (Qw...
New
Exadra37
Oh just spent so much time on this to discover now that RancherOS is in end of life but Rancher is refusing to mark the Github repo as su...
New
New
PragmaticBookshelf
Rails 7 completely redefines what it means to produce fantastic user experiences and provides a way to achieve all the benefits of single...
New
PragmaticBookshelf
Author Spotlight Jamis Buck @jamis This month, we have the pleasure of spotlighting author Jamis Buck, who has written Mazes for Prog...
New
PragmaticBookshelf
Author Spotlight: Peter Ullrich @PJUllrich Data is at the core of every business, but it is useless if nobody can access and analyze ...
New