CommunityNews

CommunityNews

Thank you, AI - End of an era for me. No more self-hosted git

Ok, it is over. End of an era for me. No more self-hosted git. I had a public git server running since 2011, and a public cvs server before that. AI scrapers have hammered the poor, little server to death by flooding the cgit frontend with tons of pointless² requests. Actually a few months ago already. Now I finally decided to not try rebuild the server, be it with or without cgit web frontend. I don’t feel like taking up the fight with the scrapers in my spare time, I leave that to people who are in a better position to do so. Most repositories had mirrors on one or two of the large gitforges already. Those are the primary repositories now. Go look at gitlab and github. Last week I’ve fixed all (I hope) dangeling links to the cgit repsitories to point to the forges instead. Now I’m down to one self-hosted service, which is the webserver hosting mainly this blog and a few more little things. In 2018 I’ve migrated the blog from wordpress to jekyll, so it is all static pages. Taking this out by AI scrapers overloading the machine should be next to impossible, and so far this has hold up. Nevertheless AI scrapers already managed to trigger one outage. Apparently millions of 404 answers where not enough to convince the bots that there is no cgit service (any more). Apache had no problems to deliver those, but the logs have filled up the disk so fast that logrotate didn’t manage to keep things under control with the default configuration. Fixed config. Knook wood. ¹ Title inspired by the 2025 edition of Security Nightmares. Fun watching if you speak german. ² Most inefficient way to get the complete repo. Just clone it, ok?

Read in full here:

Most Liked

alvinkatojr

alvinkatojr

Hmm, I believe that there are multiple solutions to this mindless scraping by AI, including open source options like Anubis. The author should have done a little more research before calling it quits..

toon159

toon159

I heard Cloudflare could prevent this easily.

alvinkatojr

alvinkatojr

Yes. They launched a service that feeds gibberish to scrapers similar to Anubus. The author should have looked into this before quitting.

Where Next?

Popular Ai topics Top

First poster: CommunityNews
AI models are increasingly applied in high-stakes domains like health and conservation. Data quality carries an elevated signifi- cance i...
New
CommunityNews
We present Imagen, a text-to-image diffusion model with an unprecedented degree of photorealism and a deep level of language understandin...
New
First poster: bot
AI video editor can recognize objects, people, and sounds, allowing editing via text.
New
First poster: bot
Ghostwriter - Code faster with AI. An AI pair programmer that helps you write better code, faster.
New
First poster: bot
Exascale Cerebras Andromeda cluster packs more cores than 1,954 Nvidia A100 GPUs.
New
First poster: alvinkatojr
Klarna CEO says the company stopped hiring a year ago because AI ‘can already do all of the jobs’. Klarna CEO Sebastian Siemiatkowski sa...
New
First poster: gflashner
Google’s openly available Gemma collection of AI models has reached a milestone: over 150 million downloads. Omar Sanseviero, a developer...
New
First poster: mercyf
It replicates your development environment and takes up to 30 minutes per task.
New
First poster: AstonJ
From fear to optimism: why I am convinced AI is worth embracing.
New
CommunityNews
Claude Code is an agentic coding tool that lives in your terminal, understands your codebase, and helps you code faster by executing rout...
New

Other popular topics Top

PragmaticBookshelf
Free and open source software is the default choice for the technologies that run our world, and it’s built and maintained by people like...
New
PragmaticBookshelf
Learn from the award-winning programming series that inspired the Elixir language, and go on a step-by-step journey through the most impo...
New
dasdom
No chair. I have a standing desk. This post was split into a dedicated thread from our thread about chairs :slight_smile:
New
AstonJ
I have seen the keycaps I want - they are due for a group-buy this week but won’t be delivered until October next year!!! :rofl: The Ser...
New
PragmaticBookshelf
Create efficient, elegant software tests in pytest, Python's most powerful testing framework. Brian Okken @brianokken Edited by Kat...
New
AstonJ
Saw this on TikTok of all places! :lol: Anyone heard of them before? Lite:
New
AstonJ
This is cool! DEEPSEEK-V3 ON M4 MAC: BLAZING FAST INFERENCE ON APPLE SILICON We just witnessed something incredible: the largest open-s...
New
AstonJ
This is a very quick guide, you just need to: Download LM Studio: https://lmstudio.ai/ Click on search Type DeepSeek, then select the o...
New
NewsBot
Node.js v22.14.0 has been released. Link: Release 2025-02-11, Version 22.14.0 'Jod' (LTS), @aduh95 · nodejs/node · GitHub
New
PragmaticBookshelf
Fight complexity and reclaim the original spirit of agility by learning to simplify how you develop software. The result: a more humane a...
New