Redirect bots from non-existing URLs

[eawmaas]

I am seeing a high amount of traffic (100´s a day) to non-existing Chyrp-URLs and my guess it´s because of the routing because it only happens in Chyrp installs and not on other pages without routing.

The bots combine folders and files outside Chyrp with URLs within the Chyrp-URLs and the server responds with a status 200, so not even a 404 error.

Besides the fact that this messes up the server-logs which is a nusance when analysing them I also suspect that this might cause high non-human traffic and a higher server-load to Chyrp. These bots keep coming back looking for content or whatever because (I guess) their request does not return the expected result.

An example of some requests (the folder readnwrite is the Chyrp install-folder):

1	https://exdomo.com/readnwrite/author/1/myownlexen/myownpages/myownpages/stamboom/addgeneanet.php
1	https://exdomo.com/readnwrite/author/1/social/beirasportugal/myownpages/stamboom/addandriesmaas1845.php
1	https://exdomo.com/readnwrite/author/1/myownpages/myownlexen/beirasportugal/stamboom/addandriesmaas1845.php
1	https://exdomo.com/readnwrite/index/myownlexen/beirasportugal/living.php/stamboom/adddiazbaumgardt.php
1	https://exdomo.com/readnwrite/index/myownlexen/se.php/myownpages/social/social
1	https://exdomo.com/readnwrite/index/social/myownpages/myowncontactnl.php/stamboom/adddiazbaumgardt.php
1	https://exdomo.com/readnwrite/category/exdomo-van-alles-allsorts/

I would like to redirect these requests to one specific page, but how can I make the server recognise this in the .htaccess-file?
What rule shoud I write?

Or can this be solved from within Chyrp itself?

[xenocrat]

Hello there,

This looks like fairly typical automated penetration traffic. Various soundrels will fire requests at your blog hoping that it's running some well-known platform and that a common vulnerability can be exploited remotely. The best thing to do about this traffic is nothing - just ignore it. If you see a lot of requests from one particular IP range or user-agent, you could try to ban it - but that game of whack-a-mole soon gets boring! Chyrp Lite will lessen the load of bot traffic by denying sessions to self-identified bots (see here and here), but that only works for bots that declare themselves.

Chyrp Lite should be responding with 404s for any unroutable request; you can check this be opening the network activity tab of developer tools and visiting an unroutable URL. There will be a pretty error page but the status code returned by Chyrp Lite is 404/403.

[eawmaas]

Ah I see, that's why they don´t end up on my own 403 and 404 page (which I monitor daily).

I will have a think about this, you're absolutely right about the boring part, I made a little script that writes a list of (mostly) scum-bots which I can copy/paste in the .htaccess once every whatever days.

I am thinking of using an API from abusedipdb dot com to add the "confidence of abuse"-score that they generate or maybe block all non-identifying bots and whitelist the ones that respect robots.txt , maybe add a honeypot...

I want real traffic and there are hardly any visitors coming from Google or Bing or so anyway.

Thanks for your info, I know now where to find solutions, I just can't ignore them, I hate this side of the web and with a strong CSP and the firewall from perishable press most of this traffic doesn't even enter the game anymore so I guess these bots are about the last ones to (try to) block...