The solution is allowing the convenient web interface only for trusted members, like e.g. 4 year old account can view all pages(with easy anubis setting) but guest/bots/everyone else has to git clone the thing and do it on their backend.
IIRC old forums also limited 'content for registered/trusted/moderators/etc' content views decades ago, turns out this crap saves gigatons of traffic.
That excludes new legitimate users, just put up a very cheap paywall for the web interface, 1 dollar for lifetime access, that way each request is tied to a user and can be rate limited. While remaining essentially free.
