I discovered yesterday evening that Lemmy.ml is blocking all inbound ActivityPub requests from /kbin instances. Specifically, a 403 ‘access denied’ is returned when the user agent contains “kbinBot” anywhere in the string. This has been causing a cascade of failures with federation for many server owners, flooding the message queue with transport errors.

This doesn’t appear to be a mistake; it has been done very deliberately, only on Lemmy.ml. Lemmy.world and other large instances do not exhibit the same behavior. It also isn’t a side effect of the bug introduced in Lemmy 0.18. You can observe by sending the following in a terminal

> curl -I --user-agent "kbinBot v0.1" https://lemmy.world/u/test
HTTP/2 200
[...]

> curl -I --user-agent "kbinBot v0.1" https://lemmy.ml/u/test                                
HTTP/2 403
[...]

> curl -I --user-agent "notKbinBot v0.1" https://lemmy.ml/u/test
HTTP/2 403
[...]

> curl -I --user-agent "placeholder-user-agent" https://lemmy.ml/u/test
HTTP/2 200
[...]

Additional evidence of this not being a Lemmy 0.18 bug:

  • This occurs when making web requests to any location on the Lemmy.ml webserver, not just ActivityPub endpoints.

  • Go to https://fedidb.org/software/lemmy and pick an instance running 0.18.0. Perform the above commands, replacing the URL for Lemmy.ml with that particular instance’s address.

If this continues, my instance may need to defederate from Lemmy.ml. This is especially problematic because Lemmy.ml continues to federate information outbound to other kbin instances while refusing to allow inbound communication from them.

Spoofing the user agent is less than ideal, and doesn’t respect Lemmy.ml’s potential wish to not be contacted by /kbin instances. I don’t post this to create division between communities, but I do hope that I can draw awareness to what’s going on here. Defederating /kbin instances entirely would even be better than arbitrarily denying access one-way. This said, we should all attempt to maintain a good-faith interpretation until otherwise indicated by the Lemmy developers. It’s possibel that this is a firewall misconfiguration or some other webserver-related bug.

Relevant comment from me (#354 - [BUG] Critical errors/failed messages during messenger:consume)

Edits:

  • Yes, people have already tried reaching out to the Lemmy instance admins in their Matrix room with no answer.

  • Someone has posed a question on Lemmy.ml about the block here: https://lemmy.ml/post/1563840

  • phi1997@kbin.social
    48·
    1 year ago

    If they don’t want us to communicate with the users of their server, they should be defederated. Sucks for the users, but they can hop to another server whose admins don’t break communication inelegantly and ghost admins who want to know what’s going on.

    If it’s just a bug, then obviously we should wait for it to be fixed, but if they’re not even saying they’ll look into it, I don’t have much hope.

      • Deceptichum@kbin.social
        27·
        1 year ago

        Before assuming no one has asked them, let’s read the post

        “Yes, people have already tried reaching out to the Lemmy instance admins in their Matrix room with no answer.”

        • ripcord@kbin.social
          20·
          1 year ago

          Ok, but, as the thread over there says, it’s not like they’re singling out kbin as far as the code shows. Just because no one there has answered yet doesn’t mean they’ve refused to answer. Give it some time or some actual replies before assuming the worst.

          Also that edit was added after my question.

          It’s been a few hours, this is way way too fast for the pitchforks to come out.

          • frex@kbin.social
            8·
            1 year ago

            If I use a firewall to block kbin in user agent then it doesn’t matter what code is running, it will throw 403.

            Edit: Just checked out, the first request with kbin user agent, it doesn’t have x-powered-by: Express, suggesting that it is blocked in nginx level, it doesn’t even reach the backend Express. This will not be reflected in the code whatsoever.

             ~  curl -I --user-agent "notKbinBot v0.1" https://lemmy.ml/u/test
HTTP/2 403
server: nginx
date: Wed, 28 Jun 2023 01:50:53 GMT
content-type: text/html
content-length: 146
vary: Accept-Encoding

 ~  curl -I --user-agent "placeholder-user-agent" https://lemmy.ml/u/test
HTTP/2 200
server: nginx
...
x-powered-by: Express
...
            • melroy@kbin.melroy.org
              3·
              1 year ago

              Yea this is bad. Lemmy.ml is actively blocking on ‘kbinbot’ (case insensitive) string in the user agent request.

              $ curl -I --user-agent "this is KBINBOT" https://lemmy.ml/u/test
HTTP/2 403
....

$ curl -I --user-agent "this is KBINBO" https://lemmy.ml/u/test
HTTP/2 200
....

$ curl -I --user-agent "this is BINBOT" https://lemmy.ml/u/test
HTTP/2 200
....
        • IntlLawGnome@kbin.social
          3·
          1 year ago

          Yeah, I posted that to !fediverse@lemmy.ml from my lemmy.ca account. I figured that if the admins aren’t answering questions via Matrix or whatever, maybe someone there would have an answer. (So far, nothing more than the same kind of speculation that’s happening here.)