• MonkderVierte@lemmy.zipEnglish
    11·
    2 months ago

    How does archive get the unpaywalled version? I don’t think they pay the subscription for every single tabloid out there?

    Asking for a friend.

    • stoly@lemmy.worldEnglish
      7·
      2 months ago

      The paywall is JavaScript but the content is still in plaintext below. The crawlers don’t read the JavaScript.

      • MonkderVierte@lemmy.zipEnglish
        8·
        2 months ago

        Disabling 3rd-party js has no paywall, but only the first paragraph too. Crawlers get full access?

    • AnarchistArtificer@slrpnk.netEnglish
      6·
      2 months ago

      I think they use the same thing that web crawlers use. If Google’s crawler couldn’t access the content of the page (or could only access a limited amount of content), it would likely rank far lower in search results

      • MonkderVierte@lemmy.zipEnglish
        3·
        2 months ago

        Btw, how come there is no search engine where you can sort and filter how you want instead of how they want? (except self-hosted i mean)

        Pornhub has better searchability than, uh, all search sites i know.