Lemmy MichaelSasser
  • Communities
  • Create Post
  • heart
    Support Lemmy
  • search
    Search
  • Login
  • Sign Up
juergen@feddit.org to Technology@lemmy.worldEnglish · 2 months ago

mwbml, a search engine where the results are community ranked, has indexed 1 billion pages.

blog.mwmbl.org

external-link
message-square
18
link
fedilink
210
external-link

mwbml, a search engine where the results are community ranked, has indexed 1 billion pages.

blog.mwmbl.org

juergen@feddit.org to Technology@lemmy.worldEnglish · 2 months ago
message-square
18
link
fedilink
Indexing a billion pages
blog.mwmbl.org
external-link
It’s two years since we launched Mwmbl, the open source, non-profit search engine, on Boxing Day 2021. A good time to take stock of where we are and where we’re going. We’ve indexed over 100 million pages Thanks to our volunteers, who crawl the web using the Firefox extension and command line script, we’re crawling up to a million pages a day, as you can see on our stats page. There are around 50-60 users crawling on an average day.
alert-triangle
You must log in or register to comment.
  • Tywèle [she|her]@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    36
    ·
    2 months ago

    What is it with open source projects and their awful names?

    • Nima@leminal.space
      link
      fedilink
      English
      arrow-up
      11
      ·
      2 months ago

      it always has to be something that sounds like it was thought up tiredly at 4am by the college freshman who coded it.

      • Tywèle [she|her]@lemmy.dbzer0.com
        link
        fedilink
        English
        arrow-up
        8
        ·
        2 months ago

        Or like something that got typed by that same college freshman by falling asleep and hitting their head on the keyboard.

  • cygnus@lemmy.ca
    link
    fedilink
    English
    arrow-up
    29
    arrow-down
    4
    ·
    2 months ago

    we launched Mwmbl, the open source, non-profit search engine, on Boxing Day 2021

    Oh and it’s nonamerican too? Definitely going to try this out.

  • Clot@lemm.ee
    link
    fedilink
    English
    arrow-up
    19
    ·
    2 months ago

    Off topic but when I googled this, I saw Lemmy post😁

    • scottrepreneur@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      ·
      2 months ago

      The OP is from late 2023…

  • brucethemoose@lemmy.world
    link
    fedilink
    English
    arrow-up
    11
    ·
    2 months ago

    Oh, this looks awesome.

  • cabbage@piefed.social
    link
    fedilink
    English
    arrow-up
    9
    ·
    2 months ago

    Does the actual search engine load for anyone else? It’s telling me “We’re sorry, but something went wrong”.

    Maybe it’s the first ever Lemmy hug of death.

    • juergen@feddit.orgOP
      link
      fedilink
      English
      arrow-up
      5
      ·
      edit-2
      2 months ago

      it works for me.

      • cabbage@piefed.social
        link
        fedilink
        English
        arrow-up
        4
        ·
        edit-2
        2 months ago

        Back up again now!

        https://mwmbl.org/

        Edit: And it seems pretty great!

        • MonkderVierte@lemmy.ml
          link
          fedilink
          English
          arrow-up
          1
          ·
          2 months ago

          deleted by creator

  • 反いじめ戦隊@ani.social
    link
    fedilink
    English
    arrow-up
    9
    ·
    2 months ago

    This should be fed to YaCY.

  • drspod@lemmy.ml
    link
    fedilink
    English
    arrow-up
    8
    ·
    2 months ago

    An SEO ghoul’s wet dream.

  • jqubed@lemmy.world
    link
    fedilink
    English
    arrow-up
    7
    ·
    2 months ago

    This blog post is from December 23, 2023. It looks like there was only one newer blog post, from November 2024.

  • MonkderVierte@lemmy.ml
    link
    fedilink
    English
    arrow-up
    6
    ·
    2 months ago

    Please remove the ‘!important’ statements in your CSS, it makes the site unusable for me.

  • blackfire@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    2 months ago

    This is effectively yacy with voting no?

    • juergen@feddit.orgOP
      link
      fedilink
      English
      arrow-up
      2
      ·
      2 months ago

      dont think so, i think yacy is more decentral where each user hosts that locally. this one is on the server.

  • iopq@lemmy.world
    link
    fedilink
    English
    arrow-up
    2
    ·
    2 months ago

    (2023)

  • spicystraw@lemmy.world
    link
    fedilink
    English
    arrow-up
    4
    arrow-down
    2
    ·
    2 months ago

    An update from November 2024 on the linked site indicates they have indexed 500 million pages. Thank you for sharing, but I would appreciate it if you could verify the information and avoid making assumptions based on misinterpretations.

Technology@lemmy.world

technology@lemmy.world

Subscribe from Remote Instance

Create a post
You are not logged in. However you can subscribe from another Fediverse account, for example Lemmy or Mastodon. To do this, paste the following into the search field of your instance: !technology@lemmy.world

This is a most excellent place for technology news and articles.


Our Rules


  1. Follow the lemmy.world rules.
  2. Only tech related news or articles.
  3. Be excellent to each other!
  4. Mod approved content bots can post up to 10 articles per day.
  5. Threads asking for personal tech support may be deleted.
  6. Politics threads may be removed.
  7. No memes allowed as posts, OK to post as comments.
  8. Only approved bots from the list below, this includes using AI responses and summaries. To ask if your bot can be added please contact a mod.
  9. Check for duplicates before posting, duplicates may be removed
  10. Accounts 7 days and younger will have their posts automatically removed.

Approved Bots


  • @L4s@lemmy.world
  • @autotldr@lemmings.world
  • @PipedLinkBot@feddit.rocks
  • @wikibot@lemmy.world
Visibility: Public
globe

This community can be federated to other instances and be posted/commented in by their users.

  • 3.53K users / day
  • 9.7K users / week
  • 18.2K users / month
  • 37.7K users / 6 months
  • 1 local subscriber
  • 69.8K subscribers
  • 10.4K Posts
  • 369K Comments
  • Modlog
  • mods:
  • L3s@lemmy.world
  • enu@lemmy.world
  • Technopagan@lemmy.world
  • L4sBot@lemmy.world
  • L3s@hackingne.ws
  • L4s@hackingne.ws
  • BE: 0.19.11
  • Modlog
  • Legal
  • Instances
  • Docs
  • Code
  • join-lemmy.org