wonderingwanderer

It would be really funny if the whole world decides to dump its US treasury bonds all at once, so that when the AI bubble does burst the US has nothing with which to bail out the tech companies responsible, and the rest of the world doesn’t have to share the brunt of the fall…

See, you have more experience in the matter than I do, hence the caveat that I’m not an expert. Thanks for sharing your experience.

Then again, I’d consider 128GB of memory to be fairly serious hardware, but if that’s common among hobbyists then I stand corrected. I was operating on the assumption that 64GB of RAM is already a lot

All in all, 106 billion parameters on 128GB of memory with quantization doesn’t surprise me all that much. But again, I’m just going off of the vague notions I’ve gathered from reading about it.

The focus of my original comment was more on the fact that self-hosting is an option, I wasn’t trying to be too precise with the specs. My bad if it came off that way

“People don’t want our product and we’re in danger of failure, wahhh” -capitalist oligarch

Something something free market dynamics something something. Oh wait, they only believe in that when it means deregulating, but if it means letting their company fail then all of a sudden everyone else has a responsibility to carry weight for them?

Privatized profits and socialized losses. Damn greedy crooks.

If Microsoft cared about privacy then they wouldn’t have made windows practically spyware. Even if they install AI locally in the OS, it’s still proprietary software that constantly sends data back to the mothership, consuming your electricity and RAM to do so. Linux has so many options, there’s really no reason not to switch.

Small LLMs already exist for local self-hosting, and there are open-source options which won’t steal your data and turn you into a product.

https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard#/

Bear in mind that the number of parameters your system can handle is limited by how much memory is available, and using a quantized version can increase the number of parameters you can handle with the same amount of memory.

Unless you have some really serious hardware, 24 billion parameters is probably the maximum that would be practical for self-hosting on a reasonable hobbyist set-up. But I’m no expert, so do some research and calculate for yourself what your system can handle.

They’ll still find a way to triangulate whose dick that is

“Okay, bluetooth disabled for 24 hours.” It’ll turn itself back on tomorrow.

Microsoft knows your dick size