Rclone can do file mounts as well as sync.

I feel like this article is exactly the type of thing it’s criticizing.

The problem is that while LLMs can translate, it’s still machine translation and isn’t always accurate. It’s also not going to just be for that. It’ll be applying “AI” to everything that looks like it might vaguely fit, and it’ll stifle productivity.

Lol, there are smaller versions of Deepseek-r1. These aren’t the “real” Deepseek model, but they are distilled from other foundation models (Qwen2.5 and Llama3 in this case).

For the 671b parameter file, the medium-quality version weighs in at 404 GB. That means you need 404 GB of RAM/VRAM just to load the thing. Then you need preferably ALL of that in VRAM (i.e. GPU memory) to get it to generate anything fast.

For comparison, I have 16 GB of VRAM and 64 GB of RAM on my desktop. If I run the 70b parameter version of Llama3 at Q4 quant (medium quality-ish), it’s a 40 GB file. It’ll run, but mostly on the CPU. It generates ~0.85 tokens per second. So a good response will take 10-30 minutes. Which is fine if you have time to wait, but not if you want an immediate response. If I had two beefy GPUs with 24 GB VRAM each, that’d be 48 total GB and I could run the whole model in VRAM and it’d be very fast.

They’re probably referring to the 671b parameter version of deepseek. You can indeed self host it. But unless you’ve got a server rack full of data center class GPUs, you’ll probably set your house on fire before it generates a single token.

If you want a fully open source model, I recommend Qwen 2.5 or maybe deepseek v2. There’s also OLmo2, but I haven’t really tested it.

Mistral small 24b also just came out and is Apache licensed. That is something I’m testing now.

Most open/local models require a fraction of the resources of chatgpt. But they are usually not AS good in a general sense. But they often are good enough, and can sometimes surpass ChatGPT in specific domains.

It’s enough to run quantized versions of the distilled r1 model based on Qwen and Llama 3. Don’t know how fast it’ll run though.

Seems like something got messed up when copy and pasting. It’s fixed now.

Thanks for catching it!

Yeah, it would be a good idea. Not to auto-update functions, because that would be very very bad, but to at least indicate there’s an update available.

Huge cliffhanger. Doesn’t end with any kind of conclusion. Can’t decide if Universe or Atlantis has the worse ending in that regard.

They can build a keyboard into it, sure. It’s just UI elements and a bunch of buttons. Won’t be a good keyboard, but it can be done.

OpenWebUI connected tabbyUI’s OpenAI endpoint. I will try reducing temperature and seeing if that makes it more accurate.

Context was set to anywhere between 8k and 16k. It was responding in English properly, and then about halfway to 3/4s of the way through a response, it would start outputting tokens in either a foreign language (Russian/Chinese in the case of Qwen 2.5) or things that don’t make sense (random code snippets, improperly formatted text). Sometimes the text was repeating as well. But I thought that might have been a template problem, because it seemed to be answering the question twice.

Otherwise, all settings are the defaults.

I tried it with both Qwen 14b and Llama 3.1. Both were exl2 quants produced by bartowski.

Perplexica works. It can understand ollama and custom OpenAI providers.

Super useful guide. However after playing around with TabbyAPI, the responses from models quickly become jibberish, usually halfway through or towards the end. I’m using exl2 models off of HuggingFace, with Q4, Q6, and FP16 cache. Any tips? Also, how do I control context length on a per-model basis? max_seq_len in config.json?

Seems to be the only necessary thing in my case! Thanks.

Yeah I definitely have the default GTK chooser. Guess I have some config playing to do later.

Can you explain a bit more about this and how to configure it? When I use FF on gnome, the save dialogue just looks like other dialogues?

How much speed are you actually getting on Mixtral (I assume that’s the 8x7b). I have 64 GB of RAM and an AMD RX 6800 XT with 16 GB of VRAM. I get like 4 tokens per second with Q5_K_M quant.