Running DeepSeek R1 locally
 

DeepSeek has its flagship V3 model equivalent to GPT4 and it's reasoning model R1 freely accessible: AI training for 5.6 million USD exceeding the quality of 100 mil to 1 bil USD. Inference of high quality is within reach of your own local environment where your data stays private. I've found their models better at reasoning than OpenAIs significantly. It's quite exciting and I'm surprised noone has brought the topic up yet, given the large amount of use cases for reverse engineering and the very low cost.

If you have a GPU recommend OLlama which works on Windows, Linux and Mac (can also rin Facebook/Meta's Llama models): . Can choose from the models listed here: . 8b is pretty lightweight but if you have a recent Nvidia GPU with a lot of RAM why not go for 32b.

For a frontend chat interface, I recommend Chatbot AI: Even better the V3 and R1 model are open source and you can do your own model finishing if you have the resources.

R1: V3:

Are there any reverse engineering (especially for decompilation), specific models?

This makes DIGITS even more interesting to me, considering buying one. (But it will only be out in may). https://www.nvidia.com/en-eu/project-digits/

What I dont get though is how 1PFLOG in FP4 is a selling point when the regular GTX5090 has like 600 TFLOP in FP32? Am I missing something or is the 128GB integrated RAM the selling factor?

There is even more powerfull ML model then DeepSeek - Alibaba’s Qwen2.5-Max

DeepSeek censorship:

It would be nice to train an RE model. The good news now is that training is being shown to be feasible possibly on an academic grant level budget. Someone should train a proper open source RE model at some point.

1PFLOP FP4 is a marketing gimmick maybe, that amount of RAM is a big plus tho. The new DeepSeek models use FP8 and have shown it's reliable for training, a good breakthrough. Sounds good enough to run good size models at moderate load.

Alibaba sounds interesting haven't heard much about it.

By the way DeepSeek censorship from demos I saw is on the website but at least running R1 locally, it seems to not be censoring those things much or at all.

Censorship update: it appears if you download models and run locally there is no censorship.

Censoring is definitely done on their public website if you run your queries on their hardware. Here is how it works:
when you send your query it goes into the real model avatars thinking or generating a response.

At the same time, it is sent to a classifier model that is far cheaper, faster and specific. This classifier is trained with a prompt similar to "is the following '<prompt>' related to the following list of sensitive topics". If it returns yes, the main query is immediately aborted and a message displayed. Sometimes you can see it start to think and cut off a few sentences into it's deepthink. Other time it cuts off so fast that it appears nothing yet was emitted.

No problem, run it locally, you won't deal with the censorship classifier. It would be interesting though to have a list of the sensitive topics but that is probably kept secretly and securely.

Update: According to this article I am mistaken and the censorship concerns mentioned are legit:

I only can prompt twice, then deepseek says,

I asked it about Falon Gong last night (locally) - and spent about 15 minutes debating it.

But let's be real.. ? Does anyone except it to behave otherwise--if so, you're kidding yourself and need to stop.

Locally its quite a useful tool... especially if you are playing with some of the more 'schizo' rethinking builds available via huggingface.

Regards,

Fyyre

I know its a late reply but the issue is still present. The reason is the recent spike in popularity through tons of youtube videos, tik toks etc etc. DeepSeek is under heavy pressure and currently not able to handle the user load. Their API was down for roughly 10 days at the beginning of february aswell.

Besides the normal user load, they currently have to manage, they are currently still under heavy DDOS attacks. Because of that its currently not even possible to top up API Credits :(

Hopefully they will resolve this soon.

In the meantime if you have an X Account Grok 3 seems to be the new kid on the block and atleast yesterday I was ablo to use it without any subscriptions.

Dont get used to it tho they hiked up the prices (they doubled them) and probably will get their asses kicked for it since nobody I know will pay 40$ a month for access to a Chat LLM if you can have slightly worse for 1/10 of the price.

Best regards

Hi Guys,

there is a nice publication about this here: https://arxiv.org/pdf/2505.19915

Does anyone here know more about this topic and used such agents?


Best regards

Is there any tutorial on how to fine-tune Deepseek Coder V2?

I am facing dependency hell when trying to setup an environment that can be used for additional training on private C source code.

ollama is good for running it on Windows/WSL, but not for additional training.

Regards,
Tom

This one is new too

https://huggingface.co/fdtn-ai/Foundation-Sec-8B

There are discussions on the Hugging Face page (e.g., fdtn-ai/Foundation-Sec-8B/discussions/10) about running this model with Ollama. Users have attempted it, and there's an Ollama entry for an "abliterated" (uncensored) version huihui_ai/foundation-sec-abliterated:8b