Open-R1: a Totally Open Reproduction Of DeepSeek-R1

Hey there! This post is an intro to the task, not a claim that we’ve reproduced R1 yet. We’re integrating in the open, so as quickly as we have examination numbers, we’ll share them. You can follow our progress on Hugging Face and GitHub.

True, but it appears like there’s absolutely nothing to be assessed since right now. I assume the ultimate objective is to train a new reasoning design and then utilize the very same assessment metrics as o1 and the DeepSeek-R1.

Well, there must be at least some sanity check and recognition to guarantee the model was trained properly.

Oh yes, if you are speaking about the examination variety of deepseek’s design it’s coming soon!

As discussed in the blog site post there is no design called Open-R1 to evaluate at all … not yet anyhow. This is a blog detailing that Hugging face will take the R1 Deepseek design, exercise how it was built as laid out in the paper and from what they released, and then replicate that procedure.

in fact this is basically how science works … A comes up with a strategy, discovery or innovation and it is tested by B, C and D to see if it is reproduceable. Thats been the cornerstone of research study now for a few centuries.

This blog is not stating they have already done so … Its a blog outlining an intent to begin training a design like R1 and calling it Open-R1.

Also DeepSeek-R1 was only launched recently, and even in their paper they laid out the calculate hours needed. While those are low compute hours for a SOTA model this does not indicate you can train said model in a week. I ‘d personally like to be able to train a transformer model in a week, but we may need to wait a while for that level of calculate technology.

So there are no standards for a model that has not been constructed yet right? As described in the blog site, and again in reply to your concern.

However fear not, there is a GitHub Repo already and factors (hell I may join myself), some prelim work done, and a plan of attack. An excellent beginning position.

n
@edbeeching
has examined the launched models already

( src: https://x.com/edwardbeeching/status/1884273209136275742)

R1 just trained on o1 outputs, so …/ s. This is what the brand-new AI czars are stating

Hi! This article is an intro to the task, not a claim that we’ve reproduced R1 yet. We will totally share the missing out on piece when we have them, you can expect the models and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

That’s great and crucial to understand this tremendous buzz that does not have technical comprehension and description. Science is about recreation, and if they claim to be open, let them fullfill the open part.

Please do release the training cost.

We will!

Excalidraw Hi n
@bojan2501
thanks, we will indeed be working hard to make sure this training recipe can work for little language models on customer hardware considering that not everybody has a cluster of H100s in your home:-RRB- The tool we utilized for the images was Excalidraw! https://excalidraw.com

looking forward to it! WTF are your talking about?

must be a joke

It’s really cool to see how the whole open source community comes together!

Ops …

5.5 M is number press reporter in the deepseekv3 tech report (just the training, not the experiment afaik), for R1 hard to estimate tbh but much less than 5.5 M imo

Historically, they have actually never released code or datasets of their LLM training, so I wouldn’t expect this time to be various. If they would launch it that would be fantastic of course!

Yes obviously!

So essentially you’re asking to replace existing censorship with another flavour of censorship?

The code for the models are inside the design repositories, e.g. for V3: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py

Hello Team, I’m Ray Bernard, the author and creator of EQUATOR. My research team will be working on a paper focused on duplicating certain parts of DeepSeek R1. Our goal is to recreate the cold start and provide your group with a dataset that includes COT and other techniques to support these efforts. We like to contribute our work to assist. Please let me know if you find this beneficial. Best, Ray Bernard https://www.facebook.com/groups/1186310571520299/

Where is the evaluation numbers? without it you can’t call it recreation.

8 replies

True, but it seems like there’s absolutely nothing to be evaluated since right now. I presume the supreme goal is to train a brand-new reasoning design and then utilize the same assessment metrics as o1 and the DeepSeek-R1.

That’s rather intriguing, I was asking myself why the questions the author exposed here are not being asked by others? I believe the work they have actually done is memorable but at the exact same time I wonder why they wouldn’t put these missing out on pieces on if they are expected to be fully open.
Why even without reproduction and comprehension of the innovation they could affect a lot the marketplace in this method?

4 replies

Hi! This article is an intro to the project, not a claim that we have actually replicated R1 yet. We will absolutely share the missing piece when we have them, you can anticipate the models and datasets to be upload in this Hugging Face org and the code to be in this GitHub repo

Interesting read, and it is good that we see more effort into this instructions: more optimization and less strength.
Also wonder what tool did the author use for producing action diagram.

2 replies

Excalidraw I’m so happy that effort like this currently exist, I’m gon na attempt to contribute:-RRB- 1 reply

anticipating it! So racist articel

2 replies

WTF are your talking about?

Awesome to have this open recreation began!

For Step # 1 check out https://github.com/open-thoughts/open-thoughts!

https://x.com/ryanmart3n/status/1884284101265612856

Let’s do this thing!

1 reply

It’s really cool to see how the whole open source community comes together!

Does anyone know the real training cost of r1? I can’t discover it in the paper or the announcement post. Is the 6M expense reported by media just the number drawn from v3’s training cost?

2 replies

Ops …

Has anybody asked the DeepSeek group to release their training data and code, or at least share them independently with an independent replication task like this? Have they rejected such a request?

A loyal duplication depends on utilizing the very same dataset and hyperparameters. Otherwise, any major disparities with the published criteria would be difficult to pin down-whether due to training information differences or the replication approach itself.

1 reply

Historically, they have actually never ever launched code or datasets of their LLM training, so I wouldn’t anticipate this time to be different. If they would release it that would be fantastic naturally!

In the meantime we need to make finest guess estimates and see if we can get there ourselves.

You provide excellent replication process of Deepseek reasoning training. I will try something similar to it.

This is really good information, can we tweak with specific usage case when code is released?

1 reply

Yes naturally!

Please consider removing biased, tainted or unaligned training data and make an effort to get rid of copyrighted works from the crawl from intake. This will make the design more functional. If you recycled anthropic curation checks, this may also assist, remove obviouslybiased information will likely add a lot of value. We do not desire another polluted, unaligned open source design, right? And no corporate would ever utilize deepseek or a design that recycles it, right?
We appreciate your work for the benefit of mankind, we hope.
Miike C from NJ

1 reply

So basically you’re asking to replace existing censorship with another flavour of censorship?

Can’t wait! Hopefully the design will be uncensored however whatever you can do is alright! Love seeing open source building itself up. I’m not wise adequate to actually assist however I can contribute moral assistance lol

Hello guys, I am even just looking for code for DeepSeek-V2, in order to completely comprehend multi-head latent attention. You do not seem to have code in Hugging Face even for that. Or am I missing out on something? Don’t see anything in src/transformers/models. MLA is not effectively described in their paper, so it would be essential to have code for this.