Our Latest Model

Our latest release, Smaug Agentic, is an agentic supervised fine-tune of Kimi K3, the frontier Mixture-of-Experts model from Moonshot AI. It continues the Smaug line — which produced the first open-source model to pass an average score of 80% on the HuggingFace Open LLM Leaderboard — scaled to a frontier MoE base and targeted at long-horizon tool use and coding agents.

The model was trained on filtered multi-turn, tool-using coding trajectories with reasoning tokens masked from the loss, so it improves agentic behavior while preserving the reasoning that made the base model worth choosing. Every architectural parameter is unchanged from the base model, so any inference stack that serves Kimi K3 serves Smaug Agentic as a drop-in replacement, and interleaved thinking is preserved across turns so multi-step agent loops keep the model's chain of thought.

On our own benchmark runs, Smaug Agentic improves on the published Kimi K3 reference figures on GPQA Diamond (94.1 vs 93.5), DeepSWE (69.9 vs 67.5), SciCode (60.8 vs 58.7), and LiveBench Agentic Coding (64.6 vs 62.2). The behavioral change is bigger than the benchmark deltas: runaway reasoning is suppressed while normal deliberation is preserved, the model emits one decisive shell action per turn, and it sustains very long agent loops — across 113 agentic coding tasks and more than seven hours of continuous work, it ran a median of 78 agent steps per task with no infrastructure errors and no timeouts. Full per-benchmark notes are in the model card on Hugging Face.

Our Benchmarks

With LLMs training on web-scale data, test-set contamination is a pervasive concern in LLM evaluation that can quickly render benchmarks obsolete. To mitigate this, many recent benchmarks crowdsource new prompts and evaluations from human or LLM judges; however, these can introduce significant biases, and they are not reliable for hard questions. For example, LLM judges make mistakes up to 40% of the time on challenging math and reasoning tasks.

To resolve this issue, we developed LiveBench, a contamination-limited benchmark that evaluates large language models on a variety of general intelligence capabilities including reasoning, coding, language understanding, data analysis, instruction following, and mathematics. By frequently releasing updated question sets and developing new tasks over time, we ensure that our results remain an accurate assessment of LLM capabilities as new models are released. Questions are constructed from a variety of recent sources such as research papers and news articles so they can be easily refreshed over time. LiveBench also uses only objective ground-truth judgment to ensure unbiased results.

Our leaderboard at livebench.ai presents results for all major model providers, both open source and proprietary. Since its release, LiveBench has remained one of the most popular LLM benchmarks and has been a featured result in major model release reports.

Our Previous Models

We have two Dracary fine tunes - One based on Llama-3.1 70b and another based on Qwen 72B.

These fine tunes enhance the coding and reasoning abilities of the base LLM. Dracarys is built on a refined formula called the "Dracarys recipe," which applies optimized fine-tuning techniques to large-scale models such as Qwen-2 72B, Qwen2.5-72B, and LLama-3.1 70B.

Our research and experiments on creating this intelligent dataset and fine-tuning methodologies significantly boost the coding and reasoning capabilities of these open-source LLMs.

The Dracarys llama model shows improvements on livebench.ai with a significant increase the coding score which improves to 36.59 from 33.49. Overall it's a better model and ranks higher on the leaderboard.

Smaug is our flagship open-source line of models, and set a new standard for open-source, topping the HuggingFace OpenLLM leaderboard with an accuracy of 80.48%, nearly 2% better than the next best model. We have introduced several Smaug fine tunes, with the flagship model being Smaug-72B. This model is a fine-tuned version of Qwen-72B, enhanced through our novel Direct Preference Optimization-Positive (DPOP) method.

Unlike traditional DPO, which focuses on improving model performance at the risk of reducing completion likelihood, DPOP introduces a new term in the loss function that penalizes any reduction in the likelihood of positive outcomes. This innovation addresses a critical shortcoming in LLM fine-tuning and significantly improves model reliability and effectiveness.

We also applied these techniques to make Smaug-34B and Smaug-Mixtral, both of which are leaders in performance in their classes.

Open-source LLMs have continued to proliferate within the AI landscape and have shown comparable performance to the close-sourced LLMs of OpenAI, Google, and others. However, open-source LLMs often come with a limited context length, limiting their utility for creating custom LLMs on your knowledge base. Since you are constrained by the amount of proprietary data you can send in a single API call, a bigger context length is crucial for various tasks.

Many methods have been proposed for context-length extrapolation; in our extensive research, we tested each approach thoroughly and proposed two new approaches. One of these approaches is truncation, which has shown promising results.

Along with this research, we released Giraffe. Based on Llama-2, this model is the world's first open-source LLM capable of handling a 32k context. This capability is vital for various applications, from complex information retrieval to sustained conversational AI and code generation on an existing use case. On an enterprise level, it can function as an AI brain for your business, boosting productivity, improving decision-making, and providing key insights.

Our Datasets

We used three datasets to create our Smaug series of models. These datasets were meant to be used to fine-tune LLMs using the DPOP loss function. Arc_DPO_FewShot was used to test the level of understanding of science at the grade-school level. HellaSwag_DPO_FewShot contains common sense inference questions that LLMs commonly struggle with. MetaMath_DPO_Fewshot was used to measure math and reasoning skills in an LLM and to align models toward being precise in the calculation.

SystemChat is a dataset with 7000 synthetic conversations generated with Mistral-Medium and Dolphin-2.7-mixtral-8x7b. It was designed to teach model compliance to the system prompt over long multiturn conversations, even with unusual or mechanical system prompts. Fine-tuning with this dataset makes it far more usable and harder to jailbreak. No guardrails or censorship are added to the dataset, and you can implement your own alignment layer.

The WikiQA task is the task of answering a question based on the information given in a Wikipedia document. We have selected large Wikipedia documents and truncated them to get multiple versions of the same document sizes varying between 2000 to 16000 tokens. Each size of document also had multiple versions which places the question and answer text at different locations.

However, a Wikipedia based dataset could correctly answer from its pretrained corpus and not from context. To combat this, we created an "altered" dataset, where the data only consists of questions which have numerical answers. This ensures that if an LLM recollects from its pretrained corpus, it gives a wrong answer.

We also created MetaMathFewShot, a new few-shot version of the popular MetaMath dataset. This allows the model to understand the concept of few-shot prompting. Our LongChat-Lines was used to evaluate the performance of a model fine-tuned to operate on longer contexts.

An AI Super Assistant For Enterprises And Professionals