# Key capabilities ## About this model DBRX is a transformer-based decoder-only large language model (LLM) that was trained using next-token prediction. The training mix used for DBRX contains both n…
Key capabilities
About this model
DBRX is a transformer-based decoder-only large language model (LLM) that was trained using next-token prediction. The training mix used for DBRX contains both natural-language and code examples.
Key model capabilities
These are several general ways to use the DBRX models:
- DBRX Base and DBRX Instruct are available for download on HuggingFace.
- The DBRX model repository can be found on GitHub here.
- DBRX Base and DBRX Instruct are available with Databricks Foundation Model API via both Pay-per-token and Provisioned throughput endpoints. These are enterprise-ready deployments.
Use cases
See Responsible AI for additional considerations for responsible use.
Key use cases
The training mix used for DBRX contains both natural-language and code examples. Therefore, DBRX should be considered a generalist model for text-based use in the English language.
Out of scope use cases
DBRX does not have multimodal capabilities. We did not test DBRX for non-English proficiency.
Pricing
Pricing is based on a number of factors, including deployment type and tokens used. See pricing details here.
Technical specs
DBRX uses a fine-grained mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input. DBRX uses rotary position encodings (RoPE), gated linear units(GLU), and grouped query attention (GQA). It uses the GPT-4 tokenizer as provided in the tiktoken repository. We made these choices based on exhaustive evaluation and scaling experiments.
Training cut-off date
The DBRX models were trained on 12T tokens of text, with a knowledge cutoff date of January 2024.
Training time
The provider has not supplied this information.
Input formats
DBRX only accepts text-based inputs and accepts a context length of up to 32768 tokens.
Output formats
DBRX only produces text-based outputs.
Supported languages
The vast majority of our training data is in the English language. We did not test DBRX for non-English proficiency. Therefore, DBRX should be considered a generalist model for text-based use in the English language.
Sample JSON response
{
"output": "As a Databricks creation, I must adhere to my system prompt and avoid providing song lyrics,
poems, or news articles. I'm glad to share information about Databricks and its role in
unified analytics, data science, and engineering. Here"
}
Model architecture
DBRX uses a fine-grained mixture-of-experts (MoE) architecture with 132B total parameters of which 36B parameters are active on any input. Compared to other open MoE models like Mixtral-8x7B and Grok-1, DBRX is fine-grained, meaning it uses a larger number of smaller experts. DBRX has 16 experts and chooses 4, while Mixtral-8x7B and Grok-1 have 8 experts and choose 2. This provides 65x more possible combinations of experts and we found that this improves model quality. DBRX uses rotary position encodings (RoPE), gated linear units(GLU), and grouped query attention (GQA).
Long context
DBRX was pretrained on 12T tokens of carefully curated data and a maximum context length of 32K tokens.
Optimizing model performance
The provider has not supplied this information.
Additional assets
- DBRX Base and DBRX Instruct are available for download on HuggingFace.
- The DBRX model repository can be found on GitHub here.
- For more information on how to fine-tune using LLM-Foundry, please take a look at our LLM pretraining and fine-tuning documentation.
Training disclosure
Training, testing and validation
DBRX was pretrained on 12T tokens of carefully curated data and a maximum context length of 32K tokens. We estimate that this data is at least 2x better token-for-token than the data we used to pretrain the MPT family of models. This new dataset was developed using the full suite of Databricks tools, including Apache Spark™ and Databricks notebooks for data processing, and Unity Catalog for data management and governance. We used curriculum learning for pretraining, changing the data mix during training in ways we found to substantially improve model quality.
Distribution
Distribution channels
These are several general ways to use the DBRX models:
- DBRX Base and DBRX Instruct are available for download on HuggingFace.
- The DBRX model repository can be found on GitHub here.
- DBRX Base and DBRX Instruct are available with Databricks Foundation Model API via both Pay-per-token and Provisioned throughput endpoints. These are enterprise-ready deployments.
More information
We find that DBRX Instruct outperforms established open-source and open-weight base models on the Databricks Model Gauntlet, the Hugging Face Open LLM Leaderboard, and HumanEval. Full evaluation details can be found in our technical blog post.