Dolly (Databricks)

Overview Dolly (specifically Dolly 2.0) was a pivotal open-source Large Language Model released by Databricks in April 2023. It was the first LLM to feature a completely open-source, commercially usable instruction-tuning dataset.

The Innovation: databricks-dolly-15k Before Dolly, most open models (like Alpaca) were trained on data generated by ChatGPT. The OpenAI Terms of Service forbade using this output to train competing commercial models. Databricks created a dataset of 15,000 high-quality prompts/responses written by their own employees (clean, human-generated data).

• License: Creative Commons (CC-BY-SA). Anyone can use it for commercial purposes.

The Model

• Base: Pythia-12B.
• Training: Fine-tuned on the 15k dataset.
• Capabilities: Brainstorming, classification, generation.

Legacy Dolly was not the smartest model, but it proved that Data Quality > Data Quantity. A small, high-quality dataset could give a model "instruction following" capabilities better than massive noisy datasets. It kicked off the "Open Source Commercial LLM" wave.