Back to Blog

Run MergeKit on a Free Colab

Published on December 3, 2023.

If you're into merging LLMs, you've probably come across MergeKit. MergeKit is an open sourced platform that allows you to easily merge models from a YAML configuration file without writing a single line of code.

When merging LLMs, there are a few essential things you need:

• A fast internet connection, especially upload speeds if you want to share your model
• Enough disk space for both models

For example, if you wanted to merge two 7B models (around 15 GB each), you'd need 30GB of disk space. The resulting model would only be 15 GB.

Another issue is memory. If you just run MergeKit normally to merge two 7B models, you'd need ~15 GB of memory. If you try to merge it on Google Colab, which only provides 12.7 GB of memory, the instance will crash.

The solution is to chunk the model. I'd like to thank the author of MergeKit in helping me figure this out, and after some experimenting, I've found that a shard size of 1B fits perfectly on a free Colab instance.

mergekit-yaml "config.yml" model-path --allow-crimes --out-shard-size 1B