Back to Blog

Run MergeKit on a Free Colab

Published on December 3, 2023.

If you're into merging LLMs, you've probably come across MergeKit. MergeKit is an open sourced platform that allows you to easily merge models from a YAML configuration file without writing a single line of code.

When merging LLMs, there are a few essential things you need:

For example, if you wanted to merge two 7B models (around 15 GB each), you'd need 30GB of disk space. The resulting model would only be 15 GB.

Another issue is memory. If you just run MergeKit normally to merge two 7B models, you'd need ~15 GB of memory. If you try to merge it on Google Colab, which only provides 12.7 GB of memory, the instance will crash.

The solution is to chunk the model. I'd like to thank the author of MergeKit in helping me figure this out, and after some experimenting, I've found that a shard size of 1B fits perfectly on a free Colab instance.

mergekit-yaml "config.yml" model-path --allow-crimes --out-shard-size 1B

© 2024 mrfakename. All rights reserved. Please acquire permission before usage, redistribution, or republication.