Setting Generative System Model

Learn how to change the system model for Composer to any supported LLM

Models & Requirements

Default Model: Mistral 7B V0.1

Model	Name	Quantization	Minimum Requirements
Mistral 7B V0.1	mistralai/Mistral-7B-Instruct-v0.1	None	A10 24gb x1
Mixtral 8x7B	casperhansen/mixtral-instruct-awq	AWQ	A100 40gb x1

Setting Your System to a New Model

To change the model from the default, you will need to ssh into the instance in question.
Once you are logged in, follow these steps:

1. Become root with sudo su -
2. Once you have root access, you will be able to edit the Layar configuration file.
   Do this by running vi /data/layar/layar.config
3. Enter edit mode by typing i
4. Add the following line to the file: TGI_MODEL: model name where 'model name' is contents of the Name column above of the model you'd like to use.
5. Once you've made the edit, save and exit by first pressing ctrl+c to exit edit mode, then type :wqfollowed by enter to write the changes and exit the file.
   Be careful not to change anything other than the new line containing TGI_MODEL: model name
6. Now that you've edited the Layar configuration file, you'll need to restart the pods that consume TGI MODEL so that they are started with the correct model in place.
7. Run k delete deployment certara-llm certara-tgi
8. Run kps every 10 seconds until you don't see the llm and tgi pod in the stack
9. Run /data/layar/deployLayar.sh to redeploy the pods with the new model.
10. You should be able to use kps and k <pod name> logsto watch as the pods start up. The model server(the TGI pod) should log that the model you specified has been loaded successfully.

Once the pods have finished loading, you should now be able to now use Composer and other apps that use the /generate endpoint

If you have issues, please contact Certara Support at [email protected]