Hi y'all, is it possible to change the LLM model being used for Assistant? How do I do that? I'd like to give each individual user the option to use whichever model they want to for the task at hand.
Actually I believe the Model Hub is only available using the Glean key. So pros of using Glean key are: access to model hub (which allows you to access a lot of different LLM options), you don't have to maintain all the different models and cost structures on your end, no need to upgrade or manage different models in your Glean > we will do that for you (especially upgrading when new models are available or deprecating old modes), and no need to forecast for consumption (which is another issue many of our customers come up against when using their own key).
Today the LLM is set globally by the admin.
This has been a long time ask at my company. I believe model flexibility especially across providers, is an obvious strategic advantage for Glean.
Even as the global admin I don't seem to be able to change which model flavor I use unless I switch to using my own OpenAI key. It seems Agents have the advantage of picking models but the regular Assistant doesnt?
That's right @nueshiraz - currently our Model Hub is only available in agents. I can see us providing this framework outside of agents in the future, but nothing currently on the roadmap.
Got it, so if I switch from using Glean's custom model to my own model with API key access (at the global level), that may be one way for me to give my global user base an alternative model? I'm unclear on the pros and cons of doing this, other than the obvious cost I have to directly incur.
Here is a link to the models Glean currently supports when using the Glean Model Hub.
Being able to select a model in Assistant will be huge. I find the default model to be over optimised for speed. It's been a long pending ask from us, to choose models that are more effective for comprehensive analysis and content creation.
