Skip to main content

HuggingGPT

Implementation of HuggingGPT. HuggingGPT is a system to connect LLMs (ChatGPT) with ML community (Hugging Face).

Set up tools​

We set up the tools available from Transformers Agent. It includes a library of tools supported by Transformers and some customized tools such as image generator, video generator, text downloader and other tools.

from transformers import load_tool
hf_tools = [
    load_tool(tool_name)
    for tool_name in [
        "document-question-answering",
        "image-captioning",
        "image-question-answering",
        "image-segmentation",
        "speech-to-text",
        "summarization",
        "text-classification",
        "text-question-answering",
        "translation",
        "huggingface-tools/text-to-image",
        "huggingface-tools/text-to-video",
        "text-to-speech",
        "huggingface-tools/text-download",
        "huggingface-tools/image-transformation",
    ]
]

Setup model and HuggingGPT​

We create an instance of HuggingGPT and use ChatGPT as the controller to rule the above tools.

from langchain.llms import OpenAI
from langchain_experimental.autonomous_agents import HuggingGPT
# %env OPENAI_API_BASE=http://localhost:8000/v1

API Reference:

llm = OpenAI(model_name="gpt-3.5-turbo")
agent = HuggingGPT(llm, hf_tools)

Run an example​

Given a text, show a related image and video.

agent.run("please show me a video and an image of 'a boy is running'")