Offline AI Chatbots: How to Install GPT4ALL and Jan on Your PC

Running a chatbot without an Internet connection is possible

If you regularly use ChatGPT, you are probably very used to accessing it through the browser. You can do it from your computer, but also from your mobile. All the orders you send are processed on a remote server and have almost no impact on your device. This approach, to which we have become accustomed thanks to thousands of apps that work in the cloud, is not always the most convenient. What if you could install language models like GPT on your computer and use them without an Internet connection?

That is precisely what we propose in this guide. In the following sections we explain what steps you must follow to enjoy a chatbot that execute the requests locally, that is, taking advantage of the power of your equipment and without connecting to the Internet. As you can imagine, all the solutions we talk about here are designed for PC and Mac. However, the best thing is that you will not need a super computer to run the LLM you prefer.

Artificial intelligence (AI) locally: what you should know

Before talking about GPT4All and Jan, which are the two tools that will help us install LLM models locally, we need to put ourselves in the background. We consider it appropriate to clarify a few things before getting to work.

These are the points you should know:

• You don't need much power. When we explain how to use Stable Diffusion without Internet, we are talking about quite high requirements. However, with the tools that we propose in this publication, you will not need a very powerful PC or even a dedicated graphics card. In fact, to know to what extent it is possible to use a modest device, we have done all the tests on a Huawei Matebook D15 from 2020.
• Performance subject to the performance of your equipment. Continuing with the previous idea, although it is true that you do not need a very powerful computer to run the tools and the LLM that they include, the response time is according to the specifications of the equipment. I can tell you that, although our humble Huawei was able to handle them, the time to obtain responses was very long.
• We do not judge the models. With the applications that we propose here you will be able to experiment with different language models. In no case are we going to judge the responses we have received, as our intention is to teach you how to use the models without an Internet connection, not to test them. Of course, we have found some funny answers that we are going to share with you.
• Multi platform. We have followed these steps on the mentioned computer because of its modest specifications, but also because it has Windows installed and is the most popular system. All in all, you can use GPT4All and Jan in Mac and Linux environments. In operation, in all cases, it is practically identical, with the exception of the installation process, which differs depending on the operating system.
• Privacy. As long as you use a local model, not one that works through a remote API, all the queries you make will stay on your computer. That is, having a chatbot that does not need an Internet connection implies that your conversations are private and are not processed by third parties. Without a doubt, this is a very important question for some.

With these clarifications on the table, we are ready to start experimenting with the language models available for local execution. Let's go there!

How to install and use GPT4All

The first option that we recommend for use an LLM based chatbot without an internet connection is GPT4All. As mentioned in his official website, you will not need to connect to any server or have a GPU (it is understood that this refers to a dedicated one). The first thing you should do is download the most appropriate installer for your computer, depending on the operating system you have installed.

The installation process is quite simple, with the typical wizard in Windows. We have also tried running the application on a MacBook Air with M2 and it was necessary to activate Rosetta. Be that as it may, follow the steps that appear on the screen.

During installation, the program will download the necessary files for everything to work. And in reality the EXE file that you are going to download from the GPT4All website only weighs about 27 MB.

Once the tool is installed, you can now run it on your computer. After a window showing what's new in the current version, you will end up in the model discovery space. It is the site from which you will be able to download the desired LLM to start interacting with it. When you find the one you want, press Download and wait for the download to complete.

Related to this, you should know that each model includes a series of comments. For example, in Nous Hermes 2 it is explained that the responses are usually fast, that it has been trained by Mistral and that it has a commercial use license. Also shown is the total weight of the model and the amount of RAM required to run. It is interesting to note that, if the computer's memory is less than recommended, a warning will appear. This is a recommendation that, in no case, will prevent you from downloading.

OpenAI models, such as GPT-3.5, are also present. But beware! They require an API to work. That means the model runs on remote servers and not on your team. Thus, GPT4All becomes a kind of ChatGPT client. It's clearly not what we're looking for right now.

GPT4All has a very extensive model database. Without going any further, when writing GPT some proposals appear, although they are in no case the official OpenAI LLMs. The most striking thing is that many of the models you will find have been entered with very specific data sets and for specific purposes.

Since the intention is not to analyze the behavior and reliability of the models, we leave the exploration and selection of these to you. We have downloaded New Hermes 2 by Mistral, a model that, by the way, needs at least 8 GB of RAM, a figure that our test team does not reach. Once you have downloaded the models you want to use, close the browsing area and click on Choose a modelat the top of the window.

After selecting the model, start a new conversation, just as you would in ChatGPT. Make your request and wait for the response. In our case, the experience has been quite slow and cumbersome. Of course, everything will depend on the performance of your equipment.

Although GPT4All warned us of a possible lack of RAM, what has suffered the most has been the Ryzen 3500U, which has slowly but surely been getting the answers from the model.

Depending on the model you choose, you will be able to launch a specific type of task. It all depends on the data they used to train the model. In the case of Nous Hermes 2, it supports the typical requests, great for processing text, creating content or translating text.

From here, you have hours of tests ahead of you, with a Huge model selection and the possibility of adjusting its internal parameters from the settings.

How to install and use Jan

Jan is another tool that helps you install language models on your computer. You can download it from your official website, for free. At the end of the installation, you will see that a window appears very similar to that of ChatGPT. You need to click the button Explore The Hub to start obtaining the models that interest you the most.

Again, the options available are many. There is also the possibility of accessing generative AI in the cloud, such as those from OpenAI. In all cases, it is specified that they are accessed through API, and it is necessary to obtain the key before launching the model. Remember that in these cases everything is processed remotely, and not on your computer.

If you select a model that can be installed locally, such as TinyLlama Chat, once downloaded it will be available in the right sidebar, in the section model.

As you see in the following image, some models still have a long way to go, especially when we use them in Spanish.

Something interesting that Jan has that we have not been able to enjoy in GPT4All is the installation of extensions. One of the most useful is the one that adds a resource monitor to the bottom area of ​​the screen. In the following screenshot you can see that the execution of Llama2 Chat, a much heavier model than TinyLlama, has saturated our beloved Matebook. Even so, the answers have been completed in all cases.

Another striking point from Jan is that he indicates the speed of the response. In this case, the number of tokens processed per second is specified.

As can be seen from what has been analyzed, Jan is an excellent application for installing all types of models. Obviously, I'm sure you can't escape the fact that we're talking about a interface that provides access to LLM models, just like GPT4All does. All in all, it has a very friendly interface and a series of functionalities that facilitate the use of chatbots.

Additionally, Jan has another capacity that we do not find in GPT4All. And it includes a API equivalent to GPT that runs on your computerin localhost. This way, applications that ask you for an OpenAI API key to use ChatGPT will be able to connect to Jan. This is an easy way to avoid connecting to remote OpenAI servers, improving your privacy even when using applications and services. from third parties.

If you are interested in using Jan's API and replacing OpenAI's, you have all the information in this page

A world of language models at your fingertips

The language models that give life to what is known as generative AI have hundreds of applications. Little by little, large companies are integrating this technology into their products, giving meaning to the capabilities of services like ChatGPT. The two proposals we are talking about here go a little further, favoring a more private execution of LLMseliminating the need to consume remote resources and turning virtually any PC into an artificial intelligence platform.

End of Article. Tell us something in the Comments!