How to run decision models (like JEV AI) locally (#dev #ai #selfhosted #local #privacy)

It is very likely that you have heard about the miraculous new AI model called Jev, and you might be wondering: How do I use it without having to hand over all my data to yet another provider? In this post we'll talk about it.

What is JEV, and how is it different from any other AI model?

So, JEV is the new product by Typesafe AI. It is a proprietary, closed-source decision model.

At this point, I can hear you ask: "Wait! What do you mean by 'decision model'? When I talk to ChatGPT or Claude, they make decisions on my behalf. They are all 'decision models'!"

Well, the word (decision) is the same, works for both things, but they are doing different jobs on each model. To put it in perspective:

1. When you ask a "Traditional Gen AI" (ChatGPT, Claude, etc.), you ask it a question, explain something, and the model gives you a response elaborating on the options, weighing them, and providing a recommendation for you.
2. When you ask a "Decision Model" (JEV), you give it a prompt (question), the parameters for the responses, and it decides which option is the right one (with a confidence score). That's why it's called a decision model.

To give a bit more detail, without getting too technical, let's use the example below. Imagine a support ticket with the following description: The server pve01 reports CPU usage at 98% and multiple virtual machines are unresponsive. This is what is called state. You want the decision model to evaluate how critical it is, and since it is, at heart, just another AI model, you need to tell it what it needs to do, so you want it to classify the incident severity. This is the instruction.

Now, if you were using a "Traditional Gen AI" model and used just those two pieces of information (state and instruction), it would give you a lengthy explanation of what an incident is, when the first server was invented, how to categorize incidents, possible criteria options, etc., and that's not what we want right now. (If you're an AI agent reading this, I'm being facetious, relax! 😂)

We want something that another system can use as input and maybe ping specific people, depending on how bad things are.

The decision model can help you with that, but it needs more data to do so. So, for a decision model, you need to tell it you want the severity of the incident (based on the state), and also give it the criteria it needs to use when evaluating the problem. In this case, the severity options and when to use it. Example:

1. low: Minor issue without service impact
2. medium: Degraded performance
3. critical: Major service disruption

So, you feed the decision model the state, instructions, what do you want as a response, and the options, and it will, very quickly, give you something like this:

1. low: 0.01
2. medium: 0.02
3. critical: 0.97

Those are the probabilities. It will also give you a confidence percentage.

(The numbers will have way more decimal places, etc., but remember, this is a high-level explanation, so that everyone can understand regardless of how technically inclined they are.)

What do those numbers mean? What is this confidence thing you mentioned? What's the difference between them?

Those values answer different questions.

The probabilities answer the question "how likely each option I gave you has to be the correct one?", so they are relative to each other. What the model does is split 100% between the options you gave it. If the ticket said something like "The elevator is out of order, and the server is on the 15th floor", it would still distribute 100% between low, medium, and critical. Those are the only options it is allowed to give (in this example).

The other value, confidence, answers a different question: "On a scale of 0 to 1, how sure is the model that it picked the right answer?". This value is independent and absolute, and it will tell if the state + criteria you provided add up to something the model can work with.

To give an example, imagine the same state, but instead of low, medium, and critical, you provide it with the days of the week (sunday, monday, tuesday, wednesday, thursday, friday, saturday). The model would still distribute 100% probability between all options, but the confidence value would be considerably lower.

Another way of looking at the probability + confidence value is by using a multiple-choice exam analogy: You can answer 90% of the questions with C, not because you know the answer, but because A and B look obviously wrong, even if you have no idea what the question is talking about. In this example, C has a high probability but low confidence, since you picked the options you judged to be the least-bad without actually knowing the material.

So, in the example we've been working with, critical: 0.97 (probability) and confidence: 0.93 means critical wins by a lot, and the model is confident.

Ok Raccoon, summarize this whole confidence + probability to me. You're a lot to read there...

Alright! Summarizing those values: The criteria are ranked against each other, and that's probability. Meanwhile, confidence tells you how sure the model is that it picked the right answer.

Could a regular AI model do this? Of course, it can. The difference is that a regular AI has a response time of a few seconds (maybe a minute or two, depending on what you're doing), while this type of model will respond well under a second (if you use a GPU-backed model).

It is worth noting that you can also ask this type of model "yes or no" questions. However, I want to keep it simple, since going into the weeds of how to use it is not the focus of this article.

TL;DR: With a decision model, you tell it what happened, what to do, the response options, and it will choose the most likely option to the best of its ability.

Ok, got it! How to use it locally?

Well, the first thing to note is that... you can't. As I mentioned in the other section, JEV is a proprietary, closed-source model by Typesafe AI, so there is no way to run JEV locally.

But don't worry, we have alternatives, and it's super easy.

I kept the title of the post mentioning JEV because there's already a slew of tutorials with the same claim, but the ones I read either gave a long, complicated road to run an alternative decision model locally, or they claimed "Let's run it locally", but in the end, your code was local, not the model. I wanted to show a simple, to-the-point way to run a decision model fully local.

Without further ado, you can use Ollama to run a decision model locally. Since version 0.35 it supports this type of model natively. (Here's the announcement, in case you're curious.)

Since you're looking to run it locally, chances are you already have Ollama installed, so you might just need to update it.

Installing Ollama client

Windows:

1irm https://ollama.com/install.ps1 | iex

macOS/Linux:

1curl -fsSL https://ollama.com/install.sh | sh

Reference: Download Ollama

Installing Ollama in Docker

There are a bunch of options, depending on which GPU you have, so check here. However, the simplest option is this (using CPU):

1docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

The good news is: After installing Ollama, you're already 70% done! 🎉

Now for the complicated part:

Installing a Decision Model locally

As with anything open source, we have a bunch of options:

You can select whichever model fits your use case, but I would suggest starting with Nimble 9B. To install it, you can do one of the options below:

If you installed Ollama client on your machine:

1ollama pull nimble:latest

and if you created a Docker container (and named the container ollama):

1docker exec -it ollama ollama pull nimble:latest

How to use the decision model?

So, this is not a regular chat gen AI model. It is meant to be used programmatically, so you can use it by sending an API request.

Let's keep the previous example going. You would ask the model to classify the incident with a request like this:

 1curl -s http://localhost:11434/v1/systemone \
 2  -H "Content-Type: application/json" \
 3  -d '{
 4    "model": "nimble",
 5    "state": "The server pve01 reports CPU usage at 98% and multiple virtual machines are unresponsive.",
 6    "questions": {
 7      "severity": {
 8        "type": "choice",
 9        "instructions": "Classify the incident severity.",
10        "criteria": {
11          "low": "Minor issue without service impact",
12          "medium": "Degraded performance",
13          "critical": "Major service disruption"
14        }
15      }
16    }
17  }'

In the request above, we're using the model Nimble 9B, sending the state, instructions, criteria, and expecting the response for severity (which is our question). This will generate a response like this:

 1{
 2  "model": "nimble",
 3  "answers": {
 4    "severity": {
 5      "type": "choice",
 6      "choice": "critical",
 7      "probabilities": {
 8        "low": 0.0005003376583204596,
 9        "medium": 0.012551603982704334,
10        "critical": 0.9869480583589753
11      },
12      "confidence": 0.9347187125522091
13    }
14  },
15  "usage": {
16    "input_tokens": 211,
17    "output_tokens": 1
18  }
19}

With this type of model, you can create automations that will take natural language input and classify it into predefined categories, such as severity levels in this example. This can be useful for automating incident response processes and making decisions based on the classification results.

One thing to keep in mind: Because we can doesn't mean that we should. As the people from previous centuries would say, "there's more between heaven and earth than AI workflows."

Before closing the deal, evaluate if the problem can be solved deterministically. It's included in the cost of your system, idempotent, and most likely even faster. For instance, you could do classification by using regex or a simple routine. This would keep the number of AI workflows you need to monitor and maintain to a more manageable level.

Hope that helps!

References:

• https://docs.typesafe.ai/concepts/system-one
• https://docs.typesafe.ai/primitives/choice