This guide explains how to deploy Qwen3 235B on JarvisLabs and expose it as an OpenAI-compatible API endpoint.
We will deploy this model:
Qwen/Qwen3-235B-A22B-Instruct-2507-FP8
We will serve it with this API model name:
qwen3-235b-classifier
We will use:
JarvisLabs
4 x NVIDIA H200 GPUs
PyTorch template
vLLM
Hugging Face
OpenAI-compatible API
1. Create a JarvisLabs instance
Go to JarvisLabs and create a new GPU instance.
Choose:
Template: PyTorch
GPU: H200
GPU count: 4
Storage: 1000 GB
The model is large, so do not use a small disk. We used 1000 GB storage.
After the instance starts, open the terminal from Jupyter or VS Code.
2. Check GPUs
Run:
nvidia-smi
You should see 4 H200 GPUs.
You can also check from Python:
python -c "import torch; print(torch.cuda.is_available()); print(torch.cuda.device_count())"
Expected output:
True
4
Run:
pip install -U pip
pip install -U vllm huggingface_hub openai
Check vLLM:
vllm --version
4. Login to Hugging Face
The model will download from Hugging Face, so login first.
Run:
hf auth login
Paste your Hugging Face token.
You can create a token from:
Hugging Face account
Settings
Access Tokens
New token
Use a read token.
Check login:
hf auth whoami
5. Set Hugging Face cache folder
The model is very large, so store it in the workspace disk.
Run:
mkdir -p /workspace/hf-cache
export HF_HOME=/workspace/hf-cache
export HUGGINGFACE_HUB_CACHE=/workspace/hf-cache
Save these settings for future terminal sessions:
echo 'export HF_HOME=/workspace/hf-cache' >> ~/.bashrc
echo 'export HUGGINGFACE_HUB_CACHE=/workspace/hf-cache' >> ~/.bashrc
After login, the token may be saved here:
/workspace/hf-cache/token
Export it:
export HF_TOKEN="$(cat /workspace/hf-cache/token)"
export HUGGING_FACE_HUB_TOKEN="$HF_TOKEN"
Check that the token is loaded:
echo $HF_TOKEN | cut -c1-8
It should print something like:
hf_xxxxx
Do not share your full Hugging Face token.
6. Stop old vLLM process if any
Before starting the model, stop any old vLLM process:
pkill -f "vllm serve"
Check GPU memory:
nvidia-smi
If old workers are still using GPU memory, wait a few seconds and check again.
7. Clean old vLLM compile cache
This helps avoid old failed kernel cache issues.
Run:
rm -rf /home/.cache/vllm/deep_gemm
rm -rf /home/.cache/vllm/torch_compile_cache
8. Start vLLM server
Set your API key:
export VLLM_API_KEY="your-strong-secret-key"
Use a strong secret key. Do not use a simple test key in production.
Now start the model:
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \
--served-model-name qwen3-235b-classifier \
--host 0.0.0.0 \
--port 6006 \
--tensor-parallel-size 4 \
--max-model-len 16384 \
--gpu-memory-utilization 0.85 \
--dtype auto \
--trust-remote-code \
--enforce-eager \
--linear-backend triton \
--moe-backend triton \
--api-key "$VLLM_API_KEY"
9. Why these flags are important
This flag sets the API model name:
--served-model-name qwen3-235b-classifier
This lets vLLM listen outside localhost:
--host 0.0.0.0
This uses port 6006:
--port 6006
This splits the model across 4 GPUs:
--tensor-parallel-size 4
This limits max context length:
--max-model-len 16384
This keeps some free GPU memory:
--gpu-memory-utilization 0.85
This helped avoid compile issues:
--enforce-eager
These two flags were important for our H200 setup:
--linear-backend triton
--moe-backend triton
Without these, vLLM tried to use a DeepGEMM path and failed with an NVCC compile error.
10. Wait for model loading
The first start will take time.
vLLM will download and load a large model.
You may see logs like:
Loading safetensors checkpoint shards
Wait until you see:
Starting vLLM server on http://0.0.0.0:6006
Application startup complete
That means the API server is ready.
11. Test inside the JarvisLabs terminal
Run:
curl http://localhost:6006/v1/models \
-H "Authorization: Bearer your-strong-secret-key"
You should see a model list.
Example:
{
"object": "list",
"data": [
{
"id": "qwen3-235b-classifier",
"object": "model"
}
]
}
Now test chat completion:
curl http://localhost:6006/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-strong-secret-key" \
-d '{
"model": "qwen3-235b-classifier",
"messages": [
{
"role": "user",
"content": "Hello"
}
],
"temperature": 0,
"max_tokens": 100
}'
If this works locally, vLLM is running correctly.
12. Get the public API endpoint
JarvisLabs gives a notebook URL.
It may look like this:
https://your-instance-id.notebooksn.jarvislabs.net
In our setup, the working API URL was directly:
https://your-instance-id.notebooksn.jarvislabs.net/v1
Do not use:
/proxy/6006/v1
That gave a Not Found error in our setup.
The correct test from local PC was:
curl https://your-instance-id.notebooksn.jarvislabs.net/v1/models \
-H "Authorization: Bearer your-strong-secret-key"
If it returns the model list, your public endpoint is ready.
13. Use the endpoint with curl from your PC
Run this from your local laptop:
curl https://your-instance-id.notebooksn.jarvislabs.net/v1/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer your-strong-secret-key" \
-d '{
"model": "qwen3-235b-classifier",
"messages": [
{
"role": "user",
"content": "Hello"
}
],
"temperature": 0,
"max_tokens": 100
}'
14. Use the endpoint with Python
Install the OpenAI SDK on your local machine:
pip install openai
Create a Python file:
from openai import OpenAI
client = OpenAI(
base_url="https://your-instance-id.notebooksn.jarvislabs.net/v1",
api_key="your-strong-secret-key",
)
response = client.chat.completions.create(
model="qwen3-235b-classifier",
messages=[
{
"role": "user",
"content": "Hello"
}
],
temperature=0,
max_tokens=100,
)
print(response.choices[0].message.content)
Run:
python test_qwen.py
15. Use the endpoint with Node.js
Install the OpenAI SDK:
npm install openai
Create a JavaScript file:
import OpenAI from "openai";
const client = new OpenAI({
baseURL: "https://your-instance-id.notebooksn.jarvislabs.net/v1",
apiKey: "your-strong-secret-key",
});
const response = await client.chat.completions.create({
model: "qwen3-235b-classifier",
messages: [
{
role: "user",
content: "Hello",
},
],
temperature: 0,
max_tokens: 100,
});
console.log(response.choices[0].message.content);
Run:
node test-qwen.js
16. Useful debug commands
Check GPU:
nvidia-smi
Check vLLM process:
ps aux | grep vllm
Check port 6006:
ss -ltnp | grep 6006
Check local model list:
curl http://localhost:6006/v1/models \
-H "Authorization: Bearer your-strong-secret-key"
Check health:
curl http://localhost:6006/health
Check Hugging Face cache size:
du -sh /workspace/hf-cache
Check disk space:
df -h
17. Common problems and fixes
Problem: Hugging Face warning
You may see:
You are sending unauthenticated requests to the HF Hub
Fix:
hf auth login
export HF_TOKEN="$(cat /workspace/hf-cache/token)"
export HUGGING_FACE_HUB_TOKEN="$HF_TOKEN"
Problem: Connection refused
You may see:
curl: Failed to connect to localhost port 6006
This means vLLM is not ready or it crashed.
Check:
ps aux | grep vllm
nvidia-smi
Wait until you see:
Application startup complete
Problem: Not Found with proxy path
Wrong:
https://your-instance-id.notebooksn.jarvislabs.net/proxy/6006/v1/models
Correct in our setup:
https://your-instance-id.notebooksn.jarvislabs.net/v1/models
Problem: NVCC compilation failed
You may see:
NVCC compilation failed
deep_gemm.py
fp8_gemm_nt_op
Fix by using Triton backend:
--linear-backend triton
--moe-backend triton
--enforce-eager
Also clean cache before restarting:
rm -rf /home/.cache/vllm/deep_gemm
rm -rf /home/.cache/vllm/torch_compile_cache
Final working command
This is the final command that worked:
export HF_HOME=/workspace/hf-cache
export HUGGINGFACE_HUB_CACHE=/workspace/hf-cache
export HF_TOKEN="$(cat /workspace/hf-cache/token)"
export HUGGING_FACE_HUB_TOKEN="$HF_TOKEN"
export VLLM_API_KEY="your-strong-secret-key"
vllm serve Qwen/Qwen3-235B-A22B-Instruct-2507-FP8 \
--served-model-name qwen3-235b-classifier \
--host 0.0.0.0 \
--port 6006 \
--tensor-parallel-size 4 \
--max-model-len 16384 \
--gpu-memory-utilization 0.85 \
--dtype auto \
--trust-remote-code \
--enforce-eager \
--linear-backend triton \
--moe-backend triton \
--api-key "$VLLM_API_KEY"
Base URL:
https://your-instance-id.notebooksn.jarvislabs.net/v1
Model name:
qwen3-235b-classifier
API key:
your-strong-secret-key
Example endpoint:
https://your-instance-id.notebooksn.jarvislabs.net/v1/chat/completions
That is it. The model is now deployed on JarvisLabs and can be used like an OpenAI-compatible API.
