MLServices

Deploy a new online inference service

Changed on
post/api/v1/mlservices

Request

  • Base URL: / (relative, and the document was published on a file host, not at the API’s origin)
  • URL: /api/v1/mlservices
  • Auth: none declared

Headers

X-Axisml-Tenantstring

Active tenant for the request (fallback for the axisml.tenant cookie).

Request body

argsstring[]

Container args override.

commandstring[]

Container entrypoint override.

descriptionstring

Free-text service description.

displayNamestring

Human-readable service label.

envFromobject[]

Environment sources injected into the serving pods (pass-through to the K8s EnvFromSource shape, including configMapRef).

imagestring required

Serving container image reference.

modelNamestring required

Model artifact definition name to serve.

modelVersionstring required

Model artifact version to serve.

namestring required

Service name (unique within the tenant).

poolNamestring required

Resource pool to schedule the service onto.

replicasinteger required

Desired replica count.

unitNamestring required

Resource unit (shape) within the pool.

volumeMountsobject[]

Container volume mounts (pass-through to the K8s VolumeMount shape).

volumesobject[]

Pod volumes (pass-through to the K8s PodSpec volumes shape, including configMap).

Example request

{
  "backend": {
    "engine": "llminference",
    "name": "kserve"
  },
  "configMaps": [
    {
      "data": {
        "server.yaml": "maxTokens: 4096\n"
      },
      "name": "llama3-serving-config"
    }
  ],
  "description": "Llama3-8B online inference service.",
  "displayName": "Llama3 chat service",
  "env": [
    {
      "name": "MAX_TOKENS",
      "value": "4096"
    }
  ],
  "envFrom": [
    {
      "configMapRef": {
        "name": "llama3-serving-config"
      }
    }
  ],
  "image": "registry.axisml.io/serving/vllm:0.6.0",
  "modelName": "llama3-8b",
  "modelVersion": "1.2.0",
  "name": "llama3-chat",
  "poolName": "gpu-a100",
  "ports": [
    {
      "name": "http",
      "port": 8080
    }
  ],
  "replicas": 3,
  "route": {
    "enabled": true,
    "path": "/v1/models/llama3-8b"
  },
  "unitName": "a100-1x",
  "volumeMounts": [
    {
      "mountPath": "/etc/axisml",
      "name": "config",
      "readOnly": true
    }
  ],
  "volumes": [
    {
      "configMap": {
        "name": "llama3-serving-config"
      },
      "name": "config"
    }
  ]
}

Response

Service created.

accessUrlstring

Resolved URL clients use to reach the service.

admissionMessagestring

Human-readable detail for the current admission wait.

admissionReasonstring

Stable reason why desired replicas are still waiting for admission.

admittedReplicasinteger

Replicas holding durable capacity and tenant-quota admission.

argsstring[]

Container args override.

commandstring[]

Container entrypoint override.

computeNamespacestring

Underlying compute (Kubernetes) namespace running the service.

createdAtstring date-time required

Time the service was created.

descriptionstring

Free-text service description.

desiredState'Running' | 'Stopped'
displayNamestring

Human-readable service label.

envFromobject[]

Environment sources injected into the serving pods (pass-through to the K8s EnvFromSource shape, including configMapRef).

idstring uuid required

Stable service identifier.

imagestring

Serving container image reference.

messagestring

Human-readable status detail for the current phase.

modelNamestring

Model artifact definition name being served.

modelVersionstring

Model artifact version being served.

namestring required

Service name (unique within the tenant).

namespacestring required

Platform tenant namespace the service belongs to.

ownerstring required

Username of the service owner.

ownerIdstring uuid

User ID of the service owner.

phase'Queued' | 'Creating' | 'Pending' | 'Ready' | 'Degraded' | 'Failed' | 'Stopped' | 'Deleting' | 'Deleted'
poolNamestring

Resource pool the service is scheduled onto.

readyReplicasinteger

Replicas that have passed readiness.

replicasinteger

Desired replica count.

resourcesResourceMap

Kubernetes-style resource quantity map (e.g., {"cpu": "100", "memory": "1Ti", "nvidia.com/gpu": "8"}).

tenantDisplayNamestring

Human-readable tenant name.

tenantNamestring required

Tenant identifier owning the service.

unitNamestring

Resource unit (shape) within the pool.

updatedAtstring date-time required

Time the service was last updated.

volumeMountsobject[]

Container volume mounts (pass-through to the K8s VolumeMount shape).

volumesobject[]

Pod volumes (pass-through to the K8s PodSpec volumes shape, including configMap).

Example response

{
  "accessUrl": "https://gateway.axisml.io/v1/models/llama3-8b",
  "args": [
    "--model",
    "meta-llama/Llama-3-8b",
    "--max-model-len",
    "8192"
  ],
  "backend": {
    "engine": "llminference",
    "name": "kserve"
  },
  "command": [
    "python",
    "-m",
    "vllm.entrypoints.openai.api_server"
  ],
  "computeNamespace": "axisml-team-nlp",
  "configMaps": [
    {
      "data": {
        "server.yaml": "maxTokens: 4096\n"
      },
      "name": "llama3-serving-config"
    }
  ],
  "createdAt": "2026-06-20T08:00:00Z",
  "description": "Llama3-8B online inference service.",
  "desiredState": "Running",
  "displayName": "Llama3 chat service",
  "env": [
    {
      "name": "MAX_TOKENS",
      "value": "4096"
    }
  ],
  "envFrom": [
    {
      "configMapRef": {
        "name": "llama3-serving-config"
      }
    }
  ],
  "id": "5d2c9b41-3e8f-4a1c-9d7e-6b4f2a1c8e90",
  "image": "registry.axisml.io/serving/vllm:0.6.0",
  "message": "All replicas ready.",
  "modelName": "llama3-8b",
  "modelVersion": "1.2.0",
  "name": "llama3-chat",
  "namespace": "team-nlp",
  "owner": "zhang.san",
  "ownerId": "9f8e7d6c-5b4a-3210-fedc-ba9876543210",
  "phase": "Ready",
  "poolName": "gpu-a100",
  "ports": [
    {
      "name": "http",
      "port": 8080
    }
  ],
  "readyReplicas": 3,
  "replicas": 3,
  "resources": {
    "cpu": "8",
    "memory": "64Gi",
    "nvidia.com/gpu": "1"
  },
  "route": {
    "enabled": true,
    "path": "/v1/models/llama3-8b"
  },
  "tenantDisplayName": "Vision Team",
  "tenantName": "team-nlp",
  "unitName": "a100-1x",
  "updatedAt": "2026-06-28T09:30:00Z",
  "volumeMounts": [
    {
      "mountPath": "/etc/axisml",
      "name": "config",
      "readOnly": true
    }
  ],
  "volumes": [
    {
      "configMap": {
        "name": "llama3-serving-config"
      },
      "name": "config"
    }
  ]
}

Changes

  • c37335303c98Revision changes
    • warning

      added the new Queued enum value to the //// response property for the response status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

  • d00d802bd072Revision changes
    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the new optional request property

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status

    • info

      added the optional property to the response with the status