--- title: "Create a new spider" method: POST path: "/spiders" tags: ["Spiders"] --- # Create a new spider `POST /spiders` ## Request body - object - `name` string, required - `description` string - `project_id` string, required — MongoDB ObjectId represented as a hexadecimal string - `cmd` string - `priority` integer - `col_id` string — MongoDB ObjectId represented as a hexadecimal string - `git_id` string — MongoDB ObjectId represented as a hexadecimal string ## Response `200` Spider created successfully - object - `code` integer - `message` string - `data` object - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `name` string - `description` string - `cmd` string — Command to run the spider - `param` string — Default task parameters - `priority` integer - `project_id` string — MongoDB ObjectId represented as a hexadecimal string - `col_id` string — MongoDB ObjectId represented as a hexadecimal string - `col_name` string — Data collection name - `db_name` string — Database name - `data_source_id` string — MongoDB ObjectId represented as a hexadecimal string - `data_source` Database - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `name` string - `description` string - `data_source` string — Type of database (mongodb, mysql, etc.) - `host` string - `port` integer - `uri` string - `database` string - `username` string - `password` string - `status` string - `error` string - `active` boolean - `active_at` string, date-time - `is_default` boolean - `mongo_params` object - `auth_source` string - `auth_mechanism` string - `postgres_params` object - `ssl_mode` string - `snowflake_params` object - `account` string - `schema` string - `warehouse` string - `role` string - `cassandra_params` object - `keyspace` string - `hive_params` object - `auth` string - `redis_params` object - `db` integer - `created_at` string, date-time - `updated_at` string, date-time - `created_by` string — MongoDB ObjectId represented as a hexadecimal string - `updated_by` string — MongoDB ObjectId represented as a hexadecimal string - `mode` string — Default task mode - `node_ids` ObjectId[] — Default node IDs for tasks - `git_id` string — MongoDB ObjectId represented as a hexadecimal string - `git_root_path` string — Root path in the Git repository - `git` Git - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `url` string — Git repository URL - `name` string — Git repository name - `auth_type` string — Authentication type (e.g., http, ssh) - `username` string — Git username for authentication - `password` string — Git password or token for authentication - `current_branch` string — Current branch of the repository - `status` string — Status of the repository (e.g., cloning, pulling, error) - `error` string — Error message if any - `refs` GitRef[] — Git references (branches, tags) - `name` string — Reference name - `type` string — Reference type (branch, tag) - `hash` string — Git commit hash - `refs_updated_at` string, date-time — When refs were last updated - `clone_logs` string[] — Logs from the clone operation - `auto_pull` boolean — Whether to automatically pull before running tasks - `created_at` string, date-time - `updated_at` string, date-time - `created_by` string — MongoDB ObjectId represented as a hexadecimal string - `updated_by` string — MongoDB ObjectId represented as a hexadecimal string - `stat` SpiderStat - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `last_task_id` string — MongoDB ObjectId represented as a hexadecimal string - `last_task` Task - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `spider_id` string — MongoDB ObjectId represented as a hexadecimal string - `status` 'pending' | 'running' | 'finished' | 'error' | 'cancelled' - `node_id` string — MongoDB ObjectId represented as a hexadecimal string - `cmd` string — Command to run the task - `param` string — Task parameters - `error` string — Error message if task failed - `pid` integer — Process ID of the running task - `schedule_id` string — MongoDB ObjectId represented as a hexadecimal string - `type` string — Type of task - `mode` string — Execution mode - `priority` integer — Task priority - `node_ids` ObjectId[] — List of node IDs for task execution - `stat` TaskStat - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `start_ts` string, date-time — When the task started - `end_ts` string, date-time — When the task ended - `wait_duration` integer — Wait duration in milliseconds - `runtime_duration` integer — Runtime duration in milliseconds - `total_duration` integer — Total duration in milliseconds - `result_count` integer — Number of results from the task - `created_at` string, date-time - `updated_at` string, date-time - `spider` Spider - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `name` string - `description` string - `cmd` string — Command to run the spider - `param` string — Default task parameters - `priority` integer - `project_id` string — MongoDB ObjectId represented as a hexadecimal string - `col_id` string — MongoDB ObjectId represented as a hexadecimal string - `col_name` string — Data collection name - `db_name` string — Database name - `data_source_id` string — MongoDB ObjectId represented as a hexadecimal string - `data_source` Database - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `name` string - `description` string - `data_source` string — Type of database (mongodb, mysql, etc.) - `host` string - `port` integer - `uri` string - `database` string - `username` string - `password` string - `status` string - `error` string - `active` boolean - `active_at` string, date-time - `is_default` boolean - `mongo_params` object - … - `postgres_params` object - … - `snowflake_params` object - … - `cassandra_params` object - … - `hive_params` object - … - `redis_params` object - … - `created_at` string, date-time - `updated_at` string, date-time - `created_by` string — MongoDB ObjectId represented as a hexadecimal string - `updated_by` string — MongoDB ObjectId represented as a hexadecimal string - `mode` string — Default task mode - `node_ids` ObjectId[] — Default node IDs for tasks - `git_id` string — MongoDB ObjectId represented as a hexadecimal string - `git_root_path` string — Root path in the Git repository - `git` Git - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `url` string — Git repository URL - `name` string — Git repository name - `auth_type` string — Authentication type (e.g., http, ssh) - `username` string — Git username for authentication - `password` string — Git password or token for authentication - `current_branch` string — Current branch of the repository - `status` string — Status of the repository (e.g., cloning, pulling, error) - `error` string — Error message if any - `refs` GitRef[] — Git references (branches, tags) - … - `refs_updated_at` string, date-time — When refs were last updated - `clone_logs` string[] — Logs from the clone operation - `auto_pull` boolean — Whether to automatically pull before running tasks - `created_at` string, date-time - `updated_at` string, date-time - `created_by` string — MongoDB ObjectId represented as a hexadecimal string - `updated_by` string — MongoDB ObjectId represented as a hexadecimal string - `stat` SpiderStat — recursive - `created_at` string, date-time - `updated_at` string, date-time - `created_by` string — MongoDB ObjectId represented as a hexadecimal string - `updated_by` string — MongoDB ObjectId represented as a hexadecimal string - `schedule` Schedule - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `name` string - `spider_id` string — MongoDB ObjectId represented as a hexadecimal string - `cron` string — Cron expression for schedule - `param` object — Schedule parameters - `enabled` boolean - `created_at` string, date-time - `updated_at` string, date-time - `created_by` string — MongoDB ObjectId represented as a hexadecimal string - `updated_by` string — MongoDB ObjectId represented as a hexadecimal string - `node` Node - `_id` string — MongoDB ObjectId represented as a hexadecimal string - `key` string — Unique key for the node - `name` string - `ip` string - `mac` string - `hostname` string - `description` string - `is_master` boolean — Whether this node is the master node - `status` string - `enabled` boolean - `active` boolean — Whether this node is currently active - `active_at` string, date-time — When this node was last active - `current_runners` integer — Number of tasks currently running on this node - `max_runners` integer — Maximum number of tasks that can run on this node - `created_at` string, date-time - `updated_at` string, date-time - `result_count` integer - `error_count` integer - `error_message` string - `created_at` string, date-time - `updated_at` string, date-time - `created_by` string — MongoDB ObjectId represented as a hexadecimal string - `updated_by` string — MongoDB ObjectId represented as a hexadecimal string - `tasks` integer — Total number of tasks for this spider - `results` integer — Total number of results from all tasks - `wait_duration` integer — Total wait duration in seconds - `runtime_duration` integer — Total runtime duration in seconds - `total_duration` integer — Total duration in seconds - `average_wait_duration` integer — Average wait duration in seconds - `average_runtime_duration` integer — Average runtime duration in seconds - `average_total_duration` integer — Average total duration in seconds - `created_at` string, date-time - `updated_at` string, date-time - `created_at` string, date-time - `updated_at` string, date-time - `created_by` string — MongoDB ObjectId represented as a hexadecimal string - `updated_by` string — MongoDB ObjectId represented as a hexadecimal string ## Changes - **2025-03-03** `14e4eafea538` — 1 breaking, 3 warning, 1 info - the response's body type/format changed from `object`/`` to ``/`` for status `200` - removed the optional property `code` from the response with the `200` status - removed the optional property `data` from the response with the `200` status - removed the optional property `message` from the response with the `200` status - …1 more - **2025-03-02** `526e5bdf6dee` — 3 breaking, 25 warning, 10 info - added the new required request property `name` - added the new required request property `project_id` - request body became required - removed the request property `data` - …34 more [Change history](https://skmtc.dev/crawlab-team/apis/crawlab-api/changes/spiders/post.md) --- [API](https://skmtc.dev/crawlab-team/apis/crawlab-api.md) · [All operations](https://skmtc.dev/crawlab-team/apis/crawlab-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/crawlab-team/crawlab-api/revisions/14e4eafea538/schema)