--- title: "Create a model" method: POST path: "/customers/{CUSTOMER_ID}/ai/models" tags: ["Manage models"] --- # Create a model `POST /customers/{CUSTOMER_ID}/ai/models` Create a custom model and start a training job. The custom model cannot be modified after it is created. ## Request body - PostModels — POST https://api.lucidworks.com/customers/{CUSTOMER_ID}/ai/models - `name` string — The user-friendly name of the model. - `modelType` string — The name of the custom model. - `region` string — The geographic region specified when the custom model is being trained. - `trainingData` TrainingData — Both the `catalog index file` and `signals query file` are required, and must contain the same `pkid` field, which links relevant queries to documents. - `catalog` string, required — The location of the catalog of the training data in Google Cloud Storage (GCS). The catalog file contains documents (products) that will be searched. The file must have a `pkid` (product key ID) column which contains the document ID or product ID. The `pkid` is a unique value for each document, so entries with a duplicate `pkid` are filtered out. However, since not every `pkid` entry is associated with a query, there may be entries in the `catalog index file` that are not associated with a `signals query entry`. The index file content format is different based on the model type to be trained. For example, a general model or an eCommerce model. - The general index file format contains: - `pkid` - The unique product key ID. Required field. This must match an entry in the `signals query file`. - `text` - A freeform text field. - The eCommerce index file format contains: - `pkid` - The unique product key ID. Required field. This must match an entry in the `signals query file`. - `name` - The freeform text field that contains the product name. - `signals` string, required — The location of signals in the training data in Google Cloud Storage (GCS). The signals file must have a `pkid` (product key ID) column which refers to the relevant document or product ID. The file may contain multiple duplicates of any `pkid` because each document could be associated with several relevant queries. NOTE: For evaluation purposes, 10% of unique queries (50 minimum and 5000 maximum) are automatically sampled into a validation set from the training query file. * The general query file format contains: - `pkid` - The unique product key ID. Required field. This must match an entry in the `catalog index file`. - `query` - A freeform text field. - The eCommerce query file format contains: - `pkid` - The unique product key ID. Required field. This must match an entry in the `catalog index file`. - `query` - A freeform text field. - `aggr_count` - The number of documents that match the query criteria. In most cases, this value is used as a weight and must be greater than zero (0). If you do not use weights or there is no value, set this value to `1`. The weight is used for training pairs sampling and to compute normalized discounted cumulative gain (NDCG) metrics. If all values are `1.0`, binary NDCG is computed. - `config` ModelConfig - `dataset_config` string — The options for the dataset format used for training are: - `mlp_general` - This is used for the general recurrent neural networks (RNN) model type. - `mlp_ecommerce` - This used for an eCommerce RNN model type. - `trainer_config` string — The options for the trainer type used for training are: - `mlp_general` - This is used for the general recurrent neural networks (RNN) model type. - `mlp_ecommerce` - This used for an eCommerce RNN model type. - `trainer_config/text_processor_config` string — This determines which type of tokenization and embedding is used as the base for the recurrent neural network (RNN) model. For example, word or bype-pair encoding (BPE). The word text processor defaults to English, and uses word-based tokenization and English pre-trained word embeddings. The maximum word vocabulary result is 100000. The BPE versions use the same tokenization, but different vocabulary sizes: * bpe_*_small embeddings have 10000 vocabulary tokens * bpe_*_large embeddings have 100000 vocabulary tokens * bpe_multi multilingual embeddings have 320000 vocabulary tokens The options for text processors are: - English - `word_en` (default) - `bpe_en_small` - `bpe_en_large` - `all_minilm_l6` - `e5_small_v2` - `e5_base_v2` - `e5_large_v2` - `gte_small` - `gte_base` - `gte_large` - `snowflake_arctic_embed_xs` - Multilingual - `bpe_multi` - `multilingual_e5_small` - `multilingual_e5_base` - `multilingual_e5_large` - Bulgarian - `bpe_bg_small` - `bpe_bg_large` - German - `bpe_de_small` - `bpe_de_large` - Spanish - `bpe_es_small` - `bpe_es_large` - French - `bpe_fr_small` - `bpe_fr_large` - Italian - `bpe_it_small` - `bpe_it_large` - Japanese - `bpe_ja_small` - `bpe_ja_large` - Korean - `bpe_ko_small` - `bpe_ko_large` - Dutch - `bpe_nl_small` - `bpe_nl_large` - Romanian - `bpe_ro_small` - `bpe_ro_large` - Chinese - `bpe_zh_small` - `bpe_zh_large` - Custom - `word_custom` - `bpe_custom` - `trainer_config.encoder_config.rnn_names_list` string[] — This determines which bi-directional recurrent neural network (RNN) layers are used. Options include `gru` and `lstm`. - `trainer_config.encoder_config.rnn_units_list` integer[] — The number of units for each recurrent neural network (RNN) layer. IMPORTANT: You must specify the same number of units for `trainer_config.encoder_config.rnn_units_list` and its similarly-named `trainer_config.encoder_config.rnn_names_list` RNN layer. For example, `rnn_units_list` needs to be the same size as `rnn_names_list`. Because this is a bi-directional RNN, the encoder's vector size is two times larger than the number of units in the last layer. For example, if one layer is 128 units, the output vector size is 256. - `trainer_config.trn_batch_size` integer — The batch size to be used for a single model training update. By default, an appropriate batch size is automatically determined based on the dataset size. If the field is set to `null`, the batch size is also automatically determined based on the dataset size. - `trainer_config.num_epochs` integer — The number of epochs the training data must complete. An epoch is a full cycle where training data passes through the designated algorithms. During one epoch, the model processes all the training data examples (queries and index documents) at least one time. - `trainer_config.monitor_patience` integer — The number of epochs the training passes before it stops if there is no validation metric improvement during the epochs. The best model state based on the monitor validation metric is used as the final model. * For the general RNN, the `mrr@3` metric is monitored and the `monitor_patience` default value is 8. * For the eCommerce RNN, the `ndcg@5` metric is monitored and the `monitor_patience` default value is 16. - `trainer_config.encoder_config.emb_spdp` number, float — This field provides a regularization effect, which is the process to simplify result answers. The regularization is applied between the token embeddings layer and the first recurrent neural network (RNN) layer. - `trainer_config.encoder_config.emb_trainable` boolean — This field determines if fine-tuning of the token embeddings, such as word or byte pair encoding (BPE) token vectors, is enabled. If set, it can improve the quality of the model if the query contains less natural language, and training is negatively affected. Because the embeddings layer is the largest layer in the network, the process to improve the model requires enough training data to prevent overfitting. - `trainingDataCredentials` object — The credentials used to access the training data URLs. - `serviceAccountKey` string — The Base64-encoded service account JSON key. ## Response `200` OK - ResponsePostModels - `id` string — The identifier of the model. The value is the universally unique identified (UUID) that is the primary key for the model. - `name` string — The user-friendly name of the model. - `modelType` string — The name of the model. - `category` string — The object that specifies the model is `custom`. - `description` string — The description of the model. - `region` string — The geographic region specified when the custom model was trained. - `trainingData` TrainingData — Both the `catalog index file` and `signals query file` are required, and must contain the same `pkid` field, which links relevant queries to documents. - `catalog` string, required — The location of the catalog of the training data in Google Cloud Storage (GCS). The catalog file contains documents (products) that will be searched. The file must have a `pkid` (product key ID) column which contains the document ID or product ID. The `pkid` is a unique value for each document, so entries with a duplicate `pkid` are filtered out. However, since not every `pkid` entry is associated with a query, there may be entries in the `catalog index file` that are not associated with a `signals query entry`. The index file content format is different based on the model type to be trained. For example, a general model or an eCommerce model. - The general index file format contains: - `pkid` - The unique product key ID. Required field. This must match an entry in the `signals query file`. - `text` - A freeform text field. - The eCommerce index file format contains: - `pkid` - The unique product key ID. Required field. This must match an entry in the `signals query file`. - `name` - The freeform text field that contains the product name. - `signals` string, required — The location of signals in the training data in Google Cloud Storage (GCS). The signals file must have a `pkid` (product key ID) column which refers to the relevant document or product ID. The file may contain multiple duplicates of any `pkid` because each document could be associated with several relevant queries. NOTE: For evaluation purposes, 10% of unique queries (50 minimum and 5000 maximum) are automatically sampled into a validation set from the training query file. * The general query file format contains: - `pkid` - The unique product key ID. Required field. This must match an entry in the `catalog index file`. - `query` - A freeform text field. - The eCommerce query file format contains: - `pkid` - The unique product key ID. Required field. This must match an entry in the `catalog index file`. - `query` - A freeform text field. - `aggr_count` - The number of documents that match the query criteria. In most cases, this value is used as a weight and must be greater than zero (0). If you do not use weights or there is no value, set this value to `1`. The weight is used for training pairs sampling and to compute normalized discounted cumulative gain (NDCG) metrics. If all values are `1.0`, binary NDCG is computed. - `config` ModelConfig - `dataset_config` string — The options for the dataset format used for training are: - `mlp_general` - This is used for the general recurrent neural networks (RNN) model type. - `mlp_ecommerce` - This used for an eCommerce RNN model type. - `trainer_config` string — The options for the trainer type used for training are: - `mlp_general` - This is used for the general recurrent neural networks (RNN) model type. - `mlp_ecommerce` - This used for an eCommerce RNN model type. - `trainer_config/text_processor_config` string — This determines which type of tokenization and embedding is used as the base for the recurrent neural network (RNN) model. For example, word or bype-pair encoding (BPE). The word text processor defaults to English, and uses word-based tokenization and English pre-trained word embeddings. The maximum word vocabulary result is 100000. The BPE versions use the same tokenization, but different vocabulary sizes: * bpe_*_small embeddings have 10000 vocabulary tokens * bpe_*_large embeddings have 100000 vocabulary tokens * bpe_multi multilingual embeddings have 320000 vocabulary tokens The options for text processors are: - English - `word_en` (default) - `bpe_en_small` - `bpe_en_large` - `all_minilm_l6` - `e5_small_v2` - `e5_base_v2` - `e5_large_v2` - `gte_small` - `gte_base` - `gte_large` - `snowflake_arctic_embed_xs` - Multilingual - `bpe_multi` - `multilingual_e5_small` - `multilingual_e5_base` - `multilingual_e5_large` - Bulgarian - `bpe_bg_small` - `bpe_bg_large` - German - `bpe_de_small` - `bpe_de_large` - Spanish - `bpe_es_small` - `bpe_es_large` - French - `bpe_fr_small` - `bpe_fr_large` - Italian - `bpe_it_small` - `bpe_it_large` - Japanese - `bpe_ja_small` - `bpe_ja_large` - Korean - `bpe_ko_small` - `bpe_ko_large` - Dutch - `bpe_nl_small` - `bpe_nl_large` - Romanian - `bpe_ro_small` - `bpe_ro_large` - Chinese - `bpe_zh_small` - `bpe_zh_large` - Custom - `word_custom` - `bpe_custom` - `trainer_config.encoder_config.rnn_names_list` string[] — This determines which bi-directional recurrent neural network (RNN) layers are used. Options include `gru` and `lstm`. - `trainer_config.encoder_config.rnn_units_list` integer[] — The number of units for each recurrent neural network (RNN) layer. IMPORTANT: You must specify the same number of units for `trainer_config.encoder_config.rnn_units_list` and its similarly-named `trainer_config.encoder_config.rnn_names_list` RNN layer. For example, `rnn_units_list` needs to be the same size as `rnn_names_list`. Because this is a bi-directional RNN, the encoder's vector size is two times larger than the number of units in the last layer. For example, if one layer is 128 units, the output vector size is 256. - `trainer_config.trn_batch_size` integer — The batch size to be used for a single model training update. By default, an appropriate batch size is automatically determined based on the dataset size. If the field is set to `null`, the batch size is also automatically determined based on the dataset size. - `trainer_config.num_epochs` integer — The number of epochs the training data must complete. An epoch is a full cycle where training data passes through the designated algorithms. During one epoch, the model processes all the training data examples (queries and index documents) at least one time. - `trainer_config.monitor_patience` integer — The number of epochs the training passes before it stops if there is no validation metric improvement during the epochs. The best model state based on the monitor validation metric is used as the final model. * For the general RNN, the `mrr@3` metric is monitored and the `monitor_patience` default value is 8. * For the eCommerce RNN, the `ndcg@5` metric is monitored and the `monitor_patience` default value is 16. - `trainer_config.encoder_config.emb_spdp` number, float — This field provides a regularization effect, which is the process to simplify result answers. The regularization is applied between the token embeddings layer and the first recurrent neural network (RNN) layer. - `trainer_config.encoder_config.emb_trainable` boolean — This field determines if fine-tuning of the token embeddings, such as word or byte pair encoding (BPE) token vectors, is enabled. If set, it can improve the quality of the model if the query contains less natural language, and training is negatively affected. Because the embeddings layer is the largest layer in the network, the process to improve the model requires enough training data to prevent overfitting. - `state` string — This field specifies the current status of the custom model. The value is `TRAINING`. - `trainingStarted` string — The date and time the training started. This field only applies to custom models. - `trainingCompleted` string — The date and time the training completed. This field only applies to custom models. - `createdBy` string — The user who created the model. --- [API](https://skmtc.dev/lucidworks/apis/rules-rewrites-api.md) · [All operations](https://skmtc.dev/lucidworks/apis/rules-rewrites-api/llms.txt) · [OpenAPI document](https://skmtc-service-production.skmtc.workers.dev/v1/apis/lucidworks/rules-rewrites-api/revisions/f2d3747848e8/schema)