This is an introduction to the LLM parameters.

#

Inference-Parameters

#

1. Temperature(Default 1 RANGE 0-2)

   high temperature will enhance llm’s personal, less temperature will keep llm’s level

   for instance: while temperature =1.0, llm’s behavior will more brave , 0.1 will be stable

   ---

   A higher temperature setting can enhance the LLM’s creativity, while a lower temperature will maintain its consistency.

   For example: When the temperature is set to 1.0, the LLM’s responses tend to be more adventurous; at 0.1, they are more conservative.

2. Max Generation Length(Default: Model-dependent,Range 1 to model’s maximum)

   Set Max Generation Length can control llm’s input context’s length

   ---

   Setting the Maximum Generation Length can control the length of the input context for the LLM."

3. Top-K(Default 0 Range 1 to vocabulary,Common Range 40,50)

   Sampling is restricted to the top k highest-probability words. For instance, with k set to 50, the selection is made from among the 50 most likely words.

4. Top-p(Default 1 Range0-1 ,Common 0.9,0.95)

   Sampling is conducted from the pool of words until their cumulative probability reaches p. For example, with p set to 0.9, the selection is made from among the words whose combined probabilities account for 90% of the total.

5. Repetion Penalty(Default 1 Range 1.0- 2.0)

   To mitigate the likelihood of generating repetitive words, a higher repetition penalty Range can be applied, which effectively reduces the occurrence of redundant content in the text.

6. Beam Searth(Default 1,Range 1 to 10)

   Beam Search is a heuristic search algorithm that maintains multiple candidate sequences during generation, ultimately selecting the most optimal text output. Key parameters for this algorithm include the Beam Width, which determines the number of candidates retained at each step.

7. Length Penalty

   Length Penalty is a setting used in text generation to help the model create text that’s not too short or too long. It works by adjusting the score of a generated text based on its length. Example: Imagine you’re asking a computer to write a summary of a book. Without Length Penalty, the summary might be too short and miss important points, or it might be too long and include unnecessary details. By using Length Penalty, you can tell the computer to aim for a summary that’s just the right length, so it includes all the key information without extra fluff.

8. frequency_penalty(Default 0 Range -2.0- +2.0)

   Description: Penalizes words based on their frequency in the generated text, reducing repetition. Example: When set to 1.0, common words are used less often, resulting in more diverse text.

9. presence_penalty(Default 0 Range -2.0- +2.0)

   Description: Penalizes words that have already appeared in the text, encouraging the use of new words. Example: When set to 1.5, the model tends to use words that haven’t appeared yet, increasing text diversity.

10. stop(Range:String or String list)

    Description: Specifies one or more tokens at which to stop generation. Example: When set to [".", “!”, “?”], the model will stop generating after producing these punctuation marks.

11. n (Default 1)

    Description: Specifies how many completions to generate. Example: When set to 3, the model will generate 3 different responses.

12. best_of

    Description: Generates multiple candidate results and returns the best n. Example: With n=2 and best_of=5, the model generates 5 candidate results and returns the 2 best ones.

13. logprobs

    Description: Returns the most likely tokens and their log probabilities. Example: When set to 3, each generated token will be accompanied by the 3 most likely alternative options and their probabilities.

14. no_repeat_ngram_size

    Description: Prevents repetition of specified length word groups. Example: When set to 3, the model avoids generating any consecutive repetition of three-word combinations.

[!CAUTION]

These parameters can be used together, but with some limitations:

1. Some parameters work against each other, like high temperature with strict top-k/top-p.
2. Beam search often overrides other sampling methods.
3. Using many complex parameters at once can slow down generation.
4. Not all models support every parameter.
5. Some parameters have specific Range ranges (e.g., temperature is usually 0-2).
6. Different tasks may need different parameter combinations.
7. Stop conditions usually take priority over other parameters.

Best practice:

• Start with defaults
• Adjust one parameter at a time
• Test thoroughly
• Keep notes on what works best

Tranining parameters

#

1. Learning Rate: Explanation: Controls the step size at each iteration while moving toward a minimum of the loss function. Example: 0.0001 (1e-4)
2. Batch Size: Explanation: The number of training examples used in one iteration. Example: 32, 64, 128
3. Optimizer: Explanation: Algorithm used to update the model’s weights. Example: Adam, SGD, RMSprop
4. Epochs: Explanation: The number of complete passes through the entire training dataset. Example: 10, 50, 100
5. Weight Initialization: Explanation: Method used to set the initial random weights of the neural network. Example: Xavier initialization, He initialization
6. Regularization: Explanation: Techniques to prevent overfitting. Example: L2 regularization (weight decay = 0.01), Dropout (rate = 0.1)
7. Learning Rate Scheduler: Explanation: Strategy to adjust the learning rate during training. Example: StepLR (step_size=30, gamma=0.1), CosineAnnealingLR
8. Model Architecture: Explanation: The structure and size of the neural network. Example: Transformer with 12 layers, 768 hidden size, 12 attention heads
9. Sequence Length: Explanation: The maximum length of input sequences. Example: 512, 1024, 2048 tokens
10. Warmup Steps: Explanation: Number of steps to gradually increase the learning rate at the start of training. Example: 1000 steps
11. Gradient Clipping: Explanation: Technique to prevent exploding gradients by limiting their magnitude. Example: max_norm=1.0
12. Mixed Precision Training: Explanation: Using lower precision (e.g., float16) to speed up training and reduce memory usage. Example: Enabled with float16
13. Distributed Training Strategy: Explanation: Method for training across multiple GPUs or nodes. Example: Data Parallel, Model Parallel
14. Attention Dropout: Explanation: Dropout rate specifically for attention layers. Example: 0.1
15. Activation Function: Explanation: Non-linear function applied to neuron outputs. Example: ReLU, GELU