Recognizing images Source: GPT-4 Technical Report
Posted: Sat Jan 25, 2025 5:34 am
The paper only mentions that GPT-4 was trained using masking and Reinforcement Learning with Human Feedback (RLHF). RLHF describes a type of reinforcement learning in which an agent - such as an AI - receives feedback from human experts to improve its decisions. The feedback can, for example, be in the form of corrections or ratings that an expert gives to the agent to influence its behavior. RLHF is often used to make reinforcement learning faster and more efficient by using human intuition and experience to guide the learning process.
What's interesting is that the good results in the human tests are mainly due to the masked pre-training, i.e. the part of the training in which the network mainly receives sentences in which the individual words are masked. With RLHF, the network is better adapted to human communication. One might therefore assume that this approach would be better albania consumer email list suited to passing human tests, but the opposite is the case. The reasons for this are not mentioned in the report.
The advantage of RLHF is not in performance, but in the fact that the model is easier for humans to handle. So you don't need a specially trained prompt engineer, anyone can do it.
The model still has weaknesses in certain areas. For example, outputs were often too vague to be useful, yielded impractical solutions, or were prone to factual errors. In addition, longer answers were more likely to contain inaccuracies. The report also notes that the model was more likely to produce a vague or inaccurate answer when it came to multi-step instructions for developing a radiological device or biochemical compound. However, it is important to note that these limitations are specific to certain fields and contexts and do not necessarily apply to all use cases. Nevertheless, GPT-4 was able to reduce these fabricated statements, referred to as hallucinations.
Risk and Mitigations
When developing GPT-4, OpenAI placed particular emphasis on improving safety and alignment. The reason for this is to better prepare GPT-4 for commercial use. Ultimately, these measures mean that the model offers significantly fewer problematic answers. Adversarial testing and a proprietary model safety pipeline were the two measures that were implemented.
What's interesting is that the good results in the human tests are mainly due to the masked pre-training, i.e. the part of the training in which the network mainly receives sentences in which the individual words are masked. With RLHF, the network is better adapted to human communication. One might therefore assume that this approach would be better albania consumer email list suited to passing human tests, but the opposite is the case. The reasons for this are not mentioned in the report.
The advantage of RLHF is not in performance, but in the fact that the model is easier for humans to handle. So you don't need a specially trained prompt engineer, anyone can do it.
The model still has weaknesses in certain areas. For example, outputs were often too vague to be useful, yielded impractical solutions, or were prone to factual errors. In addition, longer answers were more likely to contain inaccuracies. The report also notes that the model was more likely to produce a vague or inaccurate answer when it came to multi-step instructions for developing a radiological device or biochemical compound. However, it is important to note that these limitations are specific to certain fields and contexts and do not necessarily apply to all use cases. Nevertheless, GPT-4 was able to reduce these fabricated statements, referred to as hallucinations.
Risk and Mitigations
When developing GPT-4, OpenAI placed particular emphasis on improving safety and alignment. The reason for this is to better prepare GPT-4 for commercial use. Ultimately, these measures mean that the model offers significantly fewer problematic answers. Adversarial testing and a proprietary model safety pipeline were the two measures that were implemented.