当前位置: 首页 > news >正文

GPT 结束语设计 以nanogpt为例

GPT 结束语设计 以nanogpt为例

目录

GPT 结束语设计 以nanogpt为例

1、简述

2、分词设计

3、结束语断点


1、简述

在手搓gpt的时候,可能会遇到一些性能问题,即关于是否需要全部输出或者怎么节约资源。

在输出语句被max_new_tokens 限制,如果出现一些输出句子比较长,就会被限制,但如果是设计时候没有设计结束语,就会出现全部输出的问题。

如果只需要一部分的语句,或者是某一些特定的场景设计,例如:

1、gpt自动化操作

2、输出美观

3、一些较小的业务场景,特定处理的业务

以上的业务场景都是设计的时候为特定模型,即小大模型,通常不需要较大的参数,所以在设计时候如果考虑到轻量化和小型化,参数1M至100M之间的小大模型。

基于成本和开发快速考虑,可以使用nanogpt用于训练和开发,然后再进一步的微调迭代,所需要的性能和效果基本可以满足部分要求,迭代速度较快,适合单人或小团队开发特定场景。


2、分词设计

以下是关于之前做过的一个开发场景:音乐生成按键的场景

分词中加入了end的作为特定的结束语,如果后续扩展可以通过end前后设计一些音乐风格的标识符,这样通过风格的标识来达到风格的统一。


# 自定义词典
word_dict = set(['\n', ' ', '+', '.', '0', '1', '2', '3', '4'
         '6', '7', '8', '9', ':', "'a'", "'b'", "'c'", "'d'",
         "'e'", "'f'", "'g'", "'h'","'j'", "'n'","'m'","'q'","'w'","'r'","'t'","'y'","'u'",
        "'s'", "'v'", "'x'", "'z'",'<96>','<97>','<98>','<99>','<100>',
        '<101>','<102>','<103>','<104>','<105>','end'])

seg_list = max_forward_matching(data, word_dict, max(len(word) for word in word_dict))
words = list(seg_list)
# 创建一个默认字典来存储词汇到ID的映射
word_to_id = defaultdict(lambda: len(word_to_id))
# 创建一个列表来存储ID到词汇的映射(可选)
id_to_word = []
# 构建词汇到ID的映射
for word in words:
    word_id = word_to_id[word]
    # ID到词汇的映射,可以这样做:
    if word_id == len(word_to_id):  # 只有当新的ID被分配时才添加到id_to_word中
        id_to_word.append(word)

import os
import pickle
import requests
import numpy as np
from collections import defaultdict
# download the tiny shakespeare dataset
input_file_path = os.path.join(os.path.dirname(__file__), 'music.txt')
if not os.path.exists(input_file_path):data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'with open(input_file_path, 'w') as f:f.write(requests.get(data_url).text)with open(input_file_path, 'r',encoding="utf-8") as f:data = f.read()
print(f"length of dataset in characters: {len(data):,}")# get all the unique characters that occur in this text
def max_forward_matching(text, word_dict, max_len):result = []index = 0while index < len(text):found = Falsefor size in range(max_len, 0, -1):  # 从最大长度开始尝试匹配piece = text[index:index + size]if piece in word_dict:result.append(piece)index += sizefound = Truebreakif not found:  # 如果没有找到匹配的词,则按字符输出result.append(text[index])index += 1return result#自建一套
# 自定义词典
word_dict = set(['\n', ' ', '+', '.', '0', '1', '2', '3', '4''6', '7', '8', '9', ':', "'a'", "'b'", "'c'", "'d'","'e'", "'f'", "'g'", "'h'","'j'", "'n'","'m'","'q'","'w'","'r'","'t'","'y'","'u'","'s'", "'v'", "'x'", "'z'",'<96>','<97>','<98>','<99>','<100>','<101>','<102>','<103>','<104>','<105>','end'])seg_list = max_forward_matching(data, word_dict, max(len(word) for word in word_dict))
words = list(seg_list)
# 创建一个默认字典来存储词汇到ID的映射
word_to_id = defaultdict(lambda: len(word_to_id))
# 创建一个列表来存储ID到词汇的映射(可选)
id_to_word = []
# 构建词汇到ID的映射
for word in words:word_id = word_to_id[word]# ID到词汇的映射,可以这样做:if word_id == len(word_to_id):  # 只有当新的ID被分配时才添加到id_to_word中id_to_word.append(word)chars = list(word_to_id)
print(chars)
vocab_size = len(chars)print("all the unique characters:", ''.join(chars))
print(f"vocab size: {vocab_size:,}")
#Myzzb That is need about jieba to cut text
print(chars)
# create a mapping from characters to integers
stoi = { ch:i for i,ch in enumerate(chars) }
print(stoi)
itos = { i:ch for i,ch in enumerate(chars) }
print(itos)def encode(s):seg_list = max_forward_matching(data, word_dict, max(len(word) for word in word_dict))words = list(seg_list)# 创建一个默认字典来存储词汇到ID的映射word_to_id = defaultdict(lambda: len(word_to_id))# 创建一个列表来存储ID到词汇的映射id_to_word = []# 构建词汇到ID的映射for word in words:word_id = word_to_id[word]# 如果你也需要ID到词汇的映射,可以这样做:if word_id == len(word_to_id):  # 只有当新的ID被分配时才添加到id_to_word中id_to_word.append(word)return [word_to_id[word] for word in words] # encoder: take a string, output a list of integers
def decode(l):seg_list = max_forward_matching(data, word_dict, max(len(word) for word in word_dict))words = list(seg_list)# 创建一个默认字典来存储词汇到ID的映射word_to_id = defaultdict(lambda: len(word_to_id))# 创建一个列表来存储ID到词汇的映射(可选)id_to_word = []# 构建词汇到ID的映射for word in words:word_id = word_to_id[word]# 如果你也需要ID到词汇的映射,可以这样做:if word_id == len(word_to_id):  # 只有当新的ID被分配时才添加到id_to_word中id_to_word.append(word)return ''.join([word_to_id[word] for word in words]) # decoder: take a list of integers, output a string
# create the train and test splits
n = len(data)
train_data = data[:int(n*0.95)]#这里因为没写字典排序,所以训练集和测试集懒得分开
val_data = data[int(n*0.95):]
# print(val_data)
# encode both to integers
train_ids = encode(train_data)
print(train_ids)
val_ids = encode(val_data)
print(val_ids)
# print(val_ids)
print(f"train has {len(train_ids):,} tokens")
print(f"val has {len(val_ids):,} tokens")# export to bin files
train_ids = np.array(train_ids, dtype=np.uint16)
val_ids = np.array(val_ids, dtype=np.uint16)
train_ids.tofile(os.path.join(os.path.dirname(__file__), 'train.bin'))
val_ids.tofile(os.path.join(os.path.dirname(__file__), 'val.bin'))# save the meta information as well, to help us encode/decode later
meta = {'vocab_size': vocab_size,'itos': itos,'stoi': stoi,
}
with open(os.path.join(os.path.dirname(__file__), 'meta.pkl'), 'wb') as f:pickle.dump(meta, f)

3、结束语断点

通过在推理过程中检测新生成的编码是否和结束语一致,以上在设计的过程中通过字典分词,然后再分配的编码,是可以通过代码获取对应的结束语的编码。

通过在分词的时候进行对部分结束语进行输出,例子:

print(encode("\n"))
print(encode("\t"))

源码添加上,即可知道结束语的编码是多少:

"""
Prepare the Shakespeare dataset for character-level language modeling.
So instead of encoding with GPT-2 BPE tokens, we just map characters to ints.
Will save train.bin, val.bin containing the ids, and meta.pkl containing the
encoder and decoder and some other related info.
"""
import os
import pickle
import requests
import numpy as np# download the tiny shakespeare dataset
input_file_path = os.path.join(os.path.dirname(__file__), 'say.txt')
if not os.path.exists(input_file_path):data_url = 'https://raw.githubusercontent.com/karpathy/char-rnn/master/data/tinyshakespeare/input.txt'with open(input_file_path, 'w') as f:f.write(requests.get(data_url).text)with open(input_file_path, 'r',encoding="utf-8", errors='replace') as f:data = f.read()
print(f"length of dataset in characters: {len(data):,}")# get all the unique characters that occur in this text
chars = sorted(list(set(data)))
vocab_size = len(chars)
print("all the unique characters:", ''.join(chars))
print(f"vocab size: {vocab_size:,}")# create a mapping from characters to integers
stoi = { ch:i for i,ch in enumerate(chars) }
itos = { i:ch for i,ch in enumerate(chars) }def encode(s):return [stoi[c] for c in s] # encoder: take a string, output a list of integers
def decode(l):return ''.join([itos[i] for i in l]) # decoder: take a list of integers, output a stringprint(encode("\n"))
print(encode("\t"))# create the train and test splits
n = len(data)
train_data = data[:int(n*0.9)]
val_data = data[int(n*0.9):]# encode both to integers
train_ids = encode(train_data)
val_ids = encode(val_data)
print(f"train has {len(train_ids):,} tokens")
print(f"val has {len(val_ids):,} tokens")# export to bin files
train_ids = np.array(train_ids, dtype=np.uint16)
val_ids = np.array(val_ids, dtype=np.uint16)
train_ids.tofile(os.path.join(os.path.dirname(__file__), 'train.bin'))
val_ids.tofile(os.path.join(os.path.dirname(__file__), 'val.bin'))# save the meta information as well, to help us encode/decode later
meta = {'vocab_size': vocab_size,'itos': itos,'stoi': stoi,
}
with open(os.path.join(os.path.dirname(__file__), 'meta.pkl'), 'wb') as f:pickle.dump(meta, f)# length of dataset in characters:  1115394
# all the unique characters:
#  !$&',-.3:;?ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
# vocab size: 65
# train has 1003854 tokens
# val has 111540 tokens

只需要简单添加一句代码即可:

# 检查是否生成了结束语 可以获取大部分结束语的编码用于判断 也可以自拟结束语 将其处理为唯一的标识符避免干扰
if 1 in idx_next[0].tolist():break

@torch.no_grad()def generate(self, idx, max_new_tokens, temperature=1.0, top_k=None):"""Take a conditioning sequence of indices idx (LongTensor of shape (b,t)) and completethe sequence max_new_tokens times, feeding the predictions back into the model each time.Most likely you'll want to make sure to be in model.eval() mode of operation for this."""for _ in range(max_new_tokens):# if the sequence context is growing too long we must crop it at block_sizeidx_cond = idx if idx.size(1) <= self.config.block_size else idx[:, -self.config.block_size:]# forward the model to get the logits for the index in the sequencelogits, _ = self(idx_cond)# pluck the logits at the final step and scale by desired temperaturelogits = logits[:, -1, :] / temperature# optionally crop the logits to only the top k optionsif top_k is not None:v, _ = torch.topk(logits, min(top_k, logits.size(-1)))logits[logits < v[:, [-1]]] = -float('Inf')# apply softmax to convert logits to (normalized) probabilitiesprobs = F.softmax(logits, dim=-1)# sample from the distributionidx_next = torch.multinomial(probs, num_samples=1)# 检查是否生成了结束语 可以获取大部分结束语的编码用于判断 也可以自拟结束语 将其处理为唯一的标识符避免干扰if 1 in idx_next[0].tolist():break# append sampled index to the running sequence and continueidx = torch.cat((idx, idx_next), dim=1)return idx
http://www.lryc.cn/news/526378.html

相关文章:

  • FastDFS的安装及使用
  • C++ lambda表达式
  • react页面定时器调用一组多个接口,如果接口请求返回令牌失效,清除定时器不再触发这一组请求
  • Python的泛型(Generic)与协变(Covariant)
  • Python Typing: 实战应用指南
  • OpenEuler学习笔记(六):OpenEuler与其他Linux服务器的区别是什么?
  • 如何使用CRM数据分析和洞察来支持业务决策和市场营销?
  • MyBatis和JPA区别详解
  • SVN客户端使用手册
  • VsCode安装文档
  • 豆包MarsCode 蛇年编程大作战 | 高效开发“蛇年运势预测系统”
  • 【动态规划】--- 斐波那契数模型
  • 生信软件管家——conda vs pip
  • 代码随想录——串
  • 詳細講一下RN(React Native)中的列表組件FlatList和SectionList
  • TDengine 与上海电气工业互联网平台完成兼容性认证
  • 随机矩阵投影长度保持引理及其证明
  • 深度学习利用数据加载、预处理和增强数据提高模型的性能
  • ESP32服务器和PC客户端的Wi-Fi通信
  • 新型人工智能“黑帽”工具:GhostGPT带来的威胁与挑战
  • Spring MVC (三) —— 实战演练
  • 媒体新闻发稿要求有哪些?什么类型的稿件更好通过?
  • 【游戏设计原理】82 - 巴斯特原则
  • DDD架构实战第六讲总结:领域驱动设计中的聚合
  • vim如何设置自动缩进
  • C++入门14——set与map的使用
  • 单片机内存管理剖析
  • 【gopher的java学习笔记】Java中Service与Mapper的关系详解
  • 2025美赛B题完整代码+建模过程
  • 【MySQL】我在广州学Mysql 系列——MySQL用户管理详解