推荐算法策略需求-rank model优化
1.pred_oobe
(base) [rus@xx]$ pwd /home/disk2/data/xx/icode/baidu/oxygen/rus-pipeline/pipeline-migrate/UserBaseActiveStatPipeline/his_session
(base) [rus@xx]$ sh test.sh
2. user_skill_history_dict_expt2包含userid
[work@xx]$
vim /home/work/xx/du-rus/du_rus_offline/dict/skill_engine_dict/user_skill_history_dir/user_skill_history_dict_expt2
[work@xx]$ grep '2916418677' user_skill_history_dict_expt2
3.置顶技能灌库
增加
curl "10.194.200.12:8123/DproxyServer/cmd" -d '{"cmd":"zadd", "args":["appstore_user_favour_6S192231D45C9AFB_2916418677","1","LP_2155","1"," LP_4993"], "name":"show-app-store"}'
查询
curl "10.194.200.12:8123/DproxyServer/cmd" -d '{"cmd":"zrange", "args":["appstore_user_favour_6S192231D45C9AFB_2916418677","0","-1"], "name":" show-app-store"}'
删除
curl "10.194.200.12:8123/DproxyServer/cmd" -d '{"cmd":"del", "args":["appstore_user_favour_6S192231D45C9AFB_2916418677"], "name":"show-app- store"}'
4.禁用技能灌库
5.user_skill_feature灌库 /home/disk2/data/xx/icode/baidu/oxygen/rus-pipeline/pipeline-migrate/LpQsbRankPipeline/optimizer
下载文件
wget gzhxy-bdrp-ps-se-nlp-arch-xiaodu-gzhxy003.gzhxy.baidu.com:/home/disk2/data/xx/temp/test_naotu.txt
老字典: /home/disk2/data/xx/icode/baidu/oxygen/rus-pipeline/pipeline-migrate/UserBaseActiveStatPipeline/his_session, sh test.sh
部署环境中dict/skill_engine_dict/user_skill_history_dir/user_skill_history_dict_expt2中包含测试userid
增:curl "10.194.200.12:8123/DproxyServer/cmd" -d '{"cmd":"zadd", "args":["appstore_user_favour_6S192231D45C9AFB_2916418677", "1", "LP_2155", "1", "LP_4993"], "name":"show-app-store"}'
查:curl "10.194.200.12:8123/DproxyServer/cmd" -d '{"cmd":"zrange", "args":["appstore_user_favour_6S192231D45C9AFB_2916418677", "0", "-1"], "name":"show-app-store"}'
删:curl "10.194.200.12:8123/DproxyServer/cmd" -d '{"cmd":"del", "args":["appstore_user_favour_6S192231D45C9AFB_2916418677"], "name":"show-app- store"}'
/home/disk2/data/xx/icode/baidu/oxygen/rus-pipeline/pipeline-migrate/LpQsbRankPipeline/optimizer/love_forbidden_skills, sh go.sh
/home/disk2/data/xx/icode/baidu/oxygen/rus-pipeline/pipeline-migrate/LpQsbRankPipeline/post_process/user_skill_feature_process, sh test_run.sh
wget gzhxy-bdrp-ps-se-nlp-arch-xiaodu-gzhxy003.gzhxy.baidu.com:/home/disk1/rus/offline-pipeline-routine/baidu/oxygen/rus-pipeline/pipeline- migrate/LpQsbRankPipeline/post_process/train_model/launchpad_model_manager/model_version.conf， 替换部署环境中的dict/launchpad_model_manager/model_version.conf
重启rus
新字典:
1、wget gzhxy-bdrp-ps-se-nlp-arch-xiaodu-gzhxy003.gzhxy.baidu.com:/home/disk2/data/xx/icode/baidu/oxygen/rus-pipeline/pipeline- migrate/LpQsbRankPipeline/post_process/skill_feature_process/skill_feature_dict_new.tar.gz并解压 2、将部署环境中dict/skill_feature_dict/skill_feature替换skill_feature_dict_new中的skill_feature，并执行sh go.sh 3、将skill_feature_dict_new替换掉部署环境中的dict/skill_feature_dict
4、重启环境
/home/disk2/data/xx/icode/baidu/oxygen/rus-pipeline/pipeline-migrate/LpQsbRankPipeline/post_process/user_skill_feature_process_new, sh test_run.sh
1、wget gzhxy-bdrp-ps-se-nlp-arch-xiaodu-gzhxy003.gzhxy.baidu.com:/home/disk2/data/xx/icode/baidu/oxygen/rus-pipeline/pipeline- migrate/LpQsbRankPipeline/optimizer/train_model/launchpad_model_manager/model_version.conf 替换部署环境中的dict/launchpad_model_manager/model_version.conf
2、wget gzhxy-bdrp-ps-se-nlp-arch-xiaodu-gzhxy003.gzhxy.baidu.com:/home/disk2/data/xx/icode/baidu/oxygen/rus-pipeline/pipeline- migrate/LpQsbRankPipeline/optimizer/train_model/launchpad_model_manager/gbdt/optimizer_v1.tar.gz， 解压到部署环境中的dict/launchpad_model_manager/gbdt目录下
3、重启环境 机器:gzhxy-bdrp-ps-se-nlp-arch-xiaodu-gzhxy003.gzhxy.baidu.com，rus
测试userid:2916418677，cuid:6S192231D45C9AFB，clientId:lNwEAEASbln5FhNTs7A8oDGW3TlQ2khy，请求体中requester=" LAUNCHPAD_CACHE"
[work@nj02-ps-wwwda13-n1.nj02.baidu.com rus_env]$ pwd /home/work/xx/du-rus/offline-tools/du_rus/rus_env
[work@nj02-ps-wwwda13-n1.nj02.baidu.com rus_env]$
python buildenv_rus.py aabe7d6575a6dca37c5d8409475a5febcb844b10 d6a6eb0f560cf14b75ecda9a7d83e165b68cd4d3
修改完截图的配置
[work@nj02-ps-wwwda13-n1.nj02.baidu.com sample]$
vim /home/work/xx/du-rus/du_rus_offline/dict/sample/sample_variable.conf 实验组:1
对照组:0
打开日志级别
vim /du-rus/du_rus_offline/conf/log.conf 第31行rpc.log 追加,TRACE,DEBUG
31 RPC_SELFLEVEL : NOTICE,TRACE,DEBUG
重启服务 /home/work/xx/du-rus/du_rus_offline/bin ./server_control stop 暂停服务
./server_control start 启动服务
[work@nj02-ps-wwwda13-n1.nj02.baidu.com log]$ vim /home/work/xx/du-rus/du_rus_offline/log/rpc.log 日志查询candidate_features
测试结论
测试方式:捞取线上请求，java自动化方式实现
构造不同的userid 灌库 非灌库数据 不同实验组 对照组dumisid 请求 生成数据 捞取日志分析
老词典实验组 对照组 灌库数据 非灌库数据
507252, 507252, 507252, 507252,
2916418677, 2916418677, 2916418677, 2916418677,
6S192231D45C9AFB, 6S192231D45C9AFB, 6S1927504C666DF4, 6S1927504C666DF4,
日志搜索
?candidate_features
?xx: model_manager->regress succ
非灌库数据 userId=6071212314 4个特征分数为-1
灌库数据 userId=2916418677 4个特征分数均正常
实验组 507251 无特征 无分数
对照组 507252 有特征 分数
新词典对照组
实验组 507251 有特征 分数
打分如下
对照组 507252 有特征 分数
测试数据说明:
新词典：实验组 对照组打分不一致 pass
老词典：实验组无数据 对照组正常打分 pass
仅对照组，新词典与老词典同一技能打分一致 pass