跳至主要内容

dpgen simplify 数据精简二次处理

问题：

1.       Carbon的势能文件无法准确描述石墨层间距 共有204,200 bch

2.       NaSPO 的势能文件无法压缩 共有 242,423 bch

都可能是data太臃肿导致的，所以决定精简用simplify命令试试（感觉是重新采样的过程，只是节省了lammps的时间用dp test代替了，还是需要fp过程（也可以注释掉，如果需要新的VASP参数可以启用），更适合需要换fp参数的计算）。第一次取样是随机的，dp采样过程有点慢 压缩的pb需要6小时。

还可以重新fp改善数据集 https://tutorials.deepmodeling.com/en/latest/CaseStudies/Transfer-learning/Transfer-learning.html

https://docs.deepmodeling.com/projects/dpgen/en/latest/simplify/simplify-jdata.html



https://zhuanlan.zhihu.com/p/456504860
http://bohrium-doc.dp.tech/docs/software/DP-GEN_simplify

Simplify — DP-GEN documentation

先做，动起来在看

1.       收集所有数据https://hikunluo.blogspot.com/2022/12/dpgen.html

2.       准备simplify的两个json文件。

    nohup dpgen simplify simplify.json machine.json 1>log 2>err&

    (base) [kluo@condo2017 SimCarbon]$ cat simplify.json

{

     "type_map": ["C"],

     "mass_map": [12.0107],

     "init_data_prefix": "",

     "init_data_sys":  [],

     "pick_data":  "/work/qan-free/kluo/Carbondpgen/collforSim",

     "labeled": true,    #这个表示初始数据集是已经labeled 还是没有(false)，精简数据集的时候用true，transfer-learning的时候用false.

     "init_pick_number":100, #初始数据集好像真的是随机选的没有用初始势能文件测试采样，但是重新labled了。

     

     "iter_pick_number":1000,  #这个决定每一轮的采样数据 建议取总数据集的1/10以下。或者目标100w steps 对应的1w个frames，所以每一步取1000争取10轮搞定。

     "model_devi_f_trust_lo":0.25, ------一开始用约0.5*之前的数值，后面再一点点提高。

     "model_devi_f_trust_hi":0.45,

    

     "sys_configs": [null], #报错 改回  "sys_configs": [],

     "sys_batch_size": ["auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto","auto"], #大于等于数据集的个数

     "training_iter0_model_path":  ["/work/qan-free/kluo/Carbondpgen/SimCarbon/novaIter32/00[0-3]"],

     "training_init_model":   false, #应该用true(simplify 需要用false)的才是继承前面的数据(迁移学习)但是这里我用的初始模型是另外一个机器上拷贝过来的，所以很多参数其实是无法使用的，所以只能关闭了。(Transfer learning is doing 'init-model' based on the original one. )看看第二代之后能不能继承前面的数据集，通过DEEPMD INFO    ---Summary of DataSystem: training     -----------------------------------------------

DEEPMD INFO    found 3 system(s):里的内容就可以查看了。

 

     "_comment": " that's all ",

     "numb_models": 4,

     "_dp_compress": true,

     "default_training_param": {

         "model": {

             "type_map": ["C"],

             "descriptor": {

            "type":             "hybrid",

            "list" : [

              {

                 "type": "se_e3",

                 "sel": [20],  #如果需要初始化旧模型，这个必须用旧模型的确定值，不能用自动挡。

                 "rcut_smth": 0.5,

                 "rcut": 2.5,

                 "neuron": [4,8,16],

                 "resnet_dt": false,

                 "seed": 1

             },

              {

                 "type": "se_e2_a",

                 "sel": [125],

                 "rcut_smth": 1.0,

                 "rcut": 5.0,

                 "neuron": [20,40,80],

                 "resnet_dt": false,

                 "axis_neuron": 12,

                 "seed": 1

             }

                     ]},

             "fitting_net": {

                 "neuron": [120,120,120],

                 "resnet_dt": true,

                 "seed": 1

             }

         },

         "learning_rate": {

             "type": "exp",

             "start_lr": 0.001,

             "stop_lr":      3.51e-8,

             "decay_steps": 2500

         },

         "loss": {

             "type":                "ener",

             "start_pref_e": 0.02,

             "limit_pref_e": 2,

             "start_pref_f": 1000,

             "limit_pref_f": 1,

             "start_pref_v": 0.01,

             "limit_pref_v": 1

         },

         "training": {

             "validation_data":{

             "systems": [

                        "/work/qan-free/kluo/Carbondpgen/validation/CAf",

                        "/work/qan-free/kluo/Carbondpgen/validation/CAp",

                "/work/qan-free/kluo/Carbondpgen/validation/CAt",

                "/work/qan-free/kluo/Carbondpgen/validation/COf",

                "/work/qan-free/kluo/Carbondpgen/validation/COp",

                "/work/qan-free/kluo/Carbondpgen/validation/COt",

                "/work/qan-free/kluo/Carbondpgen/validation/HBf",

                "/work/qan-free/kluo/Carbondpgen/validation/HBp",

                "/work/qan-free/kluo/Carbondpgen/validation/HBt",

                "/work/qan-free/kluo/Carbondpgen/validation/HCf",

                "/work/qan-free/kluo/Carbondpgen/validation/HCp",

                "/work/qan-free/kluo/Carbondpgen/validation/HCt",

                "/work/qan-free/kluo/Carbondpgen/validation/HZf",

                "/work/qan-free/kluo/Carbondpgen/validation/HZp",

                "/work/qan-free/kluo/Carbondpgen/validation/HZt"],

 

            "batch_size": 1,

            "numb_btch":               1,

            "_comment":                "that's all"

        },

             "stop_batch": 500000,

             "disp_file": "lcurve.out",

             "disp_freq": 500,

             "numb_test": 4,

             "save_freq": 10000,

             "save_ckpt": "model.ckpt",

             "disp_training": true,

             "time_training": true,

             "profiling": false,

             "profiling_file": "timeline.json",

             "_comment": "that's all"

         }

     },

    

     "fp_style": "vasp",   #设置成When fp_style is set to none:No fp. 真正的simplify（https://docs.deepmodeling.com/projects/dpgen/en/latest/simplify/simplify-jdata.html）

     "fp_skip_bad_box":  "true",

     "_shuffle_poscar": false,

     "fp_task_max": 50, #和以前一样这个还是针对每个采样相空间的（由于重新收集归纳了数据集，新的数据集将相同原子个数的整合成一个相空间了），而不是整个数据集。所以通过前面的pick控制总的合适数据，这一步尽量不要卡，所以max=pick即可

     "fp_task_min": 0, #所以这里选0确保pick的所有数据都有效。

     "ratio_failed": 0.9,

     "fp_accurate_threshold": 0.99,

     "fp_accurate_soft_threshold": 0.01,

     "fp_pp_path": "./",

     "fp_pp_files": ["POTCAR_C"],

     "fp_incar": "./INCAR_Carbon",

     "_comment": "that's all"

}

(base) [kluo@condo2017 SimCarbon]$ cat machine.json

{

                "api_version": "1.0",

                "_deepmd_version": "2.0.3",

                "train" :[

                                {

                                                "command": "dp",

                                                "machine": {

                                                                "batch_type": "slurm",

                                                                "context_type": "local",

                                                                "local_root" : "./",

                                                                "remote_root": "./work"

                                                },

                                                "resources": {

                                                                "number_node": 1,

                                                                "cpu_per_node": 16,

                                                                "gpu_per_node": 0,

                                "_queue_name": "cpu-s1-matersimul-0",

                                "custom_flags": ["###SBATCH --account=cpu-s1-matersimul-0", "#SBATCH --job-name='dptrain'", "#SBATCH --cpus-per-task=1", "#SBATCH --hint=compute_bound","#SBATCH --ntasks=16", "#SBATCH --mail-type=ALL", "#SBATCH --time=24:00:00", "#SBATCH --mail-user=kluo@iastate.edu"],

                                "source_list": ["/work/qan-free/kluo/Carbondpgen/dp203.sh"],

                                                                "_module_list": ["intel/mkl/64/2019/5.281", "intel/mpi/64/2019/5.281"],

                                "_time_limit": "240:0:0",

                                "group_size": 1

                                                }

                                }

                ],

                "model_devi":[

                                {

                                                "command": "dp",

                                                "machine": {

                                                                "batch_type": "slurm",

                                                                "context_type": "local",

                                                                "local_root" : "./",

                                                                "remote_root": "./work"

                                                },

                                                "resources": {

                                "number_node": 1,

                                "cpu_per_node": 16,

                                "gpu_per_node": 0,

                                "_queue_name": "cpu-s1-matersimul-0",

                                "custom_flags": ["###SBATCH --account=cpu-s1-matersimul-0", "#SBATCH --job-name='lmps'", "#SBATCH --cpus-per-task=1", "#SBATCH --hint=compute_bound","#SBATCH --ntasks=8", "#SBATCH --time=1:00:00"],

                                                                "source_list": ["/work/qan-free/kluo/Carbondpgen/dp203.sh"],

                                "_module_list": ["intel/mkl/64/2019/5.281", "intel/mpi/64/2019/5.281"],

                                "_time_limit": "240:0:0",

                                "group_size": 100

                        }

                                }

                ],

                "fp":[

                                {

                                                "command": "srun -n 16 /shared/hpc/vasp/5.4.1/bin/vasp_std",

                                                "machine": {

                                "batch_type": "slurm",

                                "context_type": "local",

                                "local_root" : "./",

                                "remote_root": "./work"

                        },

                        "resources": {

                                "number_node": 1,

                                "cpu_per_node": 16,

                                "gpu_per_node": 0,

                                "_queue_name": "cpu-s1-matersimul-0",

                                "custom_flags": ["###SBATCH --account=cpu-s1-matersimul-0", "#SBATCH --job-name='fp'", "#SBATCH --cpus-per-task=1", "#SBATCH --hint=compute_bound","#SBATCH --ntasks=16", "#SBATCH --time=8:00:00"],

                                "source_list": ["/home/kluo/shpy/vasp.sh"],

                                "_module_list": ["intel/mkl/64/2019/5.281", "intel/mpi/64/2019/5.281"],

                                "_time_limit": "240:0:0",

                                "group_size": 2

                        }

                                }

                ]

标签：

评论

发表评论

此博客中的热门博文

dpgen数据收集分类续算

dpgen 推荐安装升级方法：删除之前工作目录的dpgen文件夹 Install from source code:  git clone https://github.com/deepmodeling/dpgen && pip install ./dpgen 想收集TL的数据集且按化学成分分类：/work/qan/kluo/NaSPO/run/TransL dpgen collect ./ ./collect -p simplify.json -m 不起作用,只能用 param.json格式 所以单独新建文件夹，构造虚拟的 param.json其中初始数据集地址，直接复制最后一个train的input.json.再用 dpgen collect ./ ./collect -p param.json -m就可以收集初始的数据了，但是这时候没有分类，尽管dpgen collect -m 有这个功能但没有作用。 所以，构建虚拟的simplify任务，记住是精简过程，所以       "labeled" : true ,       "init_pick_number" : 0 ,       "iter_pick_number" : 0 , .....       "training_init_model" : false , 运行之后，第一步就得到了 所有的分类数据集。 Q: /work/qan/kluo/NaSPO/run 最近几轮的准确率上不去，猜测是数据集太大 232334 batch ，而每一代的学习步长 step 只有 50 万，以及不够了。所以出现了：一些模型的准确率一直上不去，就算修改上下限也没有改观。或者一些模型学好了，另一些模型就被遗忘了，很明显是学习步长不够了。 A: 解决办法如下：将现有数据收集分类，分拣出确实学习率低的重写单独重点学习。其次，以后用 dpgen 自动采样产生数据集的时候，时刻关注准确率的变化，一旦上不去了赶紧分家。 D: 1.        收集数据 a.       ...
1 条评论
阅读全文

python相关

 在IDLE中运行pip install pandas是不行的。你需要在命令提示符（Command Prompt）或终端（Terminal）中运行该命令。下面是详细步骤： 使用命令提示符安装 pandas 打开命令提示符： 按 Win + R 键打开运行窗口，输入 cmd，然后按回车键。 或者你可以在开始菜单中搜索“命令提示符”并打开它。 而且运行python脚本的一般步骤： 1. 打开IDLE shell这是个交互窗口 不能直接将脚本内容直接复制到这里,但是应该可以用python +脚本路径运行 2. File -new file-粘贴脚本内容 3.这个窗口上有run
发表评论
阅读全文