{"id":116019,"date":"2019-05-22T11:29:45","date_gmt":"2019-05-22T03:29:45","guid":{"rendered":"http:\/\/www.fagao.me\/?p=116019"},"modified":"2019-05-22T11:29:45","modified_gmt":"2019-05-22T03:29:45","slug":"%e5%ad%a6%e7%95%8c-%e7%a5%9e%e7%bb%8f%e4%bc%98%e5%8c%96%e5%99%a8%e6%90%9c%e7%b4%a2-%e5%88%a9%e7%94%a8%e5%bc%ba%e5%8c%96%e5%ad%a6%e4%b9%a0%e8%87%aa%e5%8a%a8%e6%90%9c%e7%b4%a2%e6%9c%80%e4%bc%98","status":"publish","type":"post","link":"https:\/\/www.fagao.me\/p\/116019.htm","title":{"rendered":"\u5b66\u754c | \u795e\u7ecf\u4f18\u5316\u5668\u641c\u7d22 \u5229\u7528\u5f3a\u5316\u5b66\u4e60\u81ea\u52a8\u641c\u7d22\u6700\u4f18\u5316\u65b9\u6cd5"},"content":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/www.fagao.me\/images\/uploads\/2021_008.jpg\" alt=\"\"><\/p>\n<div class=\"focusup\"><\/div>\n<p><span><span>\u9009\u81eaarXiv<\/span><\/span><\/p>\n<p><span><strong>\u53c2\u4e0e\uff1a\u8def\u96ea<\/strong><\/span><\/p>\n<blockquote>\n<p><span>\u672c\u8bba\u6587\u901a\u8fc7\u5f3a\u5316\u5b66\u4e60\u7684\u65b9\u5f0f\u91c7\u6837\u4e0d\u540c\u7684\u66f4\u65b0\u89c4\u5219\u800c\u5f97\u51fa\u66f4\u52a0\u4f18\u79c0\u7684\u4f18\u5316\u65b9\u6cd5\uff0c\u8fd9\u4e9b\u4e0d\u540c\u7684\u4f18\u5316\u89c4\u5219\u901a\u8fc7\u91c7\u6837\u7684\u6982\u7387\u548c\u5176\u5728\u5b50\u7f51\u7edc\u7684\u6027\u80fd\u800c\u5f97\u51fa\u8be5\u89c4\u5219\u7684\u91cd\u8981\u6027\u3002\u672c\u6587\u63d0\u51fa\u7684\u8fd9\u79cd\u4f18\u5316\u65b9\u6cd5\u53ef\u4ee5\u79fb\u690d\u5230\u4e0d\u540c\u7684\u795e\u7ecf\u7f51\u7edc\u67b6\u6784\u4e2d\uff0c\u5e76\u6709\u5341\u5206\u4f18\u79c0\u7684\u6027\u80fd\u3002\u673a\u5668\u4e4b\u5fc3\u5bf9\u8be5\u8bba\u6587\u8fdb\u884c\u4e86\u7b80\u8981\u5730\u4ecb\u7ecd\u3002<\/span><\/p>\n<\/blockquote>\n<p><span>\u8bba\u6587\u5730\u5740\uff1ahttp:\/\/proceedings.mlr.press\/v70\/bello17a\/bello17a.pdf<\/span><\/p>\n<p><span>\u6211\u4eec\u63d0\u51fa\u4e86\u4e00\u79cd\u53ef\u81ea\u52a8\u63a2\u7d22\u4f18\u5316\u7b97\u6cd5\u7684\u65b9\u6cd5\uff0c\u8be5\u65b9\u6cd5\u91cd\u70b9\u5173\u6ce8\u6df1\u5ea6\u5b66\u4e60\u67b6\u6784\u3002\u6211\u4eec\u8bad\u7ec3\u4e86\u4e00\u4e2a\u5faa\u73af\u795e\u7ecf\u7f51\u7edc\u63a7\u5236\u5668\u751f\u6210\u7279\u5b9a\u57df\u8bed\u8a00\uff08domain language\uff09\u4e2d\u7684\u5b57\u7b26\u4e32\uff0c\u8be5\u8bed\u8a00\u63cf\u8ff0\u4e00\u7cfb\u5217\u57fa\u4e8e\u539f\u51fd\u6570\uff08\u6bd4\u5982\u68af\u5ea6\u53ca\u5176\u8fd0\u884c\u5e73\u5747\u6570\uff08running average\uff09\u7b49\uff09\u7684\u6570\u5b66\u66f4\u65b0\u65b9\u7a0b\u3002\u63a7\u5236\u5668\u901a\u8fc7\u5f3a\u5316\u5b66\u4e60\u8fdb\u884c\u8bad\u7ec3\u4ee5\u5728\u82e5\u5e72\u4e2a epoch \u4e4b\u540e\u6700\u5927\u5316\u6a21\u578b\u7684\u6027\u80fd\u3002\u5728 CIFAR-10 \u4e2d\uff0c\u6211\u4eec\u7684\u65b9\u6cd5\u53d1\u73b0\u4e86\u4e00\u4e9b\u66f4\u65b0\u89c4\u5219\u4f18\u4e8e\u8bb8\u591a\u5e38\u7528\u7684\u4f18\u5316\u5668\uff0c\u6bd4\u5982 Adam\u3001RMSProp\uff0c\u6216\u5728\u5377\u79ef\u7f51\u7edc\u6a21\u578b\u4e2d\u5e26\u6709\u548c\u4e0d\u5e26\u6709\u52a8\u91cf\u7684\u4f18\u5316\u5668\u3002\u8fd9\u4e9b\u4f18\u5316\u5668\u4e5f\u53ef\u4ee5\u8f6c\u79fb\u5230\u4e0d\u540c\u7684\u795e\u7ecf\u7f51\u7edc\u67b6\u6784\uff0c\u5e76\u975e\u5e38\u4f18\u79c0\u5730\u6267\u884c\uff0c\u5305\u62ec\u8c37\u6b4c\u7684\u795e\u7ecf\u673a\u5668\u7ffb\u8bd1\u7cfb\u7edf\u3002<\/span><\/p>\n<\/p>\n<p><span><em><span>\u56fe 1. \u795e\u7ecf\u4f18\u5316\u5668\u641c\u7d22\uff08Neural Optimizer Search\uff09\u6982\u89c8\u3002<\/span><\/em><\/span><\/p>\n<\/p>\n<p><span><em><span>\u56fe 2. \u4e00\u4e9b\u5e38\u7528\u4f18\u5316\u5668\uff08\u5982 SGD\u3001RMSProp\u3001Adam\uff09\u7684\u8ba1\u7b97\u56fe\u3002\u8fd9\u91cc\uff0c\u6211\u4eec\u5c55\u793a\u4e86 Adam \u5728\u7b2c\u4e00\u6b65\u548c\u7b2c\u4e8c\u6b65\u7684\u8ba1\u7b97\u8fc7\u7a0b\u3002\u84dd\u8272\u6846\u4ee3\u8868\u8f93\u5165\u57fa\u5143\uff08input primitive\uff09\u6216\u4e34\u65f6\u8f93\u51fa\uff08temporary output\uff09\uff0c\u9ec4\u8272\u6846\u4ee3\u8868\u4e00\u5143\u51fd\u6570\uff0c\u7070\u8272\u6846\u4ee3\u8868\u4e8c\u5143\u51fd\u6570\u3002g \u4ee3\u8868\u68af\u5ea6\uff0c<\/span><\/em><\/span><span><em><span>\u4ee3\u8868\u68af\u5ea6\u7684\u504f\u5dee\u4fee\u6b63\u4f30\u8ba1\uff0c<\/span><\/em><\/span><span><em><span>\u4ee3\u8868\u4e8c\u9636\u68af\u5ea6\u7684\u504f\u5dee\u4fee\u6b63\u4f30\u8ba1\u3002<\/span><\/em><\/span><\/p>\n<\/p>\n<p><span><em><span>\u56fe 3. \u63a7\u5236\u5668 RNN \u6982\u89c8\u3002\u63a7\u5236\u5668\u53ef\u4ee5\u8fed\u4ee3\u9009\u62e9 length 5 \u7684\u5b50\u5e8f\u5217\uff1a\u9996\u5148\u9009\u62e9\u7b2c 1 \u548c\u7b2c 2 \u4e2a\u64cd\u4f5c\u6570 op1 \u548c op2\uff0c\u7136\u540e\u5c06\u4e24\u4e2a\u4e00\u5143\u51fd\u6570 u1 \u548c u2 \u5e94\u7528\u5230\u4e24\u4e2a\u64cd\u4f5c\u6570\uff08operand\uff09\u4e0a\uff0c\u6700\u540e\u4e8c\u5143\u51fd\u6570 b \u8fde\u63a5\u4e00\u5143\u51fd\u6570\u7684\u8f93\u51fa\u3002\u7136\u540e b(u1(op1); u2(op2)) \u6210\u4e3a\u53ef\u5728\u540e\u7eed\u7ec4\u9884\u6d4b\u4e2d\u4f5c\u4e3a\u64cd\u4f5c\u6570\uff0c\u6216\u8005\u6210\u4e3a\u66f4\u65b0\u89c4\u5219\u3002\u6bcf\u4e00\u4e2a\u9884\u6d4b\u90fd\u7531\u6700\u540e\u7684 softmax \u5206\u7c7b\u5668\u5f97\u51fa\uff0c\u7136\u540e\u4f5c\u4e3a\u4e0b\u4e00\u4e2a\u65f6\u95f4\u6b65\u7684\u8f93\u5165\u3002<\/span><\/em><\/span><\/p>\n<\/p>\n<p><span><em><span>\u56fe 4. \u968f\u7740\u91c7\u6837\u4f18\u5316\u5668\u4e0d\u65ad\u589e\u52a0\uff0c\u63a7\u5236\u5668\u5956\u52b1\u4e5f\u968f\u7740\u65f6\u95f4\u4e0d\u65ad\u4e0a\u5347\u3002<\/span><\/em><\/span><\/p>\n<\/p>\n<p><span><em><span>\u56fe 5. \u795e\u7ecf\u4f18\u5316\u5668\u641c\u7d22\uff08Neural Optimizer Search\uff09\u4e2d\u4e24\u4e2a\u6700\u597d\u7684\u4f18\u5316\u5668\u95f4\u7684\u5bf9\u6bd4\uff0c\u5b83\u4eec\u90fd\u4f7f\u7528\u53cc\u5c42\u5377\u79ef\u7f51\u7edc\u67b6\u6784\u3002\u4f18\u5316\u5668 1 \u6307 <\/span><\/em><\/span><span><em><span>\uff0c\u4f18\u5316\u5668 2 \u6307 <\/span><\/em><\/span><span><em><span>\u3002<\/span><\/em><\/span><\/p>\n<\/p>\n<p><span><em><span>\u56fe 6. \u795e\u7ecf\u7f51\u7edc\u4f18\u5316\u5668\u641c\u7d22\u4e2d\u7684\u4e00\u4e2a\u4f18\u5316\u5668\u548c Rosenbrock \u51fd\u6570\u4e0a\u8457\u540d\u7684\u4f18\u5316\u5668\u4e4b\u95f4\u7684\u5bf9\u6bd4\u3002\u4f18\u5316\u5668 1 \u6307<\/span><\/em><\/span><span><em><span>\u3002\u9ed1\u70b9\u4ee3\u8868\u6700\u4f73\u7ed3\u679c\u3002<\/span><\/em><\/span><\/p>\n<\/p>\n<p><span><em><span>\u56fe 7. \u795e\u7ecf\u4f18\u5316\u5668\u641c\u7d22\u4e2d\u4e24\u4e2a\u6700\u597d\u7684\u4f18\u5316\u5668\u95f4\u7684\u5bf9\u6bd4\uff0c\u5b83\u4eec\u4f7f\u7528 Wide ResNet \u67b6\u6784\u3002\u4f18\u5316\u5668 1 \u6307 <\/span><\/em><\/span><span><em><span>\uff0c\u4f18\u5316\u5668 2 \u6307<\/span><\/em><\/span><span><em><span>\u3002<\/span><\/em><\/span><\/p>\n<\/p>\n<p><span><em><span>\u8868 1. \u795e\u7ecf\u7f51\u7edc\u4f18\u5316\u5668\u641c\u7d22\u548c Wide-ResNet \u67b6\u6784\u4e0a\u7684\u6807\u51c6\u4f18\u5316\u5668\u5728 CIFAR-10 \u4e0a\u7684\u6027\u80fd\u5bf9\u6bd4\u3002Final Val \u548c Final Test \u6307\u5728 300 \u4e2a epoch \u4e0a\u8bad\u7ec3\u4e4b\u540e\u7684\u6700\u7ec8\u9a8c\u8bc1\u548c\u6d4b\u8bd5\u51c6\u786e\u5ea6\u3002Best Val \u6307\u5728 300 \u4e2a epoch \u4e0a\u7684\u6700\u4f73\u9a8c\u8bc1\u51c6\u786e\u5ea6\uff0cBest Test \u6307\u9a8c\u8bc1\u51c6\u786e\u5ea6\u6700\u9ad8\u7684 epoch \u4e0a\u7684\u6d4b\u8bd5\u51c6\u786e\u5ea6\u3002\u5bf9\u4e8e\u6bcf\u4e00\u4e2a\u4f18\u5316\u5668\uff0c\u6211\u4eec\u6839\u636e\u9a8c\u8bc1\u51c6\u786e\u5ea6\u62a5\u544a 7 \u4e2a\u5b66\u4e60\u7387\u5728\u5bf9\u6570\u5c3a\u5ea6\u4e0a\u7684\u6700\u4f73\u7ed3\u679c\u3002<\/span><\/em><\/span><\/p>\n<\/p>\n<p><span><em><span>\u8868 2. \u6211\u4eec\u7684\u4f18\u5316\u5668\u4e0e\u5f3a\u5927\u7684\u57fa\u7ebf GNMT \u6a21\u578b\u4e2d\u7684\u4f18\u5316\u5668 ADAM \u5728 WMT 2014 \u82f1\u8bed\u8f6c\u5fb7\u8bed\u7ffb\u8bd1\u4efb\u52a1\u4e0a\u7684\u6027\u80fd\u5bf9\u6bd4\u3002<\/span><\/em><\/span><\/p>\n<p><strong><em><span><strong><em><span>\u672c\u6587\u4e3a\u673a\u5668\u4e4b\u5fc3\u7f16\u8bd1\uff0c<strong><em><span>\u8f6c\u8f7d\u8bf7\u8054\u7cfb\u672c\u516c\u4f17\u53f7\u83b7\u5f97\u6388\u6743<\/span><\/em><\/strong><\/span><\/em><\/strong>\u3002<\/span><\/em><\/strong><\/p>\n<div class=\"articleend\"><\/div>\n","protected":false},"excerpt":{"rendered":"<p>\u9009\u81eaarXiv \u53c2\u4e0e\uff1a\u8def\u96ea \u672c\u8bba\u6587\u901a\u8fc7\u5f3a\u5316\u5b66\u4e60\u7684\u65b9\u5f0f\u91c7\u6837\u4e0d\u540c\u7684\u66f4\u65b0\u89c4\u5219\u800c\u5f97\u51fa\u66f4\u52a0\u4f18\u79c0\u7684\u4f18\u5316\u65b9\u6cd5\uff0c\u8fd9\u4e9b\u4e0d\u540c\u7684\u4f18\u5316 [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[183],"class_list":["post-116019","post","type-post","status-publish","format-standard","hentry","category-news","tag-183"],"_links":{"self":[{"href":"https:\/\/www.fagao.me\/p\/wp-json\/wp\/v2\/posts\/116019","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.fagao.me\/p\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.fagao.me\/p\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.fagao.me\/p\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.fagao.me\/p\/wp-json\/wp\/v2\/comments?post=116019"}],"version-history":[{"count":0,"href":"https:\/\/www.fagao.me\/p\/wp-json\/wp\/v2\/posts\/116019\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.fagao.me\/p\/wp-json\/wp\/v2\/media?parent=116019"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.fagao.me\/p\/wp-json\/wp\/v2\/categories?post=116019"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.fagao.me\/p\/wp-json\/wp\/v2\/tags?post=116019"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}