文章
65subprocess: python 2.7脚本中如何运行shell命令每次用的时候都要去读一下文档，这次总结一下，记录于此，已备后用。 其实这个主要是用subprocess这个模块。 最普通的，执行命令，但不管命令的输出： 设置"shell=True" 是告诉python在后台生成一个shell进程，然后在此shell进程中执行相应命令。 这样可以利用到shell中的一些特性，但这样做有安全风险，只有传入的命令完全可控时才这样做。 命令的返回值保存在return\c…66性能测试应该怎么做？偶然间看到了阿里中间件Dubbo的性能测试报告，我觉得这份性能测试报告让人觉得做这性能测试的人根本不懂性能测试 […]67Meson: Workflow Orchestration for Netflix Recommendations<div><span>At Netflix, our goal is to predict what you want to watch before you watch it. To do this, we run a large number of machine learning (ML) workflows every day. In order to support the creation of these workflows and make efficient use of resources, we created Meson.</span></div><b><br></b><br><div dir="ltr"><span>Meson is a general purpose workflow orchestration and scheduling framework that we built to manage ML pipelines that execute workloads across heterogeneous systems. It manages the lifecycle of several ML pipelines that build, train and validate personalization algorithms that drive video recommendations.</span></div><b><br></b><br><div dir="ltr"><span>One of the primary goals of Meson is to increase the velocity, reliability and repeatability of algorithmic experiments while allowing engineers to use the technology of their choice for each of the steps themselves.</span></div><h2><span>Powering Machine Learning Pipelines</span></h2><div dir="ltr"><span>Spark, MLlib, Python, R and Docker play an important role in several current generation machine learning pipelines within Netflix.</span></div><b><br></b><br><div dir="ltr"><span>Let’s take a look at a typical machine learning pipeline that drives video recommendations and how it is represented and handled in Meson.</span></div><b><br></b><br><div dir="ltr"><div><a href="https://4.bp.blogspot.com/-HXagVpFzsS8/V02LKQ-LwoI/AAAAAAAAAmE/W8VQSRSD4oEpjcXNl_He9VrgttAc9WFDQCLcB/s1600/fictitious_ml_pipeline.png" imageanchor="1"><img border="0" height="72" src="https://4.bp.blogspot.com/-HXagVpFzsS8/V02LKQ-LwoI/AAAAAAAAAmE/W8VQSRSD4oEpjcXNl_He9VrgttAc9WFDQCLcB/s640/fictitious_ml_pipeline.png" width="600"></a></div><span></span></div><div dir="ltr"><span>(click to enlarge)</span></div><b><br></b><br><div dir="ltr"><span>The workflow involves:</span></div><ul><li dir="ltr"><div dir="ltr"><span>Selecting a set of users - This is done via a Hive query to select the cohort for analysis</span></div></li><li dir="ltr"><div dir="ltr"><span>Cleansing / preparing the data - A Python script that creates 2 sets of users for ensuring parallel paths</span></div></li><li dir="ltr"><div dir="ltr"><span>In the parallel paths, one uses Spark to build and analyze a global model with HDFS as temporary storage.</span><span><br></span><span>The other uses R to build region (country) specific models. The number of regions is dynamic based on the cohort selected for analysis. The </span><span>Build Regional Model</span><span> and </span><span>Validate Regional Model</span><span> steps in the diagram are repeated for each region (country), expanded at runtime and executed with different set of parameters as shown below</span></div></li><li dir="ltr"><div dir="ltr"><span>Validation - Scala code that tests for the stability of the models when the two paths converge. In this step we also go back and repeat the whole process if the model is not stable.</span></div></li><li dir="ltr"><div dir="ltr"><span>Publish the new model - Fire off a Docker container to publish the new model to be picked up by other production systems</span></div></li></ul><b><br></b><br><div dir="ltr"><div><a href="https://2.bp.blogspot.com/-ZPrrcALx-6M/V02OY_kY5vI/AAAAAAAAAmY/Tuc38itrtfMh9bqZc9CYsJLKjfd4afcIACLcB/s1600/Screen%2BShot%2B2016-05-18%2Bat%2B1.44.32%2BPM.png" imageanchor="1"><img border="0" height="144" src="https://2.bp.blogspot.com/-ZPrrcALx-6M/V02OY_kY5vI/AAAAAAAAAmY/Tuc38itrtfMh9bqZc9CYsJLKjfd4afcIACLcB/s640/Screen%2BShot%2B2016-05-18%2Bat%2B1.44.32%2BPM.png" width="600"></a></div><span></span></div><div dir="ltr"><span>(click to enlarge)</span></div><b><br></b><br><div dir="ltr"><span>The above picture shows a run in progress for the workflow described above</span></div><ul><li dir="ltr"><div dir="ltr"><span>The user set selection, and cleansing of the data has been completed as indicated by the steps in green. </span></div></li><li dir="ltr"><div dir="ltr"><span>The parallel paths are in progress</span></div></li><ul><li dir="ltr"><div dir="ltr"><span>The Spark branch has completed the model generation and the validation</span></div></li><li dir="ltr"><div dir="ltr"><span>The for-each branch has kicked off 4 different regional models and all of them are in progress (Yellow)</span></div></li></ul><li dir="ltr"><div dir="ltr"><span>The Scala step for model selection is activated (Blue). This indicates that one or more of the incoming branches have completed, but it is still not scheduled for execution because there are incoming branches that have either (a) not started or (b) are in progress</span></div></li><li dir="ltr"><div dir="ltr"><span>Runtime context and parameters are passed along the workflow for business decisions</span></div></li></ul><h2 dir="ltr"><span>Under the Hood</span></h2><div dir="ltr"><span>Let’s dive behind the scenes to understand how Meson orchestrates across disparate systems and look at the interplay within different components of the ecosystem. Workflows have a varying set of resource requirements and expectations on total run time. We rely on resource managers like </span><a href="http://mesos.apache.org/"><span>Apache Mesos</span></a><span> to satisfy these requirements. Mesos provides task isolation and excellent abstraction of CPU, memory, storage, and other compute resources. Meson leverages these features to achieve scale and fault tolerance for its tasks.</span></div><b><br></b><br><div dir="ltr"><span><img height="367" src="https://lh5.googleusercontent.com/hnJK2wzD9-XuftXdGbZfrtA-MiPzx04oJ1HpsZXAiL5scTxftn3c0YYmqe-ZsihzTccVaTHG8yb1iicg_vvHRfmtcsq93NeGWyfLMuUEdkekLo4y-D0-Byh4lijCbV_Lyko0Csrk" width="624"></span></div><div dir="ltr"><span></span></div><div dir="ltr"><span>Meson Scheduler </span></div><div dir="ltr"><span>Meson scheduler, which is registered as a </span><a href="http://mesos.apache.org/documentation/latest/app-framework-development-guide/"><span>Mesos framework</span></a><span>, manages the launch, flow control and runtime of the various workflows. Meson delegates the actual resource scheduling to Mesos. Various requirements including memory and CPU are passed along to Mesos. While we do rely on Mesos for resource scheduling, the scheduler is designed to be pluggable, should one choose to use another framework for resource scheduling.</span></div><b><br></b><br><div dir="ltr"><span>Once a step is ready to be scheduled, the Meson scheduler chooses the right resource offer from Mesos and ships off the task to the Mesos master.</span></div><h3 dir="ltr"><span>Meson Executor</span></h3><div dir="ltr"><span>The Meson executor is a custom Mesos executor. Writing a custom executor allows us to maintain a communication channel with Meson. This is especially useful for long running tasks where framework messages can be sent to the Meson scheduler. This also enables us to pass custom data that’s richer than just exit codes or status messages.</span></div><b><br></b><br><div dir="ltr"><span>Once Mesos schedules a Meson task, it launches a Meson executor on a slave after downloading all task dependencies. While the core task is being executed, the executor does housekeeping chores like sending heartbeats, percent complete, status messages etc.</span></div><h3 dir="ltr"><span>DSL</span></h3><div dir="ltr"><span>Meson offers a Scala based DSL that allows for easy authoring of workflows. This makes it very easy for developers to use and create customized workflows. Here is how the aforementioned workflow may be defined using the DSL.</span></div><b><br></b><br><div dir="ltr"><span>val getUsers = Step("Get Users", ...)</span><span><br></span><span>val wrangleData = Step("Wrangle Data", ...)</span></div><div dir="ltr"><span>...</span></div><div dir="ltr"><span>val regionSplit = Step("For Each Region", ...)</span></div><div dir="ltr"><span>val regionJoin = Step("End For Each", ...)</span></div><div dir="ltr"><span>val regions = Seq("US", "Canada", "UK_Ireland", "LatAm", ...)</span><span><br></span></div><div dir="ltr"><span>val wf = start -> getUsers -> wrangleData ==> (</span></div><div dir="ltr"><span> trainGlobalModel -> validateGlobalModel,</span></div><div dir="ltr"><span> regionSplit **(reg = regions) --< (trainRegModel, validateRegModel) >-- regionJoin</span></div><div dir="ltr"><span>) >== selectModel -> validateModel -> end</span></div><b><br></b><br><div dir="ltr"><span>// If verbs are preferred over operators</span></div><div dir="ltr"><span>val wf = sequence(start, getUsers, wrangleData) parallel {</span></div><div dir="ltr"><span> sequence(trainGlobalModel, validateGlobalModel)</span></div><div dir="ltr"><span> sequence(regionSplit, </span></div><div dir="ltr"><span> forEach(reg = regions) sequence(trainRegModel, validateRegModel) forEach, </span></div><div dir="ltr"><span> regionJoin)</span></div><div dir="ltr"><span>} parallel sequence(selectModel, validateModel, end)</span></div><h3 dir="ltr"><span>Extension architecture</span></h3><div dir="ltr"><span>Meson was built from the ground up to be extensible to make it easy to add custom steps and extensions. Spark Submit Step, Hive Query Step, Netflix specific extensions that allow us to reach out to microservices or other systems like Cassandra are a some examples.</span></div><b><br></b><br><div dir="ltr"><span>In the above workflow, we built a Netflix specific extension to call out to our Docker execution framework that enables developers to specify the bare minimum parameters for their Docker images. The extension handles all communications like getting all the status URLs, the log messages and monitoring the state of the Docker process.</span></div><h3 dir="ltr"><span>Artifacts</span></h3><div dir="ltr"><span>Outputs of steps can be treated as first class citizens within Meson and are stored as Artifacts. Retries of a workflow step can be skipped based on the presence or absence of an artifact id. We can also have custom visualization of artifacts within the Meson UI. For e.g. if we store feature importance as an artifact as part of a pipeline, we can plug in custom visualizations that allow us to compare the past n days of the feature importance.</span></div><div dir="ltr"><span> </span></div><div dir="ltr"><span><img alt="Screen Shot 2016-05-27 at 4.01.02 PM.png" height="351" src="https://lh6.googleusercontent.com/1-k5Pdbl8xtVLYjqxjbSjIx8Qck6V4q2IQgRHPps9ZrLwN4pJpc01LE5oVPLeqpKWePgUMKqG_6XI-Uuwm8H8MQP3SFpNyNivAxIyC0o_3luV8wLRVlu1O4zI5VpzSV1FcXlZ2wK" width="624"></span></div><h3 dir="ltr"><span>Mesos Master / Slave</span></h3><div dir="ltr"><span>Mesos is used for resource scheduling with Meson registered as the core framework. Meson’s custom Mesos executors are deployed across the slaves. These are responsible for downloading all the jars and custom artifacts and send messages / context / heartbeats back to the Meson scheduler. Spark jobs submitted from Meson share the same Mesos slaves to run the tasks launched by the Spark job.</span></div><h3 dir="ltr"><span>Native Spark Support</span></h3><div dir="ltr"><span>Supporting Spark natively within Meson was a key requirement and goal. The Spark Submit within Meson allows for monitoring of the Spark job progress from within Meson, has the ability to retry failed spark steps or kill Spark jobs that may have gone astray. Meson also supports the ability to target specific Spark versions - thus, supporting innovation for users that want to be on the latest version of Spark.</span></div><b><br></b><br><div dir="ltr"><span>Supporting Spark in a multi-tenant environment via Meson came with an interesting set of challenges. Workflows have a varying set of resource requirements and expectations on total run time. Meson efficiently utilizes the available resources by matching the resource requirements and SLA expectation to a set of Mesos slaves that have the potential to meet the criteria. This is achieved by setting up labels for groups of Mesos slaves and using the Mesos resource attributes feature to target a job to a set of slaves.</span></div><h3 dir="ltr"><span>ML Constructs</span></h3><div dir="ltr"><span>As adoption increased for Meson, a class of large scale parallelization problems like parameters sweeping, complex bootstraps and cross validation emerged.</span></div><div dir="ltr"><span> </span></div><div dir="ltr"><span>Meson offers a simple ‘for-loop’ construct that allows data scientists and researchers to express parameter sweeps allowing them to run tens of thousands of docker containers across the parameter values. Users of this construct can monitor progress across the thousands of tasks in real time, find failed tasks via the UI and have logs streamed back to a single place within Meson making managing such parallel tasks simple.</span></div><h2 dir="ltr"><span>Conclusion</span></h2><div dir="ltr"><span>Meson has been powering hundreds of concurrent jobs across multiple ML pipelines for the past year. It has been a catalyst in enabling innovation for our algorithmic teams thus improving overall recommendations to our members.</span></div><b><br></b><br><div dir="ltr"><span>We plan to open source Meson in the coming months and build a community around it. If you want to help accelerate the pace of innovation and the open source efforts</span><span>, </span><a href="https://jobs.netflix.com/"><span>join us</span></a><span>.</span></div><b><br></b><br><div dir="ltr"><span>Here are some screenshots of the Meson UI:</span></div><b><br></b><br><div dir="ltr"><span>(click to enlarge)</span></div><div dir="ltr"><table><colgroup><col width="*"><col width="*"><col width="*"></colgroup><tbody><tr><td><div dir="ltr"><div><a href="https://2.bp.blogspot.com/-dkPDrEyP13o/V02PKl5ojFI/AAAAAAAAAmg/lhOCz5s39oEQa9GPbSkRZcrOywYvFWbUQCLcB/s1600/Screen%2BShot%2B2016-05-18%2Bat%2B2.51.57%2BPM.png" imageanchor="1"><img border="0" height="108" src="https://2.bp.blogspot.com/-dkPDrEyP13o/V02PKl5ojFI/AAAAAAAAAmg/lhOCz5s39oEQa9GPbSkRZcrOywYvFWbUQCLcB/s200/Screen%2BShot%2B2016-05-18%2Bat%2B2.51.57%2BPM.png" width="175"></a></div><span></span></div></td><td><div dir="ltr"><div><a href="https://3.bp.blogspot.com/-6KNti9NUus0/V02PjIM2EJI/AAAAAAAAAmk/QJ4Qot9693UmHHYznaWa_Rj8Q5AwOEa9wCLcB/s1600/Screen%2BShot%2B2016-05-18%2Bat%2B2.54.28%2BPM.png" imageanchor="1"><img border="0" height="105" src="https://3.bp.blogspot.com/-6KNti9NUus0/V02PjIM2EJI/AAAAAAAAAmk/QJ4Qot9693UmHHYznaWa_Rj8Q5AwOEa9wCLcB/s200/Screen%2BShot%2B2016-05-18%2Bat%2B2.54.28%2BPM.png" width="175"></a></div><span></span></div></td><td rowspan="3"><div dir="ltr"><div><a href="https://2.bp.blogspot.com/-Vx4OTs9qKDY/V02StN1wttI/AAAAAAAAAnk/g0WTw-C2BD8nBtHBBs9wzAVcntZrorHtACLcB/s1600/Screen%2BShot%2B2015-10-05%2Bat%2B9.41.50%2BAM.png" imageanchor="1"><img border="0" height="320" src="https://2.bp.blogspot.com/-Vx4OTs9qKDY/V02StN1wttI/AAAAAAAAAnk/g0WTw-C2BD8nBtHBBs9wzAVcntZrorHtACLcB/s320/Screen%2BShot%2B2015-10-05%2Bat%2B9.41.50%2BAM.png" width="156"></a></div><span></span></div><br><div dir="ltr"><div><a href="https://3.bp.blogspot.com/-91Sm0ru0lFI/V02S6B4iM-I/AAAAAAAAAno/3V16X66XatQPyvHp6elGWau2kbhzJhm0ACLcB/s1600/Screen%2BShot%2B2015-10-05%2Bat%2B9.41.13%2BAM.png" imageanchor="1"><img border="0" height="28" src="https://3.bp.blogspot.com/-91Sm0ru0lFI/V02S6B4iM-I/AAAAAAAAAno/3V16X66XatQPyvHp6elGWau2kbhzJhm0ACLcB/s200/Screen%2BShot%2B2015-10-05%2Bat%2B9.41.13%2BAM.png" width="175"></a></div><span></span></div></td></tr><tr><td><div dir="ltr"><div><a href="https://1.bp.blogspot.com/-tcz3fMAlVWw/V02PynmTjfI/AAAAAAAAAmo/iOThHWn5Fs4uQ9hHPCypX7CXhU5rhVQtQCLcB/s1600/Screen%2BShot%2B2016-05-18%2Bat%2B2.59.40%2BPM.png" imageanchor="1"><img border="0" height="117" src="https://1.bp.blogspot.com/-tcz3fMAlVWw/V02PynmTjfI/AAAAAAAAAmo/iOThHWn5Fs4uQ9hHPCypX7CXhU5rhVQtQCLcB/s200/Screen%2BShot%2B2016-05-18%2Bat%2B2.59.40%2BPM.png" width="175"></a></div><span></span></div></td><td><div dir="ltr"><div><a href="https://4.bp.blogspot.com/-BXtgTNcUVOM/V02P8CWO32I/AAAAAAAAAmw/wpR_9D7gUm0DRIsu33fSyPQgUIYJvS5mwCLcB/s1600/Screen%2BShot%2B2016-05-18%2Bat%2B3.15.44%2BPM.png" imageanchor="1"><img border="0" height="125" src="https://4.bp.blogspot.com/-BXtgTNcUVOM/V02P8CWO32I/AAAAAAAAAmw/wpR_9D7gUm0DRIsu33fSyPQgUIYJvS5mwCLcB/s200/Screen%2BShot%2B2016-05-18%2Bat%2B3.15.44%2BPM.png" width="175"></a></div><span></span></div></td></tr><tr><td><div dir="ltr"><div><a href="https://4.bp.blogspot.com/-pyFr9cFqunw/V02RkKepTjI/AAAAAAAAAnI/JalUhg06mc0P7mjmqwTligGOabBkytEwQCLcB/s1600/Screen%2BShot%2B2016-05-27%2Bat%2B9.13.48%2BPM.png" imageanchor="1"><img border="0" height="83" src="https://4.bp.blogspot.com/-pyFr9cFqunw/V02RkKepTjI/AAAAAAAAAnI/JalUhg06mc0P7mjmqwTligGOabBkytEwQCLcB/s200/Screen%2BShot%2B2016-05-27%2Bat%2B9.13.48%2BPM.png" width="175"></a></div><span></span></div><br><div dir="ltr"><div><a href="https://4.bp.blogspot.com/-h4FF8FEHVBs/V02SEgesFLI/AAAAAAAAAnM/y6YMRy0KZCUjPkPGHIpHfzekoiKEdg5rgCLcB/s1600/Screen%2BShot%2B2016-05-18%2Bat%2B4.04.24%2BPM.png" imageanchor="1"><img border="0" height="114" src="https://4.bp.blogspot.com/-h4FF8FEHVBs/V02SEgesFLI/AAAAAAAAAnM/y6YMRy0KZCUjPkPGHIpHfzekoiKEdg5rgCLcB/s200/Screen%2BShot%2B2016-05-18%2Bat%2B4.04.24%2BPM.png" width="175"></a></div><span></span></div></td><td><div dir="ltr"><div><a href="https://3.bp.blogspot.com/-tC0hb1b5tzA/V02SaDWljWI/AAAAAAAAAnc/knh-jCT2w5EVk7neJcnpa81SU3vVD886wCLcB/s1600/Screen%2BShot%2B2015-08-05%2Bat%2B6.55.21%2BPM.png" imageanchor="1"><img border="0" height="200" src="https://3.bp.blogspot.com/-tC0hb1b5tzA/V02SaDWljWI/AAAAAAAAAnc/knh-jCT2w5EVk7neJcnpa81SU3vVD886wCLcB/s200/Screen%2BShot%2B2015-08-05%2Bat%2B6.55.21%2BPM.png" width="193"></a></div><span></span></div></td></tr></tbody></table></div><br><div dir="ltr"><span>- </span><a href="https://www.linkedin.com/in/aasamy"><span>Antony Arokiasamy</span></a><span>, </span><span><a href="https://www.linkedin.com/in/kedar-sadekar-2544052"><span>Kedar Sadekar</span></a></span><span>, </span><a href="https://www.linkedin.com/in/raju-uppalapati-0782001"><span>Raju Uppalapati</span></a><span>, </span><a href="https://www.linkedin.com/in/sathish-sridharan-b7a9314"><span>Sathish Sridharan</span></a><span>, </span><a href="https://www.linkedin.com/in/prasannapadmanabhan"><span>Prasanna Padmanabhan</span></a><span>, </span><a href="https://www.linkedin.com/in/prassh123"><span>Prashanth Raghavan</span></a><span>, </span><a href="https://www.linkedin.com/in/fzsiddiqi"><span>Faisal Zakaria Siddiqi</span></a><span>, </span><a href="https://www.linkedin.com/in/ellchow"><span>Elliot Chow</span></a><span> and "a man has no linkedin" (aka Davis Shepherd) for the Meson Team</span><br><span><br></span><span><br></span></div>68一些关于链接和加载的博客COMPILER, ASSEMBLER, LINKER AND LOADER: A BRIEF STORY Linkers - part I （总共20） Linking -- chapter 7 of Computer Systems: a programmers' perspective69SmartOS上更新虚拟机配置这里以一个SmartOS zone更新IP及网关为例。
用到的命令是vmadm update UUID < json.file。相关配置都需要以json的格式给出，这个我有点不习惯。或许还有其它方式，不过我不了解。70SmartOS上如何创建一个SmartOS zone如何在SmartOS上创建zones71Triton headnode的状态检查headnode启动后可以运行sdc-healthcheck命令来查看各个服务的状态。 正常应该都是online的状态。像上面的输出中，有两个zone处于svc-err，意味着这两个zone上有服务出现错误。我们可以登陆上去检查、修复。 sdc-login redis用于登陆的redis这个zone上；svcs -xv列出有错误的服务；可以看到是redis服务启动失败；svcadm restart…72Solaris 11手工网络配置Solaris 11 手动IP配置方式。也适用于smartos