Details
-
Bug
-
Status: Resolved
-
Minor
-
Resolution: Fixed
-
1.2.0
-
None
Description
Currently operations agains provisioner/docker/config/hosts may result in Permision Denied if running docker provisioner as non-root user.
+ ./gradlew -Pconfig=config_ubuntu_xenial_Bigtop-trunk-deployments.yaml -Pnum_instances=1 docker-provisioner :buildSrc:compileJava UP-TO-DATE :buildSrc:compileGroovy UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar UP-TO-DATE :buildSrc:assemble UP-TO-DATE :buildSrc:compileTestJava UP-TO-DATE :buildSrc:compileTestGroovy UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build UP-TO-DATE :docker-provisioner Environment check... Check docker: Docker version 1.11.2, build b9f10c9/1.11.2 Check docker-compose: docker-compose version 1.6.2, build 4d72027 Check ruby: ruby 2.0.0p648 (2015-12-16) [x86_64-linux] Creating and starting 1 ... Pulling bigtop (bigtop/puppet:ubuntu-16.04)... ubuntu-16.04: Pulling from bigtop/puppet Digest: sha256:71d7918e5458527c35c82eabca36401d5a58d0ac3bf6fbc99d0fe303535d6de4 Status: Downloaded newer image for bigtop/puppet:ubuntu-16.04 [1A[2K Creating and starting 1 ... done [1BBigtop Puppet configurations are shared between instances, and can be modified under config/hieradata ‘../../bigtop-deploy/puppet/hieradata/site.yaml’ -> ‘./config/hieradata/site.yaml’ ‘../../bigtop-deploy/puppet/hieradata/bigtop’ -> ‘./config/hieradata/bigtop’ ‘../../bigtop-deploy/puppet/hieradata/bigtop/ha.yaml’ -> ‘./config/hieradata/bigtop/ha.yaml’ ‘../../bigtop-deploy/puppet/hieradata/bigtop/noha.yaml’ -> ‘./config/hieradata/bigtop/noha.yaml’ ‘../../bigtop-deploy/puppet/hieradata/bigtop/cluster.yaml’ -> ‘./config/hieradata/bigtop/cluster.yaml’ ./docker-hadoop.sh: line 81: ./config/hosts: Permission denied Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: rng-tools 0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded. Need to get 21.9 kB of archives. After this operation, 139 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu xenial/universe amd64 rng-tools amd64 5-0ubuntu3 [21.9 kB] debconf: unable to initialize frontend: Dialog debconf: (TERM is not set, so the dialog frontend is not usable.) debconf: falling back to frontend: Readline debconf: unable to initialize frontend: Readline debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.22.1 /usr/local/share/perl/5.22.1 /usr/lib/x86_64-linux-gnu/perl5/5.22 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.22 /usr/share/perl/5.22 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7, <> line 1.) debconf: falling back to frontend: Teletype dpkg-preconfigure: unable to re-open stdin: Fetched 21.9 kB in 0s (89.8 kB/s) Selecting previously unselected package rng-tools. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 13889 files and directories currently installed.) Preparing to unpack .../rng-tools_5-0ubuntu3_amd64.deb ... Unpacking rng-tools (5-0ubuntu3) ... Processing triggers for systemd (229-4ubuntu12) ... Setting up rng-tools (5-0ubuntu3) ... Trying to create /dev/hwrng device inode... invoke-rc.d: policy-rc.d denied execution of start. Processing triggers for systemd (229-4ubuntu12) ... Reading package lists... Building dependency tree... Reading state information... The following NEW packages will be installed: apt-transport-https 0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded. Need to get 26.0 kB of archives. After this operation, 213 kB of additional disk space will be used. Get:1 http://archive.ubuntu.com/ubuntu xenial-updates/main amd64 apt-transport-https amd64 1.2.15ubuntu0.2 [26.0 kB] debconf: unable to initialize frontend: Dialog debconf: (TERM is not set, so the dialog frontend is not usable.) debconf: falling back to frontend: Readline debconf: unable to initialize frontend: Readline debconf: (Can't locate Term/ReadLine.pm in @INC (you may need to install the Term::ReadLine module) (@INC contains: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.22.1 /usr/local/share/perl/5.22.1 /usr/lib/x86_64-linux-gnu/perl5/5.22 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.22 /usr/share/perl/5.22 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .) at /usr/share/perl5/Debconf/FrontEnd/Readline.pm line 7, <> line 1.) debconf: falling back to frontend: Teletype dpkg-preconfigure: unable to re-open stdin: Fetched 26.0 kB in 0s (107 kB/s) Selecting previously unselected package apt-transport-https. (Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 13904 files and directories currently installed.) Preparing to unpack .../apt-transport-https_1.2.15ubuntu0.2_amd64.deb ... Unpacking apt-transport-https (1.2.15ubuntu0.2) ... Setting up apt-transport-https (1.2.15ubuntu0.2) ... local apt = false ; NOT Enabling local apt. Packages will be pulled from remote... [1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Hash. There is further documentation for validate_legacy function in the README. (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m [1;31mWarning: This method is deprecated, please use match expressions with Stdlib::Compat::Bool instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m [1;31mWarning: This method is deprecated, please use match expressions with Stdlib::Compat::Array instead. They are described at https://docs.puppet.com/puppet/latest/reference/lang_data_type.html#match-expressions. (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m [1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Pattern[]. There is further documentation for validate_legacy function in the README. (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m [1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::Bool. There is further documentation for validate_legacy function in the README. (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m [1;31mWarning: This method is deprecated, please use the stdlib validate_legacy function, with Stdlib::Compat::String. There is further documentation for validate_legacy function in the README. (at /etc/puppet/modules/stdlib/lib/puppet/functions/deprecation.rb:25:in `deprecation')[0m [mNotice: Compiled catalog for f1823aaec65e.bigtop.apache.org in environment production in 2.51 seconds[0m [mNotice: /Stage[main]/Hadoop_cluster_node/User[testuser]/ensure: created[0m [mNotice: /Stage[main]/Hadoop_cluster_node/User[jenkins]/ensure: created[0m [mNotice: /Stage[main]/Hadoop::Namenode/Hadoop::Create_storage_dir[/data/2/namenode]/Exec[mkdir /data/2/namenode]/returns: executed successfully[0m [mNotice: /Stage[main]/Hadoop::Namenode/Hadoop::Create_storage_dir[/data/1/namenode]/Exec[mkdir /data/1/namenode]/returns: executed successfully[0m [mNotice: /Stage[main]/Hadoop::Mapred_app/Hadoop::Create_storage_dir[/data/2/mapred]/Exec[mkdir /data/2/mapred]/returns: executed successfully[0m [mNotice: /Stage[main]/Hadoop::Mapred_app/Hadoop::Create_storage_dir[/data/1/mapred]/Exec[mkdir /data/1/mapred]/returns: executed successfully[0m [mNotice: /Stage[main]/Main/Apt::Conf[disable_keys]/Apt::Setting[conf-disable_keys]/File[/etc/apt/apt.conf.d/50disable_keys]/ensure: created[0m [mNotice: /Stage[main]/Hadoop_cluster_node/User[hudson]/ensure: created[0m [mNotice: /Stage[main]/Apt/Apt::Setting[conf-update-stamp]/File[/etc/apt/apt.conf.d/15update-stamp]/ensure: defined content as '{md5}0962d70c4ec78bbfa6f3544ae0c41974'[0m [mNotice: /Stage[main]/Main/Apt::Source[Bigtop]/Apt::Setting[list-Bigtop]/File[/etc/apt/sources.list.d/Bigtop.list]/ensure: created[0m [mNotice: /Stage[main]/Apt/File[preferences]/ensure: created[0m [mNotice: /Stage[main]/Apt::Update/Exec[apt_update]: Triggered 'refresh' from 1 events[0m [mNotice: /Stage[main]/Spark::Master/Package[spark-master]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Spark::Common/Package[spark-datanucleus]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Spark::Worker/Package[spark-worker]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Spark::Common/File[/etc/spark/conf/spark-defaults.conf]/ensure: defined content as '{md5}a3a08db3bf31654e86b22a51928c23e7'[0m [mNotice: /Stage[main]/Spark::Common/File[/etc/spark/conf/log4j.properties]/ensure: defined content as '{md5}89ef3bfce554862a78bf4f2e1dafd4f6'[0m [mNotice: /Stage[main]/Main/Package[jdk]/ensure: ensure changed 'purged' to 'present'[0m [mNotice: /Stage[main]/Hadoop::Resourcemanager/Package[hadoop-yarn-resourcemanager]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Hadoop::Namenode/Package[hadoop-hdfs-namenode]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Hadoop::Namenode/File[/etc/default/hadoop-hdfs-namenode]/content: content changed '{md5}0c3a4aff527c8fb16d826bf59c9bb985' to '{md5}8aaf7169ae9f4cf6863f2e8c96e8ecb8'[0m [mNotice: /Stage[main]/Hadoop::Proxyserver/Package[hadoop-yarn-proxyserver]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Hadoop::Historyserver/Package[hadoop-mapreduce-historyserver]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Hadoop::Common/File[/etc/hadoop/conf/hadoop-env.sh]/content: content changed '{md5}829beb2d3c16fa22040afdc0fa277ba1' to '{md5}b4e97a45010e60de2c7d4ca3abecfdde'[0m [mNotice: /Stage[main]/Hadoop::Mapred_app/File[/data/2/mapred]/owner: owner changed 'root' to 'yarn'[0m [mNotice: /Stage[main]/Hadoop::Mapred_app/File[/data/2/mapred]/group: group changed 'root' to 'yarn'[0m [mNotice: /Stage[main]/Hadoop::Client/Package[hadoop-doc]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Hadoop::Client/Package[libhdfs0-dev]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Hadoop::Client/Package[hadoop-hdfs-fuse]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Hadoop::Common_hdfs/File[/etc/hadoop/conf/core-site.xml]/content: content changed '{md5}315d2770c60bc9a0b3c7dc04e41fda76' to '{md5}285212e176c67db3db3670500de81713'[0m [mNotice: /Stage[main]/Hadoop::Common_mapred_app/File[/etc/hadoop/conf/taskcontroller.cfg]/ensure: defined content as '{md5}fa9d342175e04ccf0fa1aba8291bedba'[0m [mNotice: /Stage[main]/Hadoop::Common_yarn/File[/etc/hadoop/conf/container-executor.cfg]/content: content changed '{md5}8afd041c79a90945ebfdd10ccbc43d9d' to '{md5}6aa9a3efa8965dfee06966b8fcb22e19'[0m [mNotice: /Stage[main]/Hadoop::Common_hdfs/File[/etc/hadoop/conf/hdfs-site.xml]/content: content changed '{md5}06f0cc5cad0018c16ef3dcb404efb74e' to '{md5}da8fee38e5068bb587ef909127118f01'[0m [mNotice: /Stage[main]/Hadoop::Common_mapred_app/File[/etc/hadoop/conf/mapred-site.xml]/content: content changed '{md5}1b058ae210b2e2dafd3407b6bf7dad43' to '{md5}1ce1de27de34ef29d33d3d4ade6b0019'[0m [mNotice: /Stage[main]/Hadoop::Common_yarn/File[/etc/hadoop/conf/yarn-site.xml]/content: content changed '{md5}f679a7750b76612f7468f61a7c6b75e7' to '{md5}cab241efcdeba5517e1555eb33204a5c'[0m [mNotice: /Stage[main]/Hadoop::Proxyserver/Service[hadoop-yarn-proxyserver]/ensure: ensure changed 'stopped' to 'running'[0m [mNotice: /Stage[main]/Hadoop::Resourcemanager/Service[hadoop-yarn-resourcemanager]/ensure: ensure changed 'stopped' to 'running'[0m [mNotice: /Stage[main]/Hadoop::Historyserver/Service[hadoop-mapreduce-historyserver]/ensure: ensure changed 'stopped' to 'running'[0m [mNotice: /Stage[main]/Hadoop::Nodemanager/Package[hadoop-yarn-nodemanager]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Hadoop::Nodemanager/Hadoop::Create_storage_dir[/data/1/yarn]/Exec[mkdir /data/1/yarn]/returns: executed successfully[0m [mNotice: /Stage[main]/Hadoop::Nodemanager/Hadoop::Create_storage_dir[/data/2/yarn]/Exec[mkdir /data/2/yarn]/returns: executed successfully[0m [mNotice: /Stage[main]/Hadoop::Nodemanager/File[/data/2/yarn]/owner: owner changed 'root' to 'yarn'[0m [mNotice: /Stage[main]/Hadoop::Nodemanager/File[/data/2/yarn]/group: group changed 'root' to 'yarn'[0m [mNotice: /Stage[main]/Hadoop::Nodemanager/File[/data/1/yarn]/owner: owner changed 'root' to 'yarn'[0m [mNotice: /Stage[main]/Hadoop::Nodemanager/File[/data/1/yarn]/group: group changed 'root' to 'yarn'[0m [mNotice: /Stage[main]/Hadoop::Nodemanager/Service[hadoop-yarn-nodemanager]/ensure: ensure changed 'stopped' to 'running'[0m [mNotice: /Stage[main]/Hadoop::Namenode/File[/data/1/namenode]/owner: owner changed 'root' to 'hdfs'[0m [mNotice: /Stage[main]/Hadoop::Namenode/File[/data/1/namenode]/group: group changed 'root' to 'hdfs'[0m [mNotice: /Stage[main]/Hadoop::Namenode/File[/data/1/namenode]/mode: mode changed '0755' to '0700'[0m [mNotice: /Stage[main]/Hadoop::Namenode/File[/data/2/namenode]/owner: owner changed 'root' to 'hdfs'[0m [mNotice: /Stage[main]/Hadoop::Namenode/File[/data/2/namenode]/group: group changed 'root' to 'hdfs'[0m [mNotice: /Stage[main]/Hadoop::Namenode/File[/data/2/namenode]/mode: mode changed '0755' to '0700'[0m [mNotice: /Stage[main]/Hadoop::Namenode/Exec[namenode format]/returns: executed successfully[0m [mNotice: /Stage[main]/Hadoop::Namenode/Service[hadoop-hdfs-namenode]/ensure: ensure changed 'stopped' to 'running'[0m [mNotice: /Stage[main]/Hadoop::Datanode/Package[hadoop-hdfs-datanode]/ensure: ensure changed 'purged' to 'latest'[0m [mNotice: /Stage[main]/Hadoop::Datanode/Hadoop::Create_storage_dir[/data/2/hdfs]/Exec[mkdir /data/2/hdfs]/returns: executed successfully[0m [mNotice: /Stage[main]/Hadoop::Datanode/Hadoop::Create_storage_dir[/data/1/hdfs]/Exec[mkdir /data/1/hdfs]/returns: executed successfully[0m [mNotice: /Stage[main]/Hadoop::Datanode/File[/data/1/hdfs]/owner: owner changed 'root' to 'hdfs'[0m [mNotice: /Stage[main]/Hadoop::Datanode/File[/data/1/hdfs]/group: group changed 'root' to 'hdfs'[0m [mNotice: /Stage[main]/Hadoop::Datanode/File[/data/2/hdfs]/owner: owner changed 'root' to 'hdfs'[0m [mNotice: /Stage[main]/Hadoop::Datanode/File[/data/2/hdfs]/group: group changed 'root' to 'hdfs'[0m [mNotice: /Stage[main]/Hadoop::Datanode/File[/etc/default/hadoop-hdfs-datanode]/content: content changed '{md5}0c3a4aff527c8fb16d826bf59c9bb985' to '{md5}8aaf7169ae9f4cf6863f2e8c96e8ecb8'[0m [mNotice: /Stage[main]/Hadoop::Datanode/Service[hadoop-hdfs-datanode]/ensure: ensure changed 'stopped' to 'running'[0m [mNotice: /Stage[main]/Hadoop::Init_hdfs/Exec[init hdfs]/returns: executed successfully[0m [mNotice: /Stage[main]/Hadoop::Mapred_app/File[/data/1/mapred]/owner: owner changed 'root' to 'yarn'[0m [mNotice: /Stage[main]/Hadoop::Mapred_app/File[/data/1/mapred]/group: group changed 'root' to 'yarn'[0m [mNotice: /Stage[main]/Spark::Common/File[/etc/spark/conf/spark-env.sh]/content: content changed '{md5}0532a742483de44c845a66794a696e5e' to '{md5}5043b3e515ec7dd34ac4d0cb848aa2e6'[0m [mNotice: /Stage[main]/Spark::Worker/Service[spark-worker]/ensure: ensure changed 'stopped' to 'running'[0m [mNotice: /Stage[main]/Spark::Master/Service[spark-master]/ensure: ensure changed 'stopped' to 'running'[0m [mNotice: Finished catalog run in 221.64 seconds[0m BUILD SUCCESSFUL Total time: 4 mins 46.916 secs This build could be faster, please consider using the Gradle Daemon: https://docs.gradle.org/2.12/userguide/gradle_daemon.html + ./gradlew docker-provisioner-destroy :buildSrc:compileJava UP-TO-DATE :buildSrc:compileGroovy UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar UP-TO-DATE :buildSrc:assemble UP-TO-DATE :buildSrc:compileTestJava UP-TO-DATE :buildSrc:compileTestGroovy UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build UP-TO-DATE :docker-provisioner-destroyThe DOCKER_IMAGE variable is not set. Defaulting to a blank string. The MEM_LIMIT variable is not set. Defaulting to a blank string. The DOCKER_IMAGE variable is not set. Defaulting to a blank string. The MEM_LIMIT variable is not set. Defaulting to a blank string. Stopping 20170506094608r8013_bigtop_1 ... [1A[2K Stopping 20170506094608r8013_bigtop_1 ... done [1BThe DOCKER_IMAGE variable is not set. Defaulting to a blank string. The MEM_LIMIT variable is not set. Defaulting to a blank string. Removing 20170506094608r8013_bigtop_1 ... [1A[2K Removing 20170506094608r8013_bigtop_1 ... done [1B Going to remove 20170506094608r8013_bigtop_1 ./docker-hadoop.sh: line 130: ./config/hosts: Permission denied removed ‘./config/hieradata/bigtop/ha.yaml’ removed ‘./config/hieradata/bigtop/noha.yaml’ removed ‘./config/hieradata/bigtop/cluster.yaml’ removed directory: ‘./config/hieradata/bigtop’ removed ‘./config/hieradata/site.yaml’ removed ‘.provision_id’ BUILD SUCCESSFUL Total time: 32.182 secs
The way to fix it is to operate the file inside container.
Meanwhile, wee need to unmount /etc/hosts and remove the file before container destroyed. Otherwise there will be a root:root file left in the repository which can not be remove by user w/o sudo permission.
Attachments
Issue Links
- links to
