ambari 自定义服务

Ambari自定义服务学习

集成kafka-manager到ambari

存入的地址:

cd /var/lib/ambari-server/resources/stacks/HDP/2.5/services

最后的文件结构

KAFKA_MANAGER
├── configuration
│   └── kafka-manager-env.xml
├── metainfo.xml
├── package
│   ├── archive.zip
│   └── scripts
│       └── master.py
└── quicklinks
    └── quicklinks.json

1. 写metainfo.xml

<?xml version="1.0"?>
<metainfo>
  <schemaVersion>2.0</schemaVersion>
  <services>
    <service>
      <!-- Internal name for service (must be unique) -->
      <name>KAFKA_MANAGER</name>
      <!-- display name in Ambari UI -->
      <displayName>KAFKA_MANAGER</displayName>
      <!-- Description of service - will be displayed when user clicks add service -->
      <comment>the kafka's manager</comment>
      <!-- Version of service -->
      <version>1.3.2.1</version>
      <components>
        <!-- In this case, there is only one master component -->
        <component>
          <name>KAKFA_MASTER</name>
          <displayName>KAFKA_MANAGER</displayName>
          <category>MASTER</category>
          <!-- how many of these components are allowed in a cluster -->
          <cardinality>1</cardinality>
          <!-- reference to (and details of) what script is to be used to install/stop/start/config the service -->
          <commandScript>
            <script>scripts/master.py</script>
            <scriptType>PYTHON</scriptType>
            <timeout>600</timeout>
          </commandScript>
        </component>
      </components>
      <!-- 需要安装的包，依赖的 操作系统 -->
      <osSpecifics>
        <osSpecific>
          <osFamily>any</osFamily>
          <packages></packages>
        </osSpecific>
      </osSpecifics>
      <!-- 配置文件 -->
      <configuration-dependencies>
        <config-type>kafka-manager-env</config-type>
      </configuration-dependencies>
      <!-- 修改配置后，是否需要重启服务 -->
      <restartRequiredAfterChange>false</restartRequiredAfterChange>
      <!-- 快速链接 -->
      <quickLinksConfigurations>
        <quickLinksConfiguration>
          <fileName>quicklinks.json</fileName>
          <default>true</default>
        </quickLinksConfiguration>
      </quickLinksConfigurations>
    </service>
  </services>
</metainfo>

2. 写脚本master.py

#!/usr/bin/env python
from resource_management import *

class Master(Script):
    # 安装生命周期，本例子安装不做任何事情，使用安装好的包
    def install(self, env):
        #self.install_packages(env)
        print 'Install the KAFKA Master'
        
    # 停止动作，kill -9 pid，并删除pid文件
    def stop(self, env):
        Execute('ps -ef | grep kafka-manager | grep -v grep | awk  \'{print $2}\' | xargs kill -9')
        Execute('rm -f /opt/kafka-manager-1.3.2.1/RUNNING_PID')
        print 'Stop the KAFKA Master'
        
    # 启动动作，使用shell命令nohup后台启动  
    def start(self, env):
        Execute('nohup /opt/kafka-manager-1.3.2.1/bin/kafka-manager -Dconfig.file=/opt/kafka-manager-1.3.2.1/conf/application.conf -Dhttp.port=6888 > /opt/kafka-manager-1.3.2.1/kafka-manager.log 2>&1 &')

    # 检查状态脚本，使用ambari自带方法，参数给pid文件
    def status(self, env):
        check_process_status('/opt/kafka-manager-1.3.2.1/RUNNING_PID')
        
    # config文件
    def configure(self, env):
        print 'Configure the KAFKA Master';

if __name__ == "__main__":
    Master().execute()

3. 写quicklinks.json文件

{
  "name": "default",
  "description": "default quick links configuration",
  "configuration": {
    "protocol":
    {
      "type":"https",
      "checks":[
        {
          "property":"ssl_enable",
          "desired":"true",
          "site":"kafka-manager-env"
        }
      ]
    },

    "links": [
      {
        "name": "manager_ui",
        "label": "Kafka Manager",
        "requires_user_name": "false",
        "component_name": "KAKFA_MASTER",
        "url":"%@://%@:%@/",
        "port":{
          "http_property": "kafka.server.port",
          "http_default_port": "6888",
          "https_property": "kafka.server.port",
          "https_default_port": "6888",
          "regex": "^(\\d+)$",
          "site": "kafka-manager-env"
        }
      }
    ]
  }
}

4. 写配置文件

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
   Licensed to the Apache Software Foundation (ASF) under one or more
   contributor license agreements.  See the NOTICE file distributed with
   this work for additional information regarding copyright ownership.
   The ASF licenses this file to You under the Apache License, Version 2.0
   (the "License"); you may not use this file except in compliance with
   the License.  You may obtain a copy of the License at

       http://www.apache.org/licenses/LICENSE-2.0

   Unless required by applicable law or agreed to in writing, software
   distributed under the License is distributed on an "AS IS" BASIS,
   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
   See the License for the specific language governing permissions and
   limitations under the License.
-->
<configuration>
  <property>
    <name>kafka.server.port</name>
    <value>6888</value>
    <display-name>HTTP Post</display-name>
    <description>Webserver listens on this port</description>
  </property>
</configuration>  

5. 重启ambari-server

ambari-server restart

6. 在WEB上 ADD SERVICE

可以看到KAFKA_MANAGER在我们的service列表了

点击下一步，添加KAFKA_MANAGER SERVICE

并试着去点击STOP，Restart All，Quicklinks功能是否正常

验证没有问题

---

集成Java程序

Java程序文件结构，lib包太多不列出，把相关jar包全部放入lib包下，暂时只支持Xml文件格式

javaSpringboot/
├── config
│   ├── application.yml
│   └── java-config.xml
├── lib
├── log
│   └── stdio.log
└── javaControl.py

其中javaControl.py是自己写的python脚本，用来方便启动和关闭java程序
python脚本参考博客网站地址
博客作者写的还带一个程序挂了，有重启的功能，这里省略

#!/usr/bin/env python
# -*- coding: utf-8 -*-
import commands
import os
import sys
import subprocess

_programName = "springboot"  # 自定义程序名称
_programClassName = "com.sharing.App"  # java程序main Class
_pythonName = sys.argv[0]  # 获得脚本名称
_workPlace = os.getcwd() + os.sep  # 获得工作目录/当前目录
_pidFileName = "pid.txt"  # 程序pid写入文件


# 介绍脚本使用方法入口
def getProgramHelpIntroduce():
    print "请使用%s help查看，如何使用脚本操作程序" % _pythonName


# 脚本使用说明书
def getProgramHelpCommand():
    print "脚本提供三个命令如下： "
    print "启动程序命令: %s start" % _pythonName
    print "关闭程序命令: %s stop" % _pythonName
    print "重启程序命令: %s restart" % _pythonName


# pid文件是否存在
def pidIsExist():
    return os.path.exists(_workPlace + _pidFileName)


# 开启程序
def startProgram():
    if pidIsExist():
        print "程序正在运行，请先关闭程序！"
        getProgramHelpIntroduce()
    else:
        print "程序开始启动..."
        os.system(
            "exec -a %s java -Xmx1024m -cp config/:lib/* %s >> log/stdio.log 2>&1 & echo $! > %s &" % (
                _programName, _programClassName, _workPlace + _pidFileName))
        print "程序运行成功！"


# 关闭程序
def stopProgram():
    if not pidIsExist():
        print "程序没有运行！"
        os.system("rm -f %s" % _workPlace + _pidFileName)
        getProgramHelpIntroduce()
    else:
        pid = int(commands.getoutput(" cat %s " % (_workPlace + _pidFileName)))
        os.system("kill %s" % pid)
        os.system("rm -f %s" % _workPlace + _pidFileName)
        print "程序关闭成功！"


if __name__ == "__main__":
    if len(sys.argv) == 2:
        args = sys.argv[1]
    else:
        args = raw_input("Please Enter Your Command: ")

    logPath = _workPlace + "log"

    if not os.path.exists(logPath):
        os.mkdir(logPath)

    if args == "start":
        startProgram()
    elif args == "stop":
        stopProgram()
    elif args == "restart":
        stopProgram()
        startProgram()
    elif args == "help":
        getProgramHelpCommand()
    else:
        getProgramHelpIntroduce()

接下来做ambari集成java，java-config是从上面的java程序拷贝过来的
文件结构

JAVA_TEST/
├── configuration
│   └── java-config.xml
├── metainfo.xml
├── package
│   └── scripts
│       ├── master.py
│       └── params.py

metainfo.xml

<?xml version="1.0"?>
<metainfo>
  <schemaVersion>2.0</schemaVersion>
  <services>
    <service>
      <!-- Internal name for service (must be unique) -->
      <name>JAVA_TEST</name>
      <!-- display name in Ambari UI -->
      <displayName>JAVA_TEST</displayName>
      <!-- Description of service - will be displayed when user clicks add service -->
      <comment>a java application test</comment>
      <!-- Version of service -->
      <version>1.0.0.test</version>
      <components>
        <!-- In this case, there is only one master component -->
        <component>
          <name>JAVA_MASTER</name>
          <displayName>JAVA_MAIN</displayName>
          <category>MASTER</category>
          <!-- how many of these components are allowed in a cluster -->
          <cardinality>1</cardinality>
          <!-- reference to (and details of) what script is to be used to install/stop/start/config the service -->
          <commandScript>
            <script>scripts/master.py</script>
            <scriptType>PYTHON</scriptType>
            <timeout>600</timeout>
          </commandScript>
        </component>
      </components>
      <osSpecifics>
        <osSpecific>
          <osFamily>any</osFamily>
          <packages></packages>
        </osSpecific>
      </osSpecifics>
      <configuration-dependencies>
        <config-type>java-config</config-type>
      </configuration-dependencies>
      <restartRequiredAfterChange>false</restartRequiredAfterChange>
      <quickLinksConfigurations>
        <quickLinksConfiguration>
          <fileName>quicklinks.json</fileName>
          <default>true</default>
        </quickLinksConfiguration>
      </quickLinksConfigurations>
    </service>
  </services>
</metainfo>

params.py

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from resource_management.libraries.script.script import Script

config = Script.get_config()

java_config = config['configurations']['java-config']

master.py

#!/usr/bin/env python
from resource_management import *
import os


class Master(Script):
    def install(self, env):
        self.install_packages(env)

    #To stop the service, use the linux service stop command and pipe output to log file
    def stop(self, env):
        os.chdir('/usr/local/share/applications/javaSpringboot')
        Execute('./javaControl.py stop')
       
    #To start the service, use the linux service start command and pipe output to log file
    def start(self, env):
        import params
        self.configure(env)
        os.chdir('/usr/local/share/applications/javaSpringboot')
        Execute('./javaControl.py start')

    #To get status of the, use the linux service status command
    def status(self, env):
        check_process_status('/usr/local/share/applications/javaSpringboot/pid.txt')
        
    def configure(self, env):
        import params
        env.set_params(params)
        XmlConfig("java-config",
                  conf_dir="/usr/local/share/applications/javaSpringboot/config",
                  configurations=params.java_config,
                  owner="root",
                  group="root",
                  mode=0644
                  )

if __name__ == "__main__":
    Master().execute()

---

集成tomcat

ambari tomcat

---

集成中踩过的坑

1. ambari-server 重启会报错 数据库检查失败

2017-10-16 11:21:52,736  INFO - ******************************* Check database started *******************************
2017-10-16 11:21:56,029  INFO - Checking for configs not mapped to any cluster
2017-10-16 11:21:56,048  INFO - Checking for configs selected more than once
2017-10-16 11:21:56,050  INFO - Checking for hosts without state
2017-10-16 11:21:56,051  INFO - Checking host component states count equals host component desired states count
2017-10-16 11:21:56,053  INFO - Checking services and their configs
2017-10-16 11:21:58,035 ERROR - Service(s): KAFKA_MANAGER, from cluster cx has no config(s) in serviceconfig table!

对于ambari配置表的思考
ambari Server配置修改流程

2. quicklinks 无法显示

检查metainfo.xml是否正确配置快速链接，并检查quicklinks.json文件是否正确

ambari不会报错，只是不显示quicklinks

---

资源链接

• ambari自定义服务 步骤说明（一、二、三、四） CSDN
• ambari自定义服务文档
• ambari自定义服务相关脚本 github源码
• python2.7 入门教程 廖雪峰
• ambari 服务配置相关
• ambari 自定义神器