OpenTSDB参数优化和TSD服务监控

一、编译安装注意点

在编译安装OpenTSDB时，需要注意几点
1、configure时指定安装路径,make install时会把生成的tsdb bin文件和依赖文件复制这个目录中。

 ./configure --prefix=/usr/local/bin/opentsdb

2、修改bin目录中的tsdb启动文件，设置config目录，否则找不到logback.xml配置

configdir='/usr/local/bin/opentsdb/share/opentsdb/etc/opentsdb

3、在logback.xml文件中设置日志输出位置到文件中，默认是输出到STDOUT中。

  <!-- Per class logger levels -->
  <logger name="QueryLog" level="WARN" additivity="false">
    <appender-ref ref="QUERY_LOG"/>
  </logger>
  <logger name="org.apache.zookeeper" level="INFO"/>
  <logger name="org.hbase.async" level="INFO"/>
  <logger name="com.stumbleupon.async" level="INFO"/>

  <!-- Fallthrough root logger and router -->
  <root level="INFO">
    <!-- <appender-ref ref="STDOUT"/> -->
    <appender-ref ref="CYCLIC"/>
    <appender-ref ref="FILE"/>
  </root>

4、在bin/tsdb中设置JVM大小

JVMARGS=${JVMARGS-'-Xmx12000m -enableassertions -enablesystemassertions'}

二、OpenTSDB

1) 关闭tsd.storage.enable_compaction，避免每小时请求hbase读取数据然后合并数据点。
2）同时开启tsd.storage.enable_appends，提高写入性能。

tsd.network.port = 4242
tsd.http.staticroot = /data0/opentsdb/staticroot
tsd.http.cachedir = /data0/opentsdb/tsd
tsd.core.auto_create_metrics = true
tsd.storage.enable_appends = true
tsd.storage.enable_compaction = false
tsd.storage.hbase.zk_basedir = /hbase
tsd.storage.hbase.zk_quorum = 127.0.0.1:2181
tsd.http.request.enable_chunked = true
tsd.http.request.max_chunk = 65535
tsd.core.meta.enable_realtime_ts = true

三、TSD监控

可以通过tsd提供api可以直接获取json格式的stats状态信息，但是需要做个转换，做状态类型区分（GAUGE or COUNTER）。

http://192.168.6.6:4242/api/stats

stats里面的metrics和类型可以参考官方文档，user_guide/stats.html

下面一个可以向open-falcon提交数据一个plugin脚本，返回的仍然是json格式。

#!/bin/env python
#**********************************#
#created by chenqun(sylar)
#2018-05-24
#**********************************#

import socket,time,sys,os
import json,urllib
METRICS_FILE='/opt/tsd_metrics.txt'


def get_tsd_metrics():
  mtype = {}
  if os.path.exists(METRICS_FILE):
    with open(METRICS_FILE,'r') as fd:
      for line in fd:
        ct = line.strip().split('|')
        mt = ct[0] + '|' + ct[1] 
        mtype[mt] = ct[2]
  return mtype


def get_tsd_stats():
  hostname = socket.gethostname()
  fd = urllib.urlopen("http://%s:4242/api/stats" %(hostname))
  data = fd.read()  
  stats = json.loads(data)
  return stats


def parse_stats(stats,mtype):
  metrics = []
  for its in stats:
    mtr = {}
    mtr['metric'] = its['metric']
    mtr['timestamp'] = int(its['timestamp'])
    mtr['value'] = int(its['value'])
    mtr['step'] = 60
    mtr['endpoint'] = its['tags']['host']
    if 'type' in its['tags']:
      tag = 'type=' + its['tags']['type']
    elif 'method' in its['tags']:
      tag = 'method=' + its['tags']['method']
    elif 'kind' in its['tags']:
      tag = 'kind=' + its['tags']['kind']
    elif 'cache' in its['tags']:
      tag = 'cache=' + its['tags']['cache']
    else :
      tag = ''

    mt_idx = mtr['metric'] + '|' + tag
    if mtype.has_key(mt_idx):
      mtr['counterType'] = mtype[mt_idx]
      if its['tags'].has_key('class'):
        tag = tag +'&class='+ its['tags']['class']
      mtr['tags'] = tag
    else:
      continue
    metrics.append(mtr)
  return metrics


if __name__ == "__main__" :
  mtype = get_tsd_metrics()
  stats = get_tsd_stats()
  metrics = parse_stats(stats, mtype)
  print json.dumps(metrics)
  #print fd.read()  #check response 

  # or post data to agent gateway
  #hostname = socket.gethostname()
  #url = 'http://%s:1988/v1/push' %(hostname)
  #fd = urllib.urlopen(url, json.dumps(metrics))