Prometheus Alerting

1.Alerting with Prometheus is separated in to two parts 1-1Alerting rules Alerting rules is a file that formatted by yml,it’s define some exp in to the yaml file. Alerting rules is in Prometheus servers send alerts to an alert manager and alert manager managed those alerts. the rule document in there: the rules doc 1-2Alerting Manager AlertingManager is a command line tool and the command line tool flags has those functions...

December 5, 2021 · 3 min · cui

Prometheus Exporter

time series metrics collect toolkit 提供了UI服务,默认端口30313 主要作用就是监控一下我们服务的各项指标,也有许多包已经集成了这个功能,基本上查找一下配置一下就可以快速的展示出各项指标了 业务指标 但例如 java_gc,go_gc,node_request_total这种指标对我们来说没有特别大的用处,我们的着力点应该在于提取业务指标,各个业务系统数据源不同逻辑不同导致无法使用通用型第三方包来构建我们的指标。 这个时候就出现了 prometheus-exporter用来放出自定义的业务指标,等待prometheus来此处收集。 prometheus exporter 其实很简单,就是攒一些prometheus数据格式的数据放入 /metrics路由中等待scrape就好 下面使用Go来举例子 这里使用了godror驱动oracle,prometheus/client_golang 客户端,logrus日志模块 一些命令行参数 main函数主要就是构造exporter对象然后启动http服务 exporter 结构体中加入了同步锁,封装好的oracle客户端比较重要的是 metricDescriptions和 metricMap这几个指标map 作为prometheus的exporter需要实现两个接口,一个是输入描述的 Describe一个是收集数据的 Collect Collect,循环map然后将数据和字符串传入ch管道中 Describe相同都是向ch放入构建好的 指针类型对象 接下来看map中到底是什么 map为string对应的匿名函数,函数类型为MetricHandler 很好理解,将管道传入匿名函数,结构体传入匿名函数,然后利用结构体的oracleClient执行sql构建Metric结构体传入ch中即可 Over 整个流程非常简单,由于公司的exporter不可能开源,所以这里提供redis-exporter地址以供参考 https://github.com/oliver006/redis_exporter 2020结束,最后一篇简单的参考教程

December 5, 2021 · 1 min · cui

Prometheus Redis

1.下载redis_exporter 2.脚本 1 nohup ./redis_exporter -redis.addr 0.0.0.0:6379 -redis.password "password" & link 3.promethus endpoint 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 apiVersion: v1 kind: Endpoints metadata: name: redis-exporter subsets: - addresses: - ip: 222....

December 5, 2021 · 16 min · cui

Prometheus Rules

1.Two types of rules Prometheus supports two types of rules .whitch may be configured and then evaluated at regular intervals: Recording rules and Alerting rules. 2.Recording rules 1.Recording rules allow you to precompute frequently needed or computationally expensive expressions and save their result as a new set of time series. 2.Be mutch faster than executing the origin expression every time it’s need. 3.Rules within a group run sequentially at a regular interval....

December 5, 2021 · 9 min · cui