Sentry 監(jiān)控 - Snuba 數(shù)據(jù)中臺本地開發(fā)環(huán)境配置實戰(zhàn)
本文轉(zhuǎn)載自微信公眾號「黑客下午茶」,作者為少 。轉(zhuǎn)載本文請聯(lián)系黑客下午茶公眾號。
克隆倉庫
分別克隆 getsentry/sentry 與 getsentry/snuba:
git clone https://github.com/getsentry/sentry.git
git clone https://github.com/getsentry/snuba.git
安裝系統(tǒng)依賴(以 Mac 為例)
Xcode CLI tools
- xcode-select --install
 
Brewfile
進入 sentry 文件夾,你會看到一個 Brewfile 文件:
- cd sentry
 
Brewfile
- # required to run devservices
 - cask 'docker'
 - brew 'pyenv'
 - # required for pyenv's python-build
 - brew 'openssl'
 - brew 'readline'
 - # required for yarn test -u
 - brew 'watchman'
 - # required to build some of sentry's dependencies
 - brew 'pkgconfig'
 - brew 'libxslt'
 - brew 'libxmlsec1'
 - brew 'geoip'
 - # Currently needed because on Big Sur there's no wheel for it
 - brew 'librdkafka'
 - # direnv isn't defined here, because we have it configured to check for a bootstrapped environment.
 - # If it's installed in the early steps of the setup process, it just leads to confusion.
 - # brew 'direnv'
 - tap 'homebrew/cask'
 - # required for acceptance testing
 - cask 'chromedriver'
 
如果你本地已經(jīng)安裝了 Docker Desktop 并且已經(jīng)啟動,可以把 cask 'docker' 注釋掉。
接下來,運行:
- brew bundle --verbose
 
如果你之前本地沒有 Docker Desktop,則還需要手動啟動一下它:
- open -g -a Docker.app
 
構(gòu)建工具鏈
Sentry 依賴于 Python Wheels(包含二進制擴展模塊的包),官方為以下平臺分發(fā):
- Linux 兼容 PEP-513 (manylinux1)
 - macOS 10.15 或更高版本
 
如果您的開發(fā)機器沒有運行上述系統(tǒng)之一,則需要安裝 Rust 工具鏈。按照 https://www.rust-lang.org/tools/install 上的說明安裝編譯器和相關(guān)工具。安裝后,Sentry 安裝程序?qū)⒆詣邮褂?Rust 構(gòu)建所有二進制模塊,無需額外配置。
官方通常會跟蹤最新的穩(wěn)定 Rust 版本,該版本每六周更新一次。因此,請確保通過偶爾運行來使您的 Rust 工具鏈保持最新:
- rustup update stable
 
Python
Sentry 使用 pyenv 來安裝和管理 Python 版本。它是在您運行 brew bundle 時安裝的。
要安裝所需版本的 Python,您需要運行以下命令。這將需要一段時間,因為您的計算機實際上正在編譯 Python!
- make setup-pyenv
 
這里假設(shè)你是 Zsh 用戶。
如果您鍵入 which python,您應(yīng)該看到類似 $HOME/.pyenv/shims/python 而不是 /usr/bin/python 的內(nèi)容。這是因為以下內(nèi)容已添加到您的啟動腳本中:
cat ~/.zprofile,你會看到如下內(nèi)容:
- # MacPorts Installer addition on 2021-10-20_at_11:48:22: adding an appropriate PATH variable for use with MacPorts.
 - export PATH="/opt/local/bin:/opt/local/sbin:$PATH"
 - # Finished adapting your PATH environment variable for use with MacPorts.
 - # It is assumed that pyenv is installed via Brew, so this is all we need to do.
 - eval "$(pyenv init --path)"
 
虛擬環(huán)境
您現(xiàn)在已準備好創(chuàng)建 Python 虛擬環(huán)境。運行:
- python -m venv .venv
 
并激活虛擬環(huán)境:
- source .venv/bin/activate
 
如果一切正常,運行 which python 現(xiàn)在應(yīng)該會導(dǎo)致類似 /Users/you/sentry/.venv/bin/python 的結(jié)果。
Snuba 配置實戰(zhàn)
啟動 Snuba 相關(guān)依賴項容器
- cd ../sentry
 - git checkout master
 - git pull
 - source .venv/bin/activate
 - sentry devservices up --exclude=snuba
 - # 11:17:59 [WARNING] sentry.utils.geo: settings.GEOIP_PATH_MMDB not configured.
 - # 11:18:01 [INFO] sentry.plugins.github: apps-not-configured
 - # > Pulling image 'postgres:9.6-alpine'
 - # > Pulling image 'yandex/clickhouse-server:20.3.9.70'
 - # > Not starting container 'sentry_relay' because it should be started on-demand with devserver.
 - # > Creating 'sentry_redis' volume
 - # > Creating 'sentry_zookeeper_6' volume
 - # > Creating 'sentry_kafka_6' volume
 - # > Creating container 'sentry_redis'
 - # > Creating container 'sentry_zookeeper'
 - # > Creating container 'sentry_kafka'
 - # > Starting container 'sentry_redis' (listening: ('127.0.0.1', 6379))
 - # > Starting container 'sentry_kafka' (listening: ('127.0.0.1', 9092))
 - # > Starting container 'sentry_zookeeper'
 - # > Creating 'sentry_clickhouse' volume
 - # > Creating container 'sentry_clickhouse'
 - # > Creating 'sentry_postgres' volume
 - # > Creating 'sentry_wal2json' volume
 - # > Starting container 'sentry_clickhouse' (listening: ('127.0.0.1', 9000), ('127.0.0.1', 9009), ('127.0.0.1', 8123))
 - # > Creating container 'sentry_postgres'
 - # > Starting container 'sentry_postgres' (listening: ('127.0.0.1', 5432))
 
這將在 master 上獲取最新版本的 Sentry,并調(diào)出所有 snuba 的依賴項。
Snuba 主要依賴 clickhouse,zookeeper,kafka,redis 相關(guān)容器。
docker ps 查看一下:
- 1149a6f6ff23 postgres:9.6-alpine "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 127.0.0.1:5432->5432/tcp sentry_postgres
 - a7f3af7d52bb yandex/clickhouse-server:20.3.9.70 "/entrypoint.sh" 3 minutes ago Up 3 minutes 127.0.0.1:8123->8123/tcp, 127.0.0.1:9000->9000/tcp, 127.0.0.1:9009->9009/tcp sentry_clickhouse
 - 68913ee15c43 confluentinc/cp-zookeeper:6.2.0 "/etc/confluent/dock…" 3 minutes ago Up 3 minutes 2181/tcp, 2888/tcp, 3888/tcp sentry_zookeeper
 - 5a248eb26ed3 confluentinc/cp-kafka:6.2.0 "/etc/confluent/dock…" 3 minutes ago Up 3 minutes 127.0.0.1:9092->9092/tcp sentry_kafka
 - 0573aff7b5af redis:5.0-alpine "docker-entrypoint.s…" 3 minutes ago Up 3 minutes 127.0.0.1:6379->6379/tcp sentry_redis
 
設(shè)置 Snuba 虛擬環(huán)境
- cd snuba
 - make pyenv-setup
 - python -m venv .venv
 - source .venv/bin/activate
 - pip install --upgrade pip==21.1.3
 - make develop
 
查看遷移列表
- snuba migrations list
 - # system
 - # [ ] 0001_migrations
 - #
 - # events
 - # [ ] 0001_events_initial
 - # [ ] 0002_events_onpremise_compatibility
 - # [ ] 0003_errors
 - # [ ] 0004_errors_onpremise_compatibility
 - # [ ] 0005_events_tags_hash_map (blocking)
 - # [ ] 0006_errors_tags_hash_map (blocking)
 - # [ ] 0007_groupedmessages
 - # [ ] 0008_groupassignees
 - # [ ] 0009_errors_add_http_fields
 - # [ ] 0010_groupedmessages_onpremise_compatibility (blocking)
 - # [ ] 0011_rebuild_errors
 - # [ ] 0012_errors_make_level_nullable
 - # [ ] 0013_errors_add_hierarchical_hashes
 - # [ ] 0014_backfill_errors (blocking)
 - # [ ] 0015_truncate_events
 - #
 - # transactions
 - # [ ] 0001_transactions
 - # [ ] 0002_transactions_onpremise_fix_orderby_and_partitionby (blocking)
 - # [ ] 0003_transactions_onpremise_fix_columns (blocking)
 - # [ ] 0004_transactions_add_tags_hash_map (blocking)
 - # [ ] 0005_transactions_add_measurements
 - # [ ] 0006_transactions_add_http_fields
 - # [ ] 0007_transactions_add_discover_cols
 - # [ ] 0008_transactions_add_timestamp_index
 - # [ ] 0009_transactions_fix_title_and_message
 - # [ ] 0010_transactions_nullable_trace_id
 - # [ ] 0011_transactions_add_span_op_breakdowns
 - # [ ] 0012_transactions_add_spans
 - #
 - # discover
 - # [ ] 0001_discover_merge_table
 - # [ ] 0002_discover_add_deleted_tags_hash_map
 - # [ ] 0003_discover_fix_user_column
 - # [ ] 0004_discover_fix_title_and_message
 - # [ ] 0005_discover_fix_transaction_name
 - # [ ] 0006_discover_add_trace_id
 - # [ ] 0007_discover_add_span_id
 - #
 - # outcomes
 - # [ ] 0001_outcomes
 - # [ ] 0002_outcomes_remove_size_and_bytes
 - # [ ] 0003_outcomes_add_category_and_quantity
 - # [ ] 0004_outcomes_matview_additions (blocking)
 - #
 - # metrics
 - # [ ] 0001_metrics_buckets
 - # [ ] 0002_metrics_sets
 - # [ ] 0003_counters_to_buckets
 - # [ ] 0004_metrics_counters
 - # [ ] 0005_metrics_distributions_buckets
 - # [ ] 0006_metrics_distributions
 - # [ ] 0007_metrics_sets_granularity_10
 - # [ ] 0008_metrics_counters_granularity_10
 - # [ ] 0009_metrics_distributions_granularity_10
 - # [ ] 0010_metrics_sets_granularity_1h
 - # [ ] 0011_metrics_counters_granularity_1h
 - # [ ] 0012_metrics_distributions_granularity_1h
 - # [ ] 0013_metrics_sets_granularity_1d
 - # [ ] 0014_metrics_counters_granularity_1d
 - # [ ] 0015_metrics_distributions_granularity_1d
 - #
 - # sessions
 - # [ ] 0001_sessions
 - # [ ] 0002_sessions_aggregates
 - # [ ] 0003_sessions_matview
 
運行遷移
- snuba migrations migrate --force
 - # ......
 - # 2021-12-01 19:45:57,557 Running migration: 0014_metrics_counters_granularity_1d
 - # 2021-12-01 19:45:57,575 Finished: 0014_metrics_counters_granularity_1d
 - # 2021-12-01 19:45:57,589 Running migration: 0015_metrics_distributions_granularity_1d
 - # 2021-12-01 19:45:57,610 Finished: 0015_metrics_distributions_granularity_1d
 - # 2021-12-01 19:45:57,623 Running migration: 0001_sessions
 - # 2021-12-01 19:45:57,656 Finished: 0001_sessions
 - # 2021-12-01 19:45:57,669 Running migration: 0002_sessions_aggregates
 - # 2021-12-01 19:45:57,770 Finished: 0002_sessions_aggregates
 - # 2021-12-01 19:45:57,792 Running migration: 0003_sessions_matview
 - # 2021-12-01 19:45:57,849 Finished: 0003_sessions_matview
 - # Finished running migrations
 
檢查遷移
進入 Clickhouse 容器:
- docker exec -it sentry_clickhouse clickhouse-client
 - # 運行如下 `sql` 語句:
 - select count() from sentry_local
 - # ClickHouse client version 20.3.9.70 (official build).
 - # Connecting to localhost:9000 as user default.
 - # Connected to ClickHouse server version 20.3.9 revision 54433.
 - # a7f3af7d52bb :) select count() from sentry_local
 - # SELECT count()
 - # FROM sentry_local
 - # ┌─count()─┐
 - # │ 0 │
 - # └─────────┘
 - # 1 rows in set. Elapsed: 0.008 sec.
 - # a7f3af7d52bb :)
 
查看相關(guān)實體數(shù)據(jù)集
- snuba entities list
 - # Declared Entities:
 - # discover
 - # events
 - # groups
 - # groupassignee
 - # groupedmessage
 - # metrics_sets
 - # metrics_counters
 - # metrics_distributions
 - # outcomes
 - # outcomes_raw
 - # sessions
 - # org_sessions
 - # spans
 - # transactions
 - # discover_transactions
 - # discover_events
 
啟動開發(fā)服務(wù)器
此命令將啟動 api 和所有 Snuba 消費者以從 Kafka 攝取數(shù)據(jù):
- snuba devserver
 
轉(zhuǎn)到 http://localhost:1218/events/snql,你將會看到一個簡易的查詢 UI。
















 
 
 


 
 
 
 