How to Migrate Prometheus DB?

Server A์—์„œ Server B๋กœ data๋ฅผ migrationํ•˜๋Š” ๋ฐฉ๋ฒ•์€ ์—ฌ๋Ÿฌ ๊ฐ€์ง€๊ฐ€ ์žˆ๋‹ค.

ํ•„์ž๋Š” ๋‹ค์Œ๊ณผ ๊ฐ™์€ ๋ฐฉ๋ฒ•๋“ค์„ ์‚ฌ์šฉํ•ด๋ดค๋‹ค.

  1. data ์ „์ฒด๋ฅผ tar๋กœ ์••์ถ•ํ•˜๊ฑฐ๋‚˜ / ์••์ถ•ํ•˜์ง€ ์•Š๊ณ  ๊ทธ๋Œ€๋กœ
  2. scp๋กœ ๋ฐ์ดํ„ฐ๋ฅผ ์ „์†ก ํ›„ ์••์ถ• ํ•ด์ œ

ํ•˜์ง€๋งŒ ์œ„ ๋ฐฉ๋ฒ•๋“ค์„ ์‚ฌ์šฉํ•˜๊ณ  ์ถ”์ถœ์„ ํ™•์ธํ•  ๋•, ์ด๋Ÿฐ ๋ฌธ์ œ๊ฐ€ ๋ฐœ์ƒํ–ˆ๋‹คโ€ฆ

Exception: Error querying Prometheus: 422 - b'{"status":"error","errorType":"execution","error":"cannot populate chunk 334271146 from block 00000000000000000000000000: segment doesn\'t include enough bytes to read the chunk size data field - required:334271151, available:152137216"}'

๋‹คํ–‰ํžˆ๋„ Prometheus์—์„œ๋Š” snapshot์„ ์ œ๊ณตํ•œ๋‹ค๊ณ  ํ•œ๋‹ค. ํ•˜์ง€๋งŒ, snapshot์„ ์‚ฌ์šฉํ•˜๊ธฐ ์œ„ํ•ด์„  ๊ด€๋ฆฌ์ž API๋ฅผ ํ™œ์„ฑํ™”ํ•ด์•ผํ•œ๋‹ค.

์ผ๋ฐ˜์ ์œผ๋กœ๋Š” prometheus --config.file=/path/to/prometheus.yml --web.enable-admin-api๋ฅผ ํ†ตํ•ด ๊ด€๋ฆฌ์ž API๋ฅผ ํ™œ์„ฑํ™” ํ•  ์ˆ˜ ์žˆ์ง€๋งŒ, ํ˜„์žฌ ์‚ฌ์šฉ์ค‘์ธ prometheus๋Š” helm chart๋กœ ์„ค์น˜ํ•œ ํ„ฐ๋ผ values.yaml์—์„œ enableAdminAPI: true๋กœ ๋„ฃ์–ด์ค˜์•ผํ•œ๋‹ค.

prometheus:
  prometheusSpec:
    enableAdminAPI: true

์ˆ˜์ • ํ›„ ๊ผญ Helm ์ฐจํŠธ๋ฅผ ์—…๊ทธ๋ ˆ์ด๋“œ ํ•ด์ฃผ์ž.

helm upgrade [RELEASE_NAME] [CHART_NAME] -f values.yaml