Skip to content

Commit 8274c3f

Browse files
masontikhonovmikhail-klimko
masontikhonov
and
mikhail-klimko
authored
fest: create basic script for index drift detection and elimination (#1)
* build: init project * feat: add `dump` command * build: add dump of prod indexes * build: clear indexes before dump * refactor: improve logs * refactor: add newline in the end of the dump * feat: add `compare` and `sync` commands * docs: add docs * build: add Dockerfile * tests: add `sync` tests * ci: add pipeline yaml * refactor: cleanup classic indexes * refactor: fix typo * feat: add `--force` flag * ci: update pipeline * ci: fix pipeline, update docs * ci: fix repo in package.json * docs: improve `--db-map` help * docs: improve `sync` documentation * ci: remove redundant pipeline * docs: document `isIndexEqual()` * docs: document Logger behavior * docs: document `syncIndexes()` function * Revert "ci: remove redundant pipeline" This reverts commit e77b9f5. * test: fix tests * ci: fix test coverage scprit * build: bump `mongodb` * ci: build `amd64/arm64` image * build: add full dump * feat: add `stats` command * fix(stats): fix getting oldest doc * remove legacy databases * remove legacy databases * feat: add planCache stats * feat(stats): add indexStats stats * feat: update index dump * refactor: reorganize code * feat: add default DB mapping `google_production=codefresh` * feat: add ignore overrides * feat: add ignore-list for classic * fix: fix comparison for abscent DBs * fix: improve ignorelists * feat: ignore `payments` db * fix: map `charts-manager` * fix: map `kubernetes-monitor` * fix: ingnore build TTL indexes in target DB * fix: ignore `payments-accounts-archive` in dump * feat: extend “heavy collections” list * refactor: rename variables * docs: improve docs --------- Co-authored-by: Mikhail Klimko <mikhail.klimko@octopus.com>
1 parent 2f12721 commit 8274c3f

File tree

212 files changed

+17711
-1
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

212 files changed

+17711
-1
lines changed

‎.editorconfig‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
root = true
2+
3+
[*]
4+
end_of_line = lf
5+
insert_final_newline = true
6+
7+
[*.{js,json,yml}]
8+
charset = utf-8
9+
indent_style = space
10+
indent_size = 2

‎.gitattributes‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
/.yarn/** linguist-vendored
2+
/.yarn/releases/* binary
3+
/.yarn/plugins/**/* binary
4+
/.pnp.* binary linguist-generated

‎.gitignore‎

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Logs
2+
logs
3+
*.log
4+
npm-debug.log*
5+
yarn-debug.log*
6+
yarn-error.log*
7+
lerna-debug.log*
8+
.pnpm-debug.log*
9+
10+
# Diagnostic reports (https://nodejs.org/api/report.html)
11+
report.[0-9]*.[0-9]*.[0-9]*.[0-9]*.json
12+
13+
# Runtime data
14+
pids
15+
*.pid
16+
*.seed
17+
*.pid.lock
18+
19+
# Directory for instrumented libs generated by jscoverage/JSCover
20+
lib-cov
21+
22+
# Coverage directory used by tools like istanbul
23+
coverage
24+
*.lcov
25+
26+
# nyc test coverage
27+
.nyc_output
28+
29+
# Grunt intermediate storage (https://gruntjs.com/creating-plugins#storing-task-files)
30+
.grunt
31+
32+
# Bower dependency directory (https://bower.io/)
33+
bower_components
34+
35+
# node-waf configuration
36+
.lock-wscript
37+
38+
# Compiled binary addons (https://nodejs.org/api/addons.html)
39+
build/Release
40+
41+
# Dependency directories
42+
node_modules/
43+
jspm_packages/
44+
45+
# Snowpack dependency directory (https://snowpack.dev/)
46+
web_modules/
47+
48+
# TypeScript cache
49+
*.tsbuildinfo
50+
51+
# Optional npm cache directory
52+
.npm
53+
54+
# Optional eslint cache
55+
.eslintcache
56+
57+
# Optional stylelint cache
58+
.stylelintcache
59+
60+
# Microbundle cache
61+
.rpt2_cache/
62+
.rts2_cache_cjs/
63+
.rts2_cache_es/
64+
.rts2_cache_umd/
65+
66+
# Optional REPL history
67+
.node_repl_history
68+
69+
# Output of 'npm pack'
70+
*.tgz
71+
72+
# Yarn Integrity file
73+
.yarn-integrity
74+
75+
# dotenv environment variable files
76+
.env
77+
.env.development.local
78+
.env.test.local
79+
.env.production.local
80+
.env.local
81+
82+
# parcel-bundler cache (https://parceljs.org/)
83+
.cache
84+
.parcel-cache
85+
86+
# Next.js build output
87+
.next
88+
out
89+
90+
# Nuxt.js build / generate output
91+
.nuxt
92+
dist
93+
94+
# Gatsby files
95+
.cache/
96+
# Comment in the public line in if your project uses Gatsby and not Next.js
97+
# https://nextjs.org/blog/next-9-1#public-directory-support
98+
# public
99+
100+
# vuepress build output
101+
.vuepress/dist
102+
103+
# vuepress v2.x temp and cache directory
104+
.temp
105+
.cache
106+
107+
# Docusaurus cache and generated files
108+
.docusaurus
109+
110+
# Serverless directories
111+
.serverless/
112+
113+
# FuseBox cache
114+
.fusebox/
115+
116+
# DynamoDB Local files
117+
.dynamodb/
118+
119+
# TernJS port file
120+
.tern-port
121+
122+
# Stores VSCode versions used for testing VSCode extensions
123+
.vscode-test
124+
125+
# yarn v2
126+
.yarn/cache
127+
.yarn/unplugged
128+
.yarn/build-state.yml
129+
.yarn/install-state.gz
130+
.pnp.*
131+
132+
!indexes/**
133+
134+
.DS_Store

‎.nvmrc‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
v22

‎.yarnrc.yml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
nodeLinker: node-modules

‎Dockerfile‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
FROM node:22-alpine AS base
2+
WORKDIR /app
3+
RUN corepack enable
4+
COPY package.json yarn.lock .yarnrc.yml ./
5+
6+
FROM base AS build
7+
COPY . .
8+
RUN yarn workspaces focus --all \
9+
&& yarn build
10+
11+
FROM base AS prod-dependencies
12+
RUN yarn workspaces focus --all --production
13+
14+
FROM base AS production
15+
COPY --chown=node:node package.json .
16+
COPY --chown=node:node --from=prod-dependencies /app/node_modules ./node_modules
17+
COPY --chown=node:node --from=build /app/dist ./dist
18+
COPY --chown=node:node ./indexes ./indexes
19+
20+
USER node
21+
ENTRYPOINT ["node", "dist/index.js"]

‎README.md‎

Lines changed: 101 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1 +1,101 @@
1-
# index-alignment
1+
# Codefresh index alignment
2+
3+
## Usage
4+
5+
```shell
6+
docker run quay.io/codefresh/index-alignment:<version> --help
7+
```
8+
```
9+
Options:
10+
-h, --help display help for command
11+
12+
Commands:
13+
dump [options] [Internal] Dump all indexes from a MongoDB instance
14+
stats [options] Get stats for all collections from a MongoDB instance
15+
compare [options] Compare indexes from a target MongoDB instance with a recommended dump
16+
sync [options] Sync indexes from a recommended dump with a target MongoDB instance. The command
17+
will fail if it is required to create indexes on heavily populated collections and
18+
the `--force` flag has not been specified
19+
help [command] display help for command
20+
```
21+
22+
23+
24+
### Commands
25+
26+
## `compare`
27+
28+
Compare indexes from a target MongoDB instance with a recommended dump.
29+
30+
All logs are written to the STDERR stream, so they can be easily separated from the actual output, which is written to STDOUT.
31+
32+
We recommend redirecting the output of `compare` command to JSON file.
33+
34+
> [!IMPORTANT]
35+
> The credentials under which the command is run must have read access to all databases controlled by Codefresh.
36+
37+
```
38+
Options:
39+
-p, --product <product> Codefresh product: classic | gitops
40+
-u, --uri <uri> MongoDB URI
41+
-m --db-map [dump-db-name=target-db-name...] Map the databases in the dump with the target databases. We have our own naming convention for the production databases, but it is up to the customers to name their databases (default: ["google_production=codefresh","chart-manager=charts-manager","kubernetes-monitor=k8s-monitor"])
42+
-h, --help display help for command
43+
```
44+
45+
Example:
46+
47+
```shell
48+
docker run quay.io/codefresh/index-alignment:<version> compare --product "classic" --uri "<db-uri>" > classic-diff.json
49+
50+
docker run quay.io/codefresh/index-alignment:<version> compare --product "gitops" --uri "<db-uri>" > gitops-diff.json
51+
```
52+
53+
## `stats`
54+
55+
Get stats for all collections from a MongoDB instance. Following commands will be executed:
56+
57+
* `dbStats` command ([doc](https://www.mongodb.com/docs/v5.0/reference/command/dbStats/));
58+
59+
* `$collStats` aggregation ([doc](https://www.mongodb.com/docs/v5.0/reference/operator/aggregation/collStats/));
60+
61+
* `$planCacheStats` aggregation ([doc](https://www.mongodb.com/docs/v5.0/reference/operator/aggregation/plancachestats/));
62+
63+
* queries `_id` of the oldest doc in each collection.
64+
65+
> [!IMPORTANT]
66+
> The credentials under which the command is run must have permissions to execute the commands specified above.
67+
68+
All logs are written to the STDERR stream, so they can be easily separated from the actual output, which is written to STDOUT.
69+
70+
We recommend redirecting the output of `stats` command to JSON file.
71+
72+
```
73+
Options:
74+
-u, --uri <uri> MongoDB URI
75+
-h, --help display help for command
76+
```
77+
78+
Example:
79+
80+
```shell
81+
docker run quay.io/codefresh/index-alignment:<version> stats --uri "<db-uri>" > db-stats.json
82+
```
83+
84+
## `sync`
85+
86+
> [!CAUTION]
87+
> This command changes indexes in the target DB, which may have performance impact.
88+
>
89+
> We strongly advice to NOT use this command against production DB because of possible performance impact. Instead, use `compare` command to get the diff, then consider eliminating index diff manually during maintainance window.
90+
91+
92+
Sync indexes from a recommended dump with a target MongoDB instance. The command will fail if it is required to create indexes on heavily populated collections and the `--force` flag has not been specified
93+
94+
```
95+
Options:
96+
-p, --product <product> Codefresh product: classic | gitops
97+
-u, --uri <uri> MongoDB URI
98+
-f --force Create indexes even on heavily populated collections, which may take a while
99+
-m --db-map [dump-db-name=target-db-name...] Map the databases in the dump with the target databases. We have our own naming convention for the production databases, but it is up to the customers to name their databases (default: ["google_production=codefresh","chart-manager=charts-manager","kubernetes-monitor=k8s-monitor"])
100+
-h, --help display help for command
101+
```

0 commit comments

Comments
 (0)