-
Notifications
You must be signed in to change notification settings - Fork 2.3k
feat(new source): Add ODBC source implementation #24044
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
powerumc
wants to merge
193
commits into
vectordotdev:master
Choose a base branch
from
powerumc:powerumc/odbc-source
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 183 commits
Commits
Show all changes
193 commits
Select commit
Hold shift + click to select a range
9c793e2
feat(odbc source): add ODBC source for scheduled database queries
powerumc 9e3f5d7
test(odbc source): Integration tests
powerumc f383c9f
test(odbc source): prepare Dockerfile
powerumc 50fac18
feat(odbc source): implement ODBC metrics for event tracking
powerumc fa4502d
test(odbc source): integration test configuration
powerumc 3090a4e
refactor(odbc source): introduce graceful shutdown and improve schedu…
powerumc e13b1a1
chore(odbc source): clean up formatting
powerumc f39ee28
test(odbc source): ODBC init_params, tracking_columns, last_run_metad…
powerumc 831eb45
fix(odbc source): fix data type handling and comments
powerumc 64fda4e
test(odbc source): number types integration test
powerumc e9037aa
test(odbc source): string types, other types integration test
powerumc b88c42e
chore(odbc source): clean up formatting
powerumc aef4174
feat(odbc source): add support for PostgreSQL and update integration …
powerumc 32a566d
fix(odbc source): refactor test setup for consistency
powerumc 8af2a13
test(odbc source): add integration test for timestamp columns
powerumc be3e3ef
docs(odbc source): add documentation for ODBC data source configuration
powerumc 5dae7a7
Merge branch 'master' into powerumc/odbc-source
powerumc 19cf895
test(odbc source): add external network configuration for MySQL and P…
powerumc 15d4436
docs(odbc source): update source code documents.
powerumc 1a3efcb
docs(odbc source): add comprehensive CUE configuration and documentat…
powerumc 40d5795
chore(odbc source): source code formatting
powerumc c74a12a
refactor(odbc source): executes ODBC query processing with result tra…
powerumc cea0073
chore(deps): update odbc-api dependency to version 19.1.0
powerumc bb62426
chore(odbc source): update LICENSE-3rdparty.csv
powerumc 6af628d
docs(odbc source): improve documentation clarity and update examples …
powerumc f4830ed
Merge branch 'master' into powerumc/odbc-source
powerumc 3cb3d8f
chore(odbc source): update fmt and clippy
powerumc 2a12364
chore(vdev): apply vdev rust check fixes
powerumc 0b9c191
chore(vdev): apply vdev rust check fixes
powerumc ffc2a22
docs(odbc source): apply suggestion to odbc.cue
powerumc 4c2e19b
docs(odbc source): apply suggestion to odbc.cue
powerumc ff66be0
docs(odbc source): apply suggestion to odbc.cue
powerumc af3fd51
docs(odbc source): add changelog
powerumc e341811
Merge branch 'master' into powerumc/odbc-source
powerumc b266337
chore(odbc source): update spelling expect.txt and fix spell check fa…
powerumc 5e26f6e
Merge branch 'master' into powerumc/odbc-source
powerumc eae2e87
Merge branch 'master' into powerumc/odbc-source
powerumc 58bdd2f
test(e2e): add ODBC drivers to Dockerfile
powerumc c6d5e38
Merge branch 'master' into powerumc/odbc-source
powerumc 816e056
Merge branch 'master' into powerumc/odbc-source
powerumc 723d127
fix(odbc source): update ClosedError import path and fix changelog ne…
powerumc 02eb982
fix(odbc source): add ODBC metrics feature support
powerumc 49d8d7d
refactor(odbc source): extract `OdbcConfig` into `config.rs` file
powerumc a44ab52
refactor(odbc source): move `OdbcSchedule` implementation to `schedul…
powerumc f832f30
refactor(odbc source): move client functions and types to client.rs
powerumc dc85af0
test(odbc source): move the integration test file and fix the import
powerumc d0de50d
docs(odbc source): update sources/generated/odbc.cue
powerumc a2cba35
Merge branch 'master' into powerumc/odbc-source
powerumc ae48f96
Merge branch 'master' into powerumc/odbc-source
powerumc b599b75
fix(docker): add unixodbc-dev to Dockerfile dependencies
powerumc c2a48f4
Merge branch 'master' into powerumc/odbc-source
powerumc b602ac4
chore(dependencies): update dependencies in `Cargo.lock`
powerumc 3088a83
fix(odbc source): update error handling to use SendError instead of C…
powerumc 7353a02
Merge branch 'master' into powerumc/odbc-source
powerumc a196bf2
Merge branch 'master' into powerumc/odbc-source
powerumc b8b1b87
Merge branch 'master' into powerumc/odbc-source
powerumc 0e08ab7
Merge branch 'master' into powerumc/odbc-source
powerumc cf4370f
Merge branch 'master' into powerumc/odbc-source
powerumc 96ab321
fix(odbc_metrics): add NamedInternalEvent derive to event structs
powerumc 70023b1
chore(odbc source): code formatting
powerumc d23f4dc
Merge branch 'master' into powerumc/odbc-source
powerumc bde9f88
Merge branch 'vectordotdev:master' into powerumc/odbc-source
powerumc 130abe3
chore(ci): add odbc-mariadb and odbc-postgresql to CI workflows
powerumc ffd0976
Merge branch 'master' into powerumc/odbc-source
powerumc 9360370
Merge branch 'master' into powerumc/odbc-source
powerumc 980edad
Merge branch 'master' into powerumc/odbc-source
pront d579eee
fix(odbc source): correct default function for `odbc_max_str_limit` c…
powerumc 3d75710
fix(odbc source): improve handling of statement file reading
powerumc beef419
fix(odbc source): update datetime parsing to handle local timezones
powerumc 848bc9b
fix(odbc source): update comment `schedule` field and error log.
powerumc 21c8189
Merge branch 'vectordotdev:master' into powerumc/odbc-source
powerumc 18a311c
Merge branch 'master' into powerumc/odbc-source
powerumc 333e537
Merge branch 'master' into powerumc/odbc-source
powerumc 5e68722
Merge branch 'master' into powerumc/odbc-source
powerumc 7679c48
fix(odbc source): use log schema paths for inserting timestamp
powerumc 5013881
chore(ci): add odbc-mariadb and odbc-postgresql to changes.yml
powerumc 25330e8
Merge branch 'master' into powerumc/odbc-source
powerumc 7dbea60
chore: merge Cargo.lock
powerumc 91f03b2
chore: add ODBC dependencies to Debian build script
powerumc 085cf75
chore: update LICENSE-3rdparty.csv
powerumc ec83e60
refactor(odbc): simplify timestamp handling in map_value function by …
powerumc 59e4174
fix(workflows): add missing commas in changes.yml to ensure proper YA…
powerumc e6f7f80
fix: Fixed — added %.f variants to TIMESTAMP_FORMATS and tests for fr…
powerumc 5505c33
chore(tests): update test paths for ODBC integration tests
powerumc c781ca3
chore(odbc): add error_stage to OdbcFailedError logging
powerumc aa354da
feat(odbc): enhance event processing with decoding and enrichment fea…
powerumc 76bdb72
refactor(metrics): update ComponentExecutedEventsTotal enum metric
powerumc 0155e75
refactor(odbc): remove boolean mapping from map_value function
powerumc 64c0cbd
refactor(odbc): improve metadata loading logic to handle errors and s…
powerumc 3056a80
refactor(odbc): replace Environment::new() with environment() for imp…
powerumc b978597
feat(odbc): add default serialization for connection_string in OdbcCo…
powerumc f53d6c9
refactor(odbc): streamline event emission by consolidating event coun…
powerumc e16c071
refactor(odbc): change connection_string type to SensitiveString for …
powerumc 3c8d83d
refactor(odbc): remove redundant statement logging from OdbcFailedErr…
powerumc 0773036
refactor(odbc): enhance error handling by introducing Decode variant …
powerumc b1903dd
fix(odbc): disable acknowledgements feature in ODBC configuration
powerumc f83f467
refactor(odbc): enhance OdbcError handling by adding error_type and e…
powerumc 82e333d
chore(ci): add odbc-mariadb and odbc-postgresql to integration workflows
powerumc 0633d41
feat(odbc): add BlockingTask error variant and refactor query executi…
powerumc 5519f99
refactor(odbc): update schedule handling in OdbcConfig and related co…
powerumc f2e1abf
refactor(odbc): update ODBC configuration schema with improved descri…
powerumc 8db9893
refactor(odbc): enforce requirement for statement or statement_filepa…
powerumc c21ef25
refactor(odbc): update value mapping for Decimal and Numeric types to…
powerumc 50d86c6
refactor(odbc): enforce requirement for non-empty connection string o…
powerumc 5c79859
refactor(odbc): improve connection string handling in OdbcConfig to r…
powerumc 459c8cf
feat(odbc): include byte size in OdbcEventsReceived for improved metr…
powerumc 61db439
fix(odbc): handle schedule exhaustion in ODBC source to ensure gracef…
powerumc 0f1336c
refactor(odbc): enhance parameter handling in order_params and add va…
powerumc dd6dd94
refactor(odbc): improve error handling for SQL statement retrieval in…
powerumc 03506e0
fix(odbc): handle empty rows in decode_rows method to prevent unneces…
powerumc 31f9193
fix(odbc): add truncation check in query execution to handle potentia…
powerumc 3a9e087
refactor(odbc): enhance error handling in OdbcFailedError to categori…
powerumc 2641994
refactor(odbc): add login_timeout configuration
powerumc 7db485c
refactor(odbc): enhance datetime handling by introducing naive_local_…
powerumc cea7290
docs(odbc): update OdbcConfig documentation to clarify timeout behavi…
powerumc c086681
fix(odbc): enforce odbc_batch_size to be greater than 0 in configuration
powerumc 538fba6
fix(odbc): preserve out-of-range integer values as bytes
powerumc 07cc01a
refactor(odbc): add validation for tracking_columns
powerumc fbec868
refactor(odbc): update max_str_limit handling to support optional val…
powerumc 80175bc
fix(odbc): format tracking timestamps in local timezone for SQL binds
powerumc 9b79856
docs(odbc): clarify behavior of can_acknowledge method regarding row …
powerumc fc3c923
Merge branch 'master' into powerumc/odbc-source
powerumc c586b84
chore: code formatting
powerumc 57970f9
refactor(odbc): improve boolean handling in SQL parameter conversion
powerumc bb313e4
fix(odbc): handle unrepresentable floats in value mapping
powerumc db5e69f
refactor(odbc): enhance ODBC query processing with byte size tracking
powerumc 267317f
refactor(odbc): remove unused OdbcEventsReceived struct and related m…
powerumc 276ff82
test(odbc): add test-threads argument to ODBC integration test config…
powerumc 8bc1411
refactor(odbc): Reject missing tracking parameter values
powerumc 8c3ea93
docs(odbc): update delivery method to best_effort and add warning for…
powerumc b015ebb
refactor(odbc): add prepare_metadata_path function to create parent d…
powerumc 96405ec
refactor(odbc): stream large ODBC result sets in bounded chunks
powerumc 69958d9
refactor(odbc): preserve original time and date values as bytes for t…
powerumc 85f77e1
test(odbc): update SQL query in integration tests to order results by…
powerumc bf082b2
feat(odbc): add error handling for duplicate column names in SQL queries
powerumc b8d1df3
refactor(odbc): improve parameter saving by using temporary files for…
powerumc 749830f
refactor(odbc): update tracking column extraction to store SQL parame…
powerumc 8f28c63
refactor(odbc): update default decoding to use JSON deserializer for …
powerumc 0218de2
test(odbc): add tests for preserving zero dates and duration times as…
powerumc 2f9a82f
feat(odbc): enhance timestamp handling by adding support for SQL-styl…
powerumc 24cd8c7
refactor(odbc): clean up unused timestamp handling and simplify schem…
powerumc 9973264
fix(odbc): update timestamp handling to correctly preserve SQL-style …
powerumc 2628416
refactor(odbc): implement distinct temporary file handling for checkp…
powerumc f455fff
refactor(odbc): simplify value extraction from event logs and update …
powerumc 3b4c198
fix(odbc): implement atomic file replacement for checkpointing on Win…
powerumc c43b0bd
refactor(odbc): replace JSON decoding with direct conversion of ODBC …
powerumc bfc00d3
refactor(odbc): enhance timestamp handling to preserve SQL and RFC333…
powerumc 2017f6c
feat(odbc): implement graceful shutdown handling for ODBC source, enh…
powerumc 7eed844
fix(odbc): strip trailing whitespace from connection string file cont…
powerumc 490cb0d
refactor(odbc): introduce OdbcStatementParam struct for enhanced SQL …
powerumc f83181d
refactor(odbc): improve SQL parameter handling to preserve RFC3339-li…
powerumc d235d71
Merge branch 'master' into powerumc/odbc-source
powerumc 2bc04ab
refactor(odbc): enhance tracking semantics for ODBC source by adding …
powerumc 2909e82
chore(deps): update Cargo.lock and LICENSE-3rdparty.csv
powerumc 6dd88e5
feat(odbc): implement binary column handling and buffer management fo…
powerumc cf7c43d
refactor(odbc): improve error handling and metrics emission logic in …
powerumc 78be871
docs(odbc): enhance tracking and buffering logic in ODBC source docum…
powerumc 9f60660
chore(odbc): update SQL statement to include ORDER BY clause for cons…
powerumc 7204fde
ci: install unixODBC for default feature builds
powerumc 414b935
chore(odbc): enhance installation and runtime requirements for ODBC s…
powerumc 5f8722f
Merge branch 'master' into powerumc/odbc-source
powerumc 0f2b514
Merge branch 'master' into powerumc/odbc-source
powerumc 2c041f3
fix(cross-compilation): update ODBC installation for target-specific …
powerumc e9075bb
fix(odbc): clarify support requirements for ODBC in cue configuration
powerumc 3c4f5b3
fix(odbc): update ODBC configuration notes and remove unixodbc from C…
powerumc 5633ac3
fix(odbc): update SQL statements in ODBC configuration examples to in…
powerumc 77df396
feat(odbc): implement connection retry logic for ODBC database readiness
powerumc 0b41fd7
fix(odbc): add unixodbc to CI workflow for improved ODBC support
powerumc 683b20e
fix(odbc): clarify ODBC support requirements and update macOS build n…
powerumc 0b78056
Merge branch 'master' into powerumc/odbc-source
powerumc 4f63625
chore(odbc): update SQL statements in ODBC configuration examples to …
powerumc 8f09559
fix(odbc): enhance ODBC requirements documentation for clarity on Win…
powerumc 8dea7d3
Merge branch 'master' into powerumc/odbc-source
pront 18692f7
chore(odbc): formating and clippy fixes
powerumc d390bfb
docs(odbc): more specific ODBC changelog
powerumc 0d45d81
refactor(odbc): enhance send_enriched_batch to process events in chun…
powerumc 853d002
fix(odbc): implement streaming fetch for variable-width columns and e…
powerumc b3531b0
fix(odbc): update metric counter for executed events to collect compl…
powerumc 3bcc65c
refactor(odbc): streamline event processing and metric recording in t…
powerumc dcde863
fix(odbc): handle SendError cases in send_enriched_batch and log Stre…
powerumc 80499c1
fix(odbc): improve error handling for column data type retrieval in O…
powerumc 7a7224e
fix(odbc): enhance error handling in send_enriched_batch and track dr…
powerumc a17af1b
chore(odbc): simplify execute_streaming_query call in ODBC client
powerumc 65b4349
fix(docker): add libodbc2 installation to Dockerfile for ODBC support
powerumc 6e72d5b
fix(odbc): enhance shutdown error handling to track dropped events af…
powerumc 18119b5
docs(odbc): enable checkpoint collection in ODBC source configuration
powerumc b5824e5
docs(odbc): update connection string example in configuration to use …
powerumc 6e2aefe
fix(odbc): clarify terminal and retryable error handling for SendErro…
powerumc f4348a7
fix(odbc): adjust shutdown error reporting to exclude in-flight event…
powerumc 9ed6526
fix(odbc): improve error handling for SendError cases by tracking dro…
powerumc bedc8f3
Merge branch 'master' into powerumc/odbc-source
powerumc 65b8a4b
Merge branch 'master' into powerumc/odbc-source
powerumc 55493e6
Merge branch 'master' into powerumc/odbc-source
powerumc File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -81,6 +81,7 @@ jobs: | |
| protoc: true | ||
| libsasl2: true | ||
| cargo-cache: true | ||
| unixodbc: true | ||
|
|
||
| - name: Run tests | ||
| run: | | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.