UJDS: a Universal JSON Data Storage

The Universal JSON Data Storage stores arbitrary JSON data and keeps changes history. Data are being stored in indices as records. Indices may have schema to check incoming data upon updates.

Configuration

The service can be configured in three ways: via YAML file, via env variables or using both. Env variables take precedence over config file.

If the config.yaml file is found in the current directory, it will be loaded before env variables. It is possible to change default config file location using APP_CONFIG_PATH env variable.

File

required object db: database configuration.
- required string dsn: database source name.
optional object server: server configuration.
- optional string address: network address, default is :9000.
- optional string auth_token: authorization token (full access).
- optional string auth_token_ro: read-only authorization token; grants access to read-only methods only (IndexService/Get, IndexService/List, RecordService/Get, RecordService/Find, RecordService/History).

Env variables

required string UJDS_DB_DSN: database source name.
optional string UJDS_SERVER_ADDRESS: server network address.
optional string UJDS_SERVER_AUTHTOKEN: server authorization token (full access).
optional string UJDS_SERVER_AUTHTOKENRO: read-only server authorization token.

HTTP API

Methods

All the requests must be performed use POST method.

Response JSON types

Please note that numerical data in responses are encoded as strings.

Authorization

If the server.auth_token configuration parameter is specified, the server will expect an Authorization: Bearer XXX HTTP header where XXX matches the configured token, otherwise 401 is returned. If server.auth_token_ro is also configured, that token may be used instead for read-only methods (IndexService/Get, IndexService/List, RecordService/Get, RecordService/Find, RecordService/History). Using the read-only token on a write method returns 403.

Error handling

If a request is not successful, the service responds with an HTTP status code other than 200, providing a JSON object with the code and message fields. Use that information to understand what went wrong. Example:

curl --request POST \
  --url https://localhost:9000/ujds.index.v1.IndexService/Push \
  --header 'Authorization: Bearer WrongAuthToken' \
  --header 'Content-Type: application/json' \
  --data '{}'

< HTTP/2 401
< content-type: application/json
<
{"code":"unauthenticated","message":"not authorized"}

Search query syntax

The RecordService/Find method provides a method of filtering result using search queries. The syntax has to be described here.

IndexService/Push

Creates a new index or updates an existing one.

Request fields:
- required string name: index name. The allowed format: ^[a-zA-Z0-9.-]{1,255}$.
- optional string title: index title.

Request example:

curl --request POST \
  --url https://localhost:9000/ujds.index.v1.IndexService/Push \
  --header 'Authorization: Bearer YourAuthToken' \
  --header 'Content-Type: application/json' \
  --data '{
	"name": "books",
	"title": "The books"
}'

IndexService/Get

Returns an index metadata.

Request fields:
- required string name: index name. The allowed format: ^[a-zA-Z0-9.-]{1,255}$.
Response fields:
- string name: index name.
- string title: index title.
- int createdAt: creation UNIX timestamp.
- int updatedAt: update UNIX timestamp.
- []string schemas: the record validation schemas that apply to the index. Schemas are bound to index name regexp patterns (see validation.index configuration) and are applied to record data on RecordService/Push; these are exactly the ones a record pushed to that index would be validated against. May be empty if no schema is configured for the index. The implicit catch-all (which only requires record data to be valid JSON) is not included. Each item is a JSON schema encoded as a string. A $schema dialect declaration is added if the configured schema does not already have one; the service validates against draft-07 (http://json-schema.org/draft-07/schema#), so that is the dialect stamped onto schemas missing one.

Request example:

curl --request POST \
  --url https://localhost:9000/ujds.index.v1.IndexService/Get \
  --header 'Authorization: Bearer YourAuthToken' \
  --header 'Content-Type: application/json' \
  --data '{"name": "books"}'

Response example:

{
  "name": "books",
  "title": "The books",
  "createdAt": "1693768684",
  "updatedAt": "1693769057",
  "schemas": [
    "{\"$schema\":\"http://json-schema.org/draft-07/schema#\",\"type\":\"object\",\"required\":[\"title\"]}"
  ]
}

IndexService/List

Returns existing indices list.

Request fields:
- optional object filter: filter.
  - optional []string names: index name patterns. Allowed wildcard symbols: *.
Response fields:
- []object indices
  - string name: index name.
  - string title: index title.

Request example:

curl --request POST \
  --url https://localhost:9000/ujds.index.v1.IndexService/List \
  --header 'Authorization: Bearer YourAuthToken' \
  --header 'Content-Type: application/json' \
  --data '{"filter":{"names": ["book*", "recip*", "cartoons"]}}'

Response example:

{
  "indices": [
    {
      "name": "books",
      "title": "The books"
    },
    {
      "name": "recipes",
      "title": "The recipes"
    },
    {
      "name": "cartoons",
      "title": "The cartoons"
    }
  ]
}

IndexService/Clear

Clears all index records.

Request fields:
- required string name: index name. The allowed format: ^[a-zA-Z0-9.-]{1,255}$.

Request example:

curl --request POST \
  --url https://localhost:9000/ujds.index.v1.IndexService/Clear \
  --header 'Authorization: Bearer YourAuthToken' \
  --header 'Content-Type: application/json' \
  --data '{}'

RecordService/Push

Creates records in the index or updates existing ones.

Request fields:
- required []object records: records.
  - required string index: index name. The allowed format: ^[a-zA-Z0-9.-]{1,255}$.
  - required string id: record ID.
  - required string data: record JSON data.

Request example:

curl --request POST \
  --url https://localhost:9000/ujds.record.v1.RecordService/Push \
  --header 'Authorization: Bearer YourAuthToken' \
  --header 'Content-Type: application/json' \
  --data '{
	"records": [
		{
			"index": "books",
			"id": "castaneda-001",
			"data": "{\"title\": \"Tales of Power\", \"author\": \"Carlos Castaneda\", \"isbn\":\"978-0-671-73252-3\"}"
		},
		{
			"index": "books",
			"id": "tanenbaum-001",
			"data": "{\"author\":\"M. van Steen and A.S. Tanenbaum\", \"title\":\"Distributed Systems, 4th ed.\"}"
		}
	]
}'

RecordService/Get

Returns a single record.

Request fields:
- required string index: index name. The allowed format: ^[a-zA-Z0-9.-]{1,255}$.
Response field:
- object record
  - string id: ID.
  - string index: index name.
  - string rev: revision number.
  - string createdAt: creation time as UNIX timestamp.
  - string updatedAt: last change time as UNIX timestamp.
  - string touchedAt: last update time as UNIX timestamp.
  - string data: data.

Request example:

curl --request POST \
  --url https://localhost:9000/ujds.record.v1.RecordService/Get \
  --header 'Authorization: Bearer YourAuthToken' \
  --header 'Content-Type: application/json' \
  --data '{
	"index": "books",
	"id": "castaneda-001"
}'

Response example:

{
  "record": {
    "id": "castaneda-001",
    "rev": "227",
    "index": "books",
    "createdAt": "1694109017",
    "updatedAt": "1694237265",
    "touchedAt": "1702938162",
    "data": "{\"title\": \"Tales of Power\", \"author\": \"Carlos Castaneda\", \"isbn\":\"978-0-671-73252-3\"}"
  }
}

RecordService/Find

Returns all records from the index.

Request fields:
- required string index: index name. The allowed format: ^[a-zA-Z0-9.-]{1,255}$.
- optional string search: search query. TODO: describe search query syntax.
- optional int since: return only records, that have been modified since provided UNIX timestamp.
- optional int touchedSince: return only records, that have been touched since a UNIX timestamp.
- optional int notTouchedSince: return only records, that have not been touched since a UNIX timestamp.
- optional int cursor: pagination: return records starting from provided position.
- optional int limit: get only specified number of records; default and maximum is 500.
Response fields:
- string cursor: pagination cursor position, that should be used to retrieve the next result set.
- []object records
  - string id: ID.
  - string index: index name.
  - string rev: revision number.
  - string createdAt: creation time as UNIX timestamp.
  - string updatedAt: last change time as UNIX timestamp.
  - string touchedAt: last update time as UNIX timestamp.
  - string data: data.

Request example:

curl --request POST \
  --url https://localhost:9000/ujds.record.v1.RecordService/Find \
  --header 'Authorization: Bearer YourAuthToken' \
  --header 'Content-Type: application/json' \
  --data '{
	"index": "books",
	"search": "author=\"Carlos Castaneda\"",
	"since": 1694109017,
	"cursor": 226,
	"limit": 2
}'

Response example:

{
  "cursor": "228",
  "records": [
    {
      "id": "castaneda-001",
      "rev": "227",
      "index": "books",
      "createdAt": "1694109017",
      "updatedAt": "1694109017",
      "touchedAt": "1702938162",
      "data": "{\"title\": \"Tales of Power\", \"author\": \"Carlos Castaneda\", \"isbn\":\"978-0-671-73252-3\"}"
    },
    {
      "id": "castaneda-002",
      "rev": "228",
      "index": "books",
      "createdAt": "1694109017",
      "updatedAt": "1694109017",
      "touchedAt": "1702938162",
      "data": "{\"title\": \"The Fire From Within\", \"author\": \"Carlos Castaneda\", \"isbn\":\"978-0-671-73250-9\"}"
    }
  ]
}

RecordService/History

Returns record history.

Request fields:
- required string index: index name. The allowed format: ^[a-zA-Z0-9.-]{1,255}$.
- required string id: record id.
- optional int since: return only history records, which have been created since provided UNIX timestamp.
- optional int cursor: pagination: return records starting from provided position.
- optional int limit: get only specified number of records; default and maximum is 500.
Response fields:
- string cursor: pagination cursor position, which should be used to retrieve the next result set.
- []object records
  - string id: record ID.
  - string index: index name.
  - string rev: revision number.
  - string createdAt: creation time as UNIX timestamp.
  - string data: data.

Request example:

curl --request POST \
  --url http://localhost:9000/ujds.record.v1.RecordService/History \
  --header 'Authorization: Bearer YourAuthToken' \
  --header 'Content-Type: application/json' \
  --data '{
	"index": "books",
	"id": "castaneda-001",
	"since": 1696767680,
	"cursor": 28,
	"limit": 2
}'

Response example:

{
  "records": [
    {
      "id": "castaneda-001",
      "rev": "30",
      "index": "books",
      "createdAt": "1696768530",
      "data": "{\"title\": \"Tales of Power, second edition\", \"author\": \"Carlos Castaneda\", \"isbn\":\"978-0-671-73252-3\"}"
    },
    {
      "id": "castaneda-001",
      "rev": "28",
      "index": "books",
      "createdAt": "1696767687",
      "data": "{\"title\": \"Tales of Power\", \"author\": \"Carlos Castaneda\", \"isbn\":\"978-0-671-73252-3\"}"
    }
  ]
}

Developers notes

Create migration:

go install -tags 'postgres' github.com/golang-migrate/migrate/v4/cmd/migrate@latest
migrate create -ext .sql -dir internal/migration/migrations foobar

Changelog

0.12 (2026-06-29)

auth_token_ro config option added. When set, allows read-only access to IndexService/Get, IndexService/List, RecordService/Get, RecordService/Find, and RecordService/History without granting write access.

0.11 (2026-06-11)

IndexService.GetSchema() RPC removed; IndexService.Get() now returns the matching schemas in a schemas array.

0.10.1 (2026-06-08)

IndexService.GetSchema() RPC response improved.

0.10 (2026-06-08)

IndexService.GetSchema() RPC added.

0.9.2 (2025-12-09)

Fixed config parsing.

0.9.1 (2025-12-09)

Fixed tests.

0.9 (2025-12-09)

Add config-based per-index schema validation.

0.8 (2025-12-05)

Get rid of per-index schema.
Global schema introduced.

0.7.1 (2025-12-04)

Fixed HTTP server default address configuration.

0.7 (2025-12-04)

RecordService/Find request got the new touchedSince argument.

0.6 (2025-09-02)

RecordService/Find request got the new notTouchedSince argument.

0.5 (2024-03-30)

The filter request field added to IndexService/List method.

0.4 (2024-03-09)

RecordService/Push:
- now the index should be specified on each record;
- a new touched_at field added; it is always being updated with a current timestamp, even if record hasn't changed.

0.3 (2023-11-29)

RecordService/GetAll API method renamed to RecordService/Find.
A new search optional field added to RecordService/Find.

0.2 (2023-10-08)

RecordService/History API method added.
Index name length extended to 255 chars.
Slash is not allowed in index names anymore; replaced with dot.

0.1 (2023-09-07)

Initial release.

Authors

Oleksandr Shepetko

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
docs/superpowers		docs/superpowers
helm		helm
internal		internal
proto/ujds		proto/ujds
sdk		sdk
sql		sql
tests		tests
.gitignore		.gitignore
CONTRIBUTING.md		CONTRIBUTING.md
README.md		README.md
Taskfile.yaml		Taskfile.yaml
buf.gen.yaml		buf.gen.yaml
buf.yaml		buf.yaml
docker-compose.tests.yaml		docker-compose.tests.yaml
go.mod		go.mod
go.sum		go.sum
main.go		main.go

Latest commit

History

Folders and files

Repository files navigation

UJDS: a Universal JSON Data Storage

Configuration

File

Env variables

HTTP API

Methods

Response JSON types

Authorization

Error handling

Search query syntax

IndexService/Push

IndexService/Get

IndexService/List

IndexService/Clear

RecordService/Push

RecordService/Get

RecordService/Find

RecordService/History

Developers notes

Changelog

0.12 (2026-06-29)

0.11 (2026-06-11)

0.10.1 (2026-06-08)

0.10 (2026-06-08)

0.9.2 (2025-12-09)

0.9.1 (2025-12-09)

0.9 (2025-12-09)

0.8 (2025-12-05)

0.7.1 (2025-12-04)

0.7 (2025-12-04)

0.6 (2025-09-02)

0.5 (2024-03-30)

0.4 (2024-03-09)

0.3 (2023-11-29)

0.2 (2023-10-08)

0.1 (2023-09-07)

Authors

About

Resources

Contributing

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages