{"user":"gstack","name":"keyval-resource","namespace":"gstack","repository_type":"image","status":1,"status_description":"active","description":"A Concourse resource that passes key-value pairs between jobs","is_private":false,"is_automated":false,"star_count":3,"pull_count":1966261,"last_updated":"2025-03-16T17:11:38.468443Z","last_modified":"2025-03-16T17:11:38.290733Z","date_registered":"2021-11-07T15:49:49.813256Z","collaborator_count":0,"affiliation":null,"hub_user":"bgandon","has_starred":false,"full_description":"\n[![Docker Stars](https://img.shields.io/docker/stars/gstack/keyval-resource.svg?style=plastic)](https://registry.hub.docker.com/v2/repositories/gstack/keyval-resource/stars/count/)\n[![Docker pulls](https://img.shields.io/docker/pulls/gstack/keyval-resource.svg?style=plastic)](https://registry.hub.docker.com/v2/repositories/gstack/keyval-resource)\n\u003c!--\n[![Concourse Build](https://ci.gstack.io/api/v1/teams/gk-plat-devs/pipelines/keyval-resource/jobs/build/badge)](https://ci.gstack.io/teams/gk-plat-devs/pipelines/keyval-resource)\n--\u003e\n[![dockeri.co](https://dockeri.co/image/gstack/keyval-resource)](https://hub.docker.com/r/gstack/keyval-resource/)\n\n# Concourse key-value resource\n\nImplements a resource that passes sets of key-value pairs between jobs without\nusing any external storage with resource like [Git][git_resource] or\n[S3][s3_resource].\n\nPulled by a `get` step, key-value pairs are provided to the build plan as an\nartifact directory with one file per key-value pair. The name of a file is\nthe “key”, and its contents is the “value”. These key-value pairs can then be\nloaded as local build vars using a [`load_var` step][load_var_step].\n\nPushed by a `put` step, key-value pairs are persisted in the Concourse SQL\ndatabase. For this to be possible, the trick is that they are serialized as\nkeys and values in [`version` JSON objects][version_schema]. As such, they\nare designed to hold _small_, _textual_, _non-secret_ data.\n\nIn case you're dealing with large text, binary data or secrets, we recommend\nyou opt for other solutions. Indeed, secrets will be best stored in a vault\nlike [CredHub][credhub], large text files in [Git][git_resource], and binary\ndata in some [object storage][s3_resource] or [Git][git_resource] with\nGit-LFS, the “[Large File Storage][git_lfs]” addon.\n\n[git_resource]: https://github.com/concourse/git-resource\n[s3_resource]: https://github.com/concourse/s3-resource\n[load_var_step]: https://concourse-ci.org/load-var-step.html\n[version_schema]: https://concourse-ci.org/config-basics.html#schema.version\n[git_lfs]: https://git-lfs.com/\n[credhub]: https://github.com/pivotal/credhub-release\n\n\n\n## Credits\n\nThis resource is a fork of the [`keyval` resource][moredhel_gh] by\n[@moredhel](https://github.com/moredhel), thanks for the work done!\n\n## Comparison to similar resources\n\n### SWCE/keyval-resource (Java `.properties`-based)\n\nCompared to the [original `keyval` resource][swce_gh] from SWCE by\n[@regevbr](https://github.com/regevbr) and [@ezraroi](https://github.com/ezraroi),\nwriting key-value pairs as plain files in some resource folder is more\nconsistent with usual conventions in Concourse, when it comes to storing\nanything in step artifacts. It is also compliant with the ConfigMap pattern\nfrom Kubernetes.\n\nWriting/reading files is always easier in Bash scripts than parsing some Java\nProperties file, because much less boilerplate code is required.\n\n### moredhel/keyval-resource (unfinished)\n\nInital [v1.1.0][v1_1_0] release has fixed many defects, that made it not\nusable (non-working `put` steps).\n\n### cludden/concourse-keyval-resource\n\nThat one implements [Bloblang][about_bloblang]-generated contents, and only\nallow creating new key-values through the `params` of `put` steps, not through\nwriting plain files in some artifact directory of `task`steps like we do here.\n\n[moredhel_gh]: https://github.com/moredhel/keyval-resource\n[swce_gh]: https://github.com/SWCE/keyval-resource\n[v1_1_0]: https://github.com/gstackio/keyval-resource/releases/tag/v1.1.0\n[about_bloblang]: https://www.benthos.dev/docs/guides/bloblang/about\n\n\n\n## Source Configuration\n\n```yaml\nresource_types:\n  - name: key-value\n    type: registry-image\n    source:\n      repository: gstack/keyval-resource\n\nresources:\n  - name: key-value\n    type: key-value\n```\n\n#### Parameters\n\n- `history_identifier`: _Optional._\n  When the [“global resources” feature][gbl_rsc_docs] is enabled on your\n  Concourse installation, and you don't want a single resource history for all\n  the keyval resources defined in your Concourse installation, then set this\n  property to a relevant identifier, possibly unique or not. See the\n  [“global resources” section](#discussion-on-global-resources) for a detailed\n  discussion on use-cases and solutions.\n\n[gbl_rsc_docs]: https://concourse-ci.org/global-resources.html\n\n## Behavior\n\n### `check` Step (`check` script): Report the latest stored key-value pairs\n\nThis is a version-less resource so `check` behavior is no-op.\n\nIt will detect the latest store key/value pairs, if any, and won't provide any\nversion history.\n\n#### Parameters\n\n*None.*\n\n### `get` Step (`in` script): Fetch the latest stored key-value pairs from the Concourse SQL database\n\nFetches the given key \u0026 values from the stored resource version JSON (in the\nConcourse SQL database) and write them in their respective files where the\nkey is the file name and the value is the file contents.\n\n```json\n\"version\": { \"some_key\": \"some_value\" }\n```\n\nwould result in:\n\n```bash\n$ cat resource/some_key\nsome_value\n```\n\n#### Parameters\n\n*None.*\n\n### `put` Step (`out` script): Store new set of key-value pairs to the Concourse SQL database\n\nConverts each file in the artifact directory designated by `directory` to a\nset of key-value pairs, where file names are the keys and file contents are\nthe values. This set of key-value pairs is persisted in the `version` JSON\nobject, to be stored in the Concourse SQL database.\n\nA value from a file in `directory` can be overridden by a matching key with\ndifferent value in the dictionary given as the `overrides` parameter. If you\nneed to store some Concourse `((vars))` value in a key-value resource, then\nadd it to the `overrides` parameter of some `put` step.\n\n#### Parameters\n\n- `directory`: *Required.* The artifact directory to be scanned for files, in\n  order to generate key-value pairs\n\n- `overrides`: *Optional.* A dictionary of key-value pairs that will override\n  any matching pair with same key found in `directory`.\n\n\n\n## Examples\n\n### Summarized example\n\nThis example make intentional ellipsis in order to focus on the main ideas\nbehind the “keyval” resource. Seasoned Concourse practitioners can find an\nillustration here in one catch.\n\n```yaml\nresource_types:\n  - name: key-value\n    type: registry-image\n    source:\n      repository: gstack/keyval-resource\n\nresources:\n  - name: build-info\n    type: key-value\n\njobs:\n\n  - name: build\n    plan:\n      - task: build\n        file: tools/tasks/build/task.yml # \u003c- must declare a 'build-info' output artifact\n      - put: build-info\n        params:\n          directory: build-info\n\n  - name: test-deploy\n    plan:\n      - in_parallel:\n          - get: build-info\n            passed: [ build ]\n      - task: test-deploy\n        file: tools/tasks/task.yml # \u003c- must declare a 'build-info' input artifact\n```\n\nThe `build` task writes all the key-value pairs it needs to pass along in\nfiles inside the `build-info` output artifact directory.\n\nThe `test-deploy` job then reads the files from the `build-info` resource,\nwhich produces a `build-info` artifact directory to be used by the\n`test-deploy` task.\n\n\n### Detailed example\n\nThis fully-working and detailed example goes deeper in showcasing what the\nresource can actually do and how. Concourse beginners are recommended to read\nthis as it details very clearly the relation between resource, artifact\ndirectories, and tasks.\n\n```yaml\nresource_types:\n  - name: key-value\n    type: registry-image\n    source: { repository: gstack/keyval-resource }\n\nresources:\n  - name: some-keyval-resource\n    type: key-value\n  - name: runner-image\n    type: registry-image\n    source: { repository: busybox }\n\njobs:\n  - name: step-1-job\n    plan:\n      - get: runner-image\n      - task: write-keyval-aaa-1-task\n        image: runner-image\n        config:\n          platform: linux\n          outputs: [ { name: created-keyvals-artifact } ]\n          run:\n            path: sh\n            args:\n              - -exc\n              - |\n                echo \"1\" \u003e created-keyvals-artifact/aaa\n      - put: some-keyval-resource\n        params:\n          directory: created-keyvals-artifact\n  - name: step-2-job\n    plan:\n      - in_parallel:\n          - get: keyvals-artifact          # here artifact directory is\n            resource: some-keyval-resource # different from resource name\n            trigger: true\n            passed: [ step-1-job ]\n          - get: runner-image\n      - task: read-aaa-keyval-task\n        image: runner-image\n        config:\n          platform: linux\n          inputs: [ { name: keyvals-artifact } ]\n          run:\n            path: sh\n            args:\n              - -exc\n              - |\n                cat keyvals-artifact/aaa  # -\u003e 1\n      - task: write-bbb-keyval-task\n        image: runner-image\n        config:\n          platform: linux\n          inputs:  [ { name: keyvals-artifact } ]\n          outputs: [ { name: keyvals-artifact } ]\n          run:\n            path: sh\n            args:\n              - -exc\n              - |\n                echo \"2\" \u003e build-info/bbb\n      - put: some-keyval-resource\n        params:\n          directory: keyvals-artifact\n          overrides:\n            aaa: \"11\"\n            ccc: \"3\"\n  - name: step-3-job\n    plan:\n      - in_parallel:\n          - get: some-keyval-resource # artifact dir will have same name\n            trigger: true\n            passed: [ step-2-job ]\n          - get: runner-image\n      - task: read-aaa-bbb-ccc-keyvals-task\n        image: runner-image\n        config:\n          platform: linux\n          inputs: [ { name: some-keyval-resource } ]\n          run:\n            path: sh\n            args:\n              - -exc\n              - |\n                cat build-info/aaa  # -\u003e 11\n                cat build-info/bbb  # -\u003e 2\n                cat build-info/ccc  # -\u003e 3\n```\n\nThe `write-keyval-aaa-1-task` creates a file named `aaa` with content `1` to\nthe `created-keyvals-artifact` output artifact directory. The\n`some-keyval-resource` resource will read files in the\n`created-keyvals-artifact` directory and store a key-value pair `\n{\"aaa\": \"1\"}`.\n\nThe `read-aaa-keyval-task` reads the value from the `aaa` file in\n`keyvals-artifact` input artifact directory provided from the\n`some-keyval-resource` resource. This outputs `1`.\n\nThe `write-bbb-keyval-task` creates a file named `bbb` with content `2` to\n`keyvals-artifact` output artifact directory. Because this directory is same\nas `keyvals-artifact` input artifact directory which already contains `aaa`.\nThe `some-keyval-resource` resource will read all files in the\n`keyvals-artifact` directory and store key-value pairs `{\"aaa\": \"1\"}` and `\n{\"bbb\": \"2\"}`.\n\nThe `put: some-keyval-resource` in `step-2-job` provides the `overrides`\noption, which changes the original key-value pair `{\"aaa\": \"1\"}` to `\n{\"aaa\": \"11\"}` and add a new pair `{\"ccc\": \"3\"}`.\n\nThe `read-aaa-bbb-ccc-keyvals-task` reads values from files in the\n`some-keyval-resource` input artifact directory, as provided by the\n`some-keyval-resource` resource.\n\n\n\n## Discussion on “global resources”\n\nWhen the “[global resources][gbl_rsc_docs]” feature is enabled, all resources\nwith same `resource.type` and `resource.source` configuration will share the\nsame version history. If you leave the `resource.source` configuration blank\nin all your keyval resources, then they will _all_ be considered the exact\nsame resource by Concourse, sharing the exact same history.\n\nFor most keyval resource-related use-case though, this is not releavant and\nthus requires proper scoping.\n\n### Scenario #1: pipeline-private key-values\n\nIn many scearios, the key-value resources is used to transmit\npipeline-specific data between jobs of the same pipeline. In such case,\nsharing resource history is most probably irrelevant. In order to avoid this,\nyou can set the `history_identifier` to some value that will be unique in your\nConcourse installation.\n\nFor best portability of your pipeline across different Concourse\ninstallations, we recommend that you use a UUID that can be generated with the\n`uuidgen` command-line tool like this:\n\n```shell\n$ uuidgen | tr [:upper:] [:lower:]\nfc4cb2ba-d0d4-44e2-8589-8fa89a8271fd\n```\n\nThen use it in the resource configuration, so that the resource history is\nscoped privately:\n\n```yaml\nresources:\n  - name: key-value\n    type: key-value\n    source:\n      history_identifier: fc4cb2ba-d0d4-44e2-8589-8fa89a8271fd\n```\n\n### Scenario #2: shared key-values between pipelines\n\nIn some scenarios though, it may be interesting to share the resources history\nbetween different pipelines. Then you can leverage key-value resources that\nshare the same `history_identifier` value.\n\nAs a result, as soon as a new version is pushed on the shared key-value\nresources, all other pipelines will see it.\n\n#### Example use-case: triggering other pipelines\n\nThis is interesting in case some pipeline has to trigger other pipelines. A\nusual solution is to use a “dummy” `semver` resource, backed by Git or some\nobject storage.\n\nUsing the keyval resource can bring an elegant alternative. A limitation is\nthat this resource basically has no version history. At every point in time,\nonly the last vesion exisits for the resource. This is not an issue for this\nuse-case, though. Indeed with a “dummy” `semver` resource, experience shows\nthat nobody actually pays attention to the version history anyway.\n\nWith the keyval resource, the triggering version only need to specify the date\nand relevant data showing the reason why the pipeline has been triggered.\nThese will appear and properly stay in job build logs, for later inspection.\n\n\n\n## Migrating from previous key-value resources\n\n### Migrating from `SWCE/keyval-resource`\n\nKey-value pairs are no more written as Java `.properties` file, but rather one\nfile per key-value pair. The name of a file is a “key”, and its contents is\nthe related “value”.\n\nThe required `file` paramerter for `put` steps is replaced by `directory`.\n\n### Migrating from `moredhel/keyval-resource`\n\nThe required `directory` paramerter has been added to `put` steps.\n\nThe `file` parameter of `put` steps is renamed `overrides`.\n\n\n\n\n\n\n## Author and license\n\nCopyright © 2021-present, Benjamin Gandon, Gstack\n\nLike Concourse, the key-value resource is released under the terms of the\n[Apache 2.0 license](http://www.apache.org/licenses/LICENSE-2.0).\n\n\u003c!--\n# Local Variables:\n# indent-tabs-mode: nil\n# End:\n--\u003e","permissions":{"read":true,"write":false,"admin":false},"media_types":["application/vnd.docker.container.image.v1+json"],"content_types":["image"],"categories":[],"immutable_tags_settings":{"enabled":false,"rules":[".*"]},"storage_size":113252726,"source":null}