This repository contains ELT code for an Iceberg-based data warehouse, together with infrastructure code used to run the warehouse both locally (docker-compose) and in an OpenStack cloud (Ansible).
Use this document as a concise, developer-friendly reference when working with the repository or providing context to an assistant/agent about the project structure and common developer tasks.
- Give human developers and LLM-based agents the essential repository layout and pointers for local development, testing and deployment.
- Point to the configuration locations used for certificates, local environments, and deployable Ansible playbooks/roles.
docs-devel/src/index.md#repository-overviewdescribes the repository layout.
- Static code checks are enforced using
prek. Install with your environment's tooling and run checks on all files withprek run --all-files. Install the pre-commit hook withprek install.
- There are both unit and e2e tests for the
elt-commonpackage underelt-common/tests/. They are written usingpytest. elt-common/README.mdexplains how to run tests for theelt-commonPython package
- Use the Ansible playbooks in
infra/ansible/together with theinventories/**files. See theinfra/ansible/readme.mdfor role and variable guidance.
- When creating pull requests:
-
Read the current PR template: Always check
.github/PULL_REQUEST_TEMPLATE.mdfor the latest format -
Follow PR title conventions: Use Conventional Commits
- Format:
type(scope): description - Example:
fix(warehouse/accelerator): fix join in model - Types:
fix,feat,docs,style,refactor,perf,test,chore
- Format:
Important: Always reference the actual template file at .github/PULL_REQUEST_TEMPLATE.md instead of using cached content, as the template may be updated over time.
- Docker resource issues: the local compose stack can be resource heavy. Ensure Docker Desktop has enough CPU/memory.
- Ansible role errors: ensure you have required galaxy roles (see
infra/ansible/ansible-galaxy-requirements.yaml) and the correct Python and Ansible versions installed.
- Read
docs/anddocs-devel/for high-level architecture and deployment instructions. - Inspect
warehouses/for per-warehouse ELT implementations and examples.
This file was created to provide a concise, shareable context document for humans and LLM-based agents working with the analytics-data-platform monorepo.