Define specs for CTA project layout

Context

Following #1265 and #1231, I think it is good that we clearly define the specs of what we are after. I already have all of this in my head (the linked issues were two of the major subtasks in this), but I think it's good to write it down so that we can agree on it. It also ensures that we have a clear goal and stay consistent in the future. Once we agree on this spec, I will add this to the public docs.

I realise this is quite a bit of work and moving things around; I will definitely not be implementing this in one go. I will split the work as necessary (e.g. per directory) to ensure reviews are easy and merge conflicts are minimised.

---

Spec

Conventions

The keywords “MUST”, “MUST NOT”, “REQUIRED”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “NOT RECOMMENDED”, “MAY”, and “OPTIONAL” in the specification are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

Definitions

• Library: Code meant to be consumed by other code; no runnable production entry points.

  • A library MUST be located under lib/
  • A library MUST NOT define a production entry point.
  • Test binaries are permitted but MUST reside under the test/ directory.

• Tool: Code that runs to completion and exits (finite lifetime). Typically CLI utilities.

  • A tool MUST be located under tools/
  • A tool MUST have at least one entry point (binary).
  • A tool SHOULD produce exactly one binary.
  • If multiple binaries are defined, they MUST be tightly coupled variants of the same conceptual tool.
    • Example: foo-encode and foo-decode may be produced from one foo/ tool directory if they share nearly all code.
    • Counterexample: foo and bar that only share a helper library MUST live in separate directories under /tools/.

• App: Code with a long-running or indefinite lifetime (services, daemons, GUIs).

  • An app MUST be located under app/
  • An app MUST have exactly one primary entry point (one deployed binary).

• Module: A module is a direct subdirectory of /app, /tools, or /lib.

  • Each module MUST produce exactly one output:
    • In /app and /tools, one executable.
    • In /lib, one library.
  • The layout MUST remain flat: modules MUST be direct children of /app, /tools, or /lib.
  • If modules are closely related and clarity is improved, a shared prefix MUST be used (e.g. frontend-xrd, frontend-grpc) rather than introducing nested directories.
  • A module directory MUST NOT contain subdirectories that define additional modules or outputs.

Top-level directory layout

The top-level directory should contain at least these files

app/ # Applications (one subdir per app)
cmake/ # cmake files 
external/ # Git submodules or external dependencies shared across multiple modules
lib/ # Libraries (one subdir per library)
tools/ # Short-lived tools/CLIs (one subdir per tool)
test/ # Integration and end-to-end tests
CMakeLists.txt

Module layout (applies to every subdirectory of /app, /tools, /lib)

• Source code MUST NOT live outside a module.

• Each module MUST contain:

  • src/ - source code only.
  • README.md - brief purpose and usage.
  • CMakeLists.txt - defines targets for this module (if C++).

• Each module MAY contain:

  • test/ - unit tests only; if present, tests MUST live here.
  • include/ - libraries only; public headers (see C++).
  • resources/ - runtime assets (non-source).
  • external/ - if the module uses a submodule that is not shared elsewhere.
  • scripts/ - for any utility (bash) scripts

• Code MUST NOT be shared across modules except via libraries under /lib.

• Libraries MUST NOT form cyclic dependencies. Apps and tools MUST depend only on libraries and external dependencies.

• Non-code files such as pdf, conf, service units, logrotate, txt, migrations, schemas MUST NOT live in src/.

• Module-specific runtime assets MUST be placed under that module’s resources/. Examples: myservice.conf, myservice.service, myservice.logrotate.

submodules:

• Any submodule MUST reside in an external/ directory.
• If a submodule is shared by multiple modules, it MUST live in the top-level /external/.
• If a submodule is only used by one module, it MAY live in that module’s external/.

Testing

• Per-module test/ directories SHOULD contain only unit tests and MUST mirror the src/ layout. Per-module test/ directories MAY contain integration tests if they only use components from that single module.
• Integration tests that require more than one module MUST be placed under the top-level /test/integration/ directory.
• End to end (or system) tests MUST be placed under the top-level /test/e2e/ directory.

C++ specifics

• Libraries MUST have include/ for public headers. Apps and tools MUST NOT have include/; all their headers go in src/.
• Public include layout MUST mirror namespaces:
  • include/foo/bar.hpp corresponds to namespace foo { ... } or namespace foo::bar { ... }.
• The include/<lib>/ directory MUST mirror the structure of the corresponding src/ directory.
  • Example: for a file src/rdbms/RdbmsCatalogue.cpp, the matching public header is include/catalogue/rdbms/RdbmsCatalogue.hpp (for library catalogue).
• Public headers MUST use the library name as the root include path.
  • Example: #include <catalogue/rdbms/RdbmsCatalogue.hpp>.
  • As a result, all public headers MUST live in include/<lib>/.
• Private headers MAY live in src/ and MUST NOT be installed or exposed.
• Product code MUST NOT include headers directly from arbitrary external paths; dependencies MUST be consumed via proper build targets.
• Headers MUST use #pragma once (or header guards consistently).
• Header files MUST end with .hpp. Source files MUST end with .cpp.
• Each .cpp SHOULD correspond to a same-named .hpp when applicable.
• C++ file names MUST use CamelCase and MUST NOT contain hyphens or underscores.
  • Example: MyService.cpp and MyService.hpp.
• All C++ code MUST follow the .clang-format defined at the repository root.

---

Let's try to keep the discussion of this issue focused on the spec and not the logistics.