Atlas Open Magic REST API

This project provides a RESTful API for managing "releases" and their associated datasets. It is built with FastAPI for the web server, SQLAlchemy for database interactions via an ORM, and Pydantic for data validation.

Table of Contents

• Atlas Open Magic REST API
  • Table of Contents
  • Features
  • Requirements
  • Installation
  • Configuration
  • Database and Migrations
    • Database Models
  • Running the Server
  • API Reference
    • Create or Update a Release
    • Update Dataset(s) Fields
      • Update a single dataset:
      • Bulk update multiple datasets:
    • List Releases
    • Get Release by Name
    • List Datasets
    • Get Dataset by Number
  • Project Structure
  • Logging
  • Error Handling

Features

• Upsert Datasets: Creates a release if it doesn't exist and then creates or updates datasets from an uploaded JSON or CSV file.
• Granular Updates: Easily update individual fields in one dataset or multiple datasets without re-uploading full files.
• Idempotent: The upload process is idempotent; datasets are identified by their dataset_number, preventing duplicates and allowing for easy updates.
• Data Validation: Leverages Pydantic for robust validation of incoming data against a defined schema.
• Database Agnostic: Built with SQLAlchemy to support different database backends (e.g., PostgreSQL, SQLite).
• Automatic API Docs: Provides interactive API documentation via Swagger UI and ReDoc.

Requirements

• Python 3.9+
• A database supported by SQLAlchemy (e.g., PostgreSQL for production, SQLite for development). Python Dependencies
• fastapi
• uvicorn
• sqlalchemy
• alembic (for database migrations)
• pandas
• pydantic
• A database driver (e.g., psycopg2-binary for PostgreSQL)

Installation

1. Clone the repository:

   git clone https://github.com/your_org/atlas-open-magic.git
   cd atlas-open-magic

2. Create and activate a virtual environment:

   python -m venv .venv
   source .venv/bin/activate

3. Install dependencies:

   pip install -r requirements.txt

4. Create your local configuration file:

   cp alembic.ini.example alembic.ini

   Now, edit alembic.ini to add your database credentials.

Configuration

The application and Alembic are configured via alembic.ini and environment variables.

• alembic.ini: This file contains the primary configuration, including the sqlalchemy.url. Do not commit this file to Git, as it contains secrets. Use alembic.ini.example as a template.
• SQLALCHEMY_DATABASE_URL: Your FastAPI application may use this environment variable to connect to the database.

For PostgreSQL:

# In alembic.ini
sqlalchemy.url = postgresql://user:pass@localhost:5432/atlasdb

For SQLite (Development):

# In alembic.ini
sqlalchemy.url = sqlite:///./app.db

Database and Migrations

This project uses Alembic to manage database schema migrations. This ensures that the database schema stays in sync with the SQLAlchemy models defined in app/models.py. The migration scripts are stored in the app/alembic/versions directory and must be committed to version control.

1. Initialize the Database (First-Time Setup): After configuring your alembic.ini, create all tables by running:

   alembic upgrade head

2. Applying Model Changes (After you modify app/models.py): a. Generate a migration script: Alembic will compare your models to the database and generate a script.

   alembic revision --autogenerate -m "A brief description of model changes"

   b. Apply the changes: Run the migration script to alter the database.

   alembic upgrade head

Database Models

• Release (releases table)
  • name (String, Primary Key)
  • id (Integer, auto-incrementing)
  • datasets (Relationship to Dataset)
• Dataset (datasets table)
  • release_name (String, Foreign Key to releases.name)
  • dataset_number (Integer, Primary Key)
  • ...all other dataset-specific fields as listed above...

Running the Server

Start the API server using Uvicorn:

uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

The API documentation is automatically generated by FastAPI and available at:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

API Reference

Create or Update a Release

Creates a release if it doesn't exist and then creates or updates (upserts) datasets from an uploaded JSON or CSV file. Datasets are identified by their dataset_number.

PUT /release/create/{release_name}
Content-Type: multipart/form-data

• Path Parameters:
  • release_name (string, required): The name of the release to create or update.
• Request Body:
  • file (file, required): JSON or CSV of datasets.
• Responses: 200, 400, 415, 422, 500
• Example:

  curl -X PUT -F "file=@/path/to/your/data.json" http://localhost:8000/release/create/2024-new-release

Update Dataset(s) Fields

Update individual fields in one dataset, or across multiple datasets, without needing to re-upload all data.

Update a single dataset:

PATCH /release/update/{release_name}/{dataset_number}
Content-Type: application/json

Path Parameters:

• release_name (string, required): The name of the release the dataset belongs to.
• dataset_number (string, required): The unique dataset identifier. Request Body:
• updates (object): Dictionary of field names and new values.

Example:

curl -X PATCH "http://localhost:8000/release/update/2025e/301204" \
  -H "Content-Type: application/json" \
  -d '{"updates": {"sumOfWeightsSquared": 1}}'

Bulk update multiple datasets:

PATCH /release/update/{release_name}
Content-Type: application/json

Path Parameters:

• release_name (string, required): The name of the release to update datasets in. Request Body:
• datasets (list): List of updates, each entry should have a dataset_number and a dictionary of updates.

Example:

curl -X PATCH "http://localhost:8000/release/update/2025e" \
  -H "Content-Type: application/json" \
  -d '{"datasets":[{"dataset_number":"301204","updates":{"sumOfWeightsSquared":1}},{"dataset_number":"301209","updates":{"kFactor":2.5,"cross_section_pb":111.22}}]}'

Responses:

• 200 OK: Success summary (how many updated, not found).
• 404 Not Found: If a dataset or release does not exist.
• 422 Unprocessable Entity: Invalid update structure or field type.
• 500 Internal Server Error: Unexpected server error.

List Releases

GET /releases

Retrieve a list of all releases.

Get Release by Name

GET /releases/{release_name}

Fetch a specific release and its associated datasets.

List Datasets

GET /datasets?release_name={name}&skip=0&limit=100

Retrieve a list of datasets, with optional pagination and filtering by release.

Get Dataset by Number

GET /metadata/{release_name}/{dataset_number}

Fetch a single dataset by its unique number and release.

Project Structure

.
├── app/
│   ├── alembic/              # Alembic migration scripts (commit to Git)
│   │   └── versions/
│   ├── crud.py           # Database CRUD operations
│   ├── database.py       # DB engine and session dependency
│   ├── main.py           # API endpoints and startup logic
│   ├── models.py         # SQLAlchemy model definitions
│   └── schemas.py        # Pydantic schemas for validation
├── alembic.ini           # Alembic config (DO NOT COMMIT)
├── alembic.ini.example   # Template config (commit to Git)
├── requirements.txt      # Python dependencies
└── README.md             # This file

Logging

• Uses Python's built-in logging module.
• Logs key lifecycle events, transaction statuses, and errors for easier debugging.

Error Handling

• Uses FastAPI's HTTPException for meaningful error codes.
• Upload and update operations run in a single DB transaction for atomicity: failures roll back.