dfdr/README.md

DefDer (Data Registry Definition) - dfdr

A Python command-line tool for managing local data registries, inspired by DVC and Git but focused specifically on data registry functionality.

Overview

dfdr allows you to:

• Declare local data sources (folders containing CSV, JSON, YAML, TXT files)
• Add specific files from data registries to your working copy
• Mirror data locally for efficient access
• Track file changes with checksums
• Keep your working copy synchronized with local registries

Installation

Prerequisites

• Python 3.8 or higher
• pip (Python package installer)

Install from PyPI

pip install dfdr

Install from source

git clone git@defder.fr:dfdr.git
cd dfdr
pip install -e .

Quick Start

1. Initialize a dfdr repository:

dfdr init

2. Add a local data registry:

dfdr registry add myregistry /path/to/local/data/folder

3. Add files from the registry to your working copy:

dfdr add myregistry:datasets/sales.csv
dfdr add myregistry:models/config.json

4. Add a file to a specific subdirectory:

dfdr add myregistry:datasets/customers.csv -d ./data/customers

5. Update your local cache:

dfdr fetch

6. Check the status of your files:

dfdr status

7. Update your working copy:

dfdr pull

8. Push changes back to the registry:

dfdr push datasets/sales.csv

9. Move a file within the working copy:

dfdr move models/config.json ./configs/model_config.json

Detailed Usage

Managing Local Registries

# Add a local registry
dfdr registry add production /path/to/production/data

# List all registries
dfdr registry list

# Remove a registry
dfdr registry remove staging

Working with Files

# Add a specific file
dfdr add production:datasets/customer_data.csv

# Add a file to a specific subdirectory
dfdr add production:datasets/sales.csv -d ./data/sales

# Add an entire directory
dfdr add production:models

# Fetch all data to local cache
dfdr fetch

# Update working copy (all files)
dfdr pull

# Update a specific file
dfdr pull datasets/customer_data.csv

# Check status of all tracked files
dfdr status

# Push changes back to the registry
dfdr push datasets/customer_data.csv

# Move a file within the working copy
dfdr move models/config.json ./configs/model_config.json

Command Reference

dfdr init

Initialize a new dfdr repository in the current directory.

dfdr registry

Manage local data registries.

• add <name> <path>: Add a local data registry
• list: List all configured registries
• remove <name>: Remove a local data registry

dfdr add <remote_name>:<file_path> [-d <destination>]

Add a specific file or directory from a registry to your working copy. Optionally specify a destination subdirectory.

dfdr fetch

Mirror all data to local .dfdr storage from all remotes.

dfdr pull [file_path]

Update working copy from cache (all files or specific file).

dfdr status

Show sync status of files, including origin information.

dfdr push <file_path>

Push changes in a file back to its origin data registry.

dfdr move <current_path> <new_path>

Move a tracked file to a new destination within the working copy.

Data registry structure

Local data registries should follow this structure:

• files are stored in local directories
• the tool will automatically discover files in the directory

For example:

/path/to/registry/
├── datasets/
│   ├── sales.csv
│   └── customers.json
└── models/
    └── config.yaml

Architecture

dfdr is designed with a modular architecture:

• cli.py: Command-line interface using Click and Rich for display
• config.py: Configuration management (local registries, paths)
• storage.py: Local storage management and synchronization
• fetcher.py: Local file system operations
• checksum.py: MD5 checksum calculation and verification
• exceptions.py: Custom exception handling

Local Storage

dfdr creates a .dfdr directory in your project containing:

• config.json: Local registry configuration
• storage/: Local mirror of registry data
• *.dfdr files: YAML files containing MD5 checksums and origin information for each data file (e.g., sales.csv.dfdr)

The .dfdr files now include additional information:

• MD5 checksum of the file
• Registry name (origin)
• Original path in the registry

This enhanced structure allows for better tracking and management of files across different registries.

Development Setup

1. Clone the repository
2. Create a virtual environment: python -m venv venv
3. Activate the virtual environment:
   • On Windows: venv\Scripts\activate
   • On macOS and Linux: source venv/bin/activate
4. Install development dependencies: pip install -r requirements.txt
5. Install the package in editable mode: pip install -e .

License

This project is licensed under the MIT License - see the LICENSE file for details.