dfdr/README.md

# DefDer (Data Registry Definition) - dfdr

A Python command-line tool for managing local data registries, inspired by DVC and Git but focused specifically on data registry functionality.

## Overview

dfdr allows you to:

- Declare local data sources (folders containing CSV, JSON, YAML, TXT files)
- Add specific files from data registries to your working copy
- Mirror data locally for efficient access
- Track file changes with checksums
- Keep your working copy synchronized with local registries

## Installation

### Prerequisites

- Python 3.8 or higher
- pip (Python package installer)

### Install from PyPI

```bash
pip install dfdr
```

### Install from source

```bash
git clone git@defder.fr:dfdr.git
cd dfdr
pip install -e .
```

## Quick Start

1. Initialize a dfdr repository:

```bash
dfdr init
```

2. Add a local data registry:

```bash
dfdr registry add myregistry /path/to/local/data/folder
```

3. Add files from the registry to your working copy:

```bash
dfdr add myregistry:datasets/sales.csv
dfdr add myregistry:models/config.json
```

4. Add a file to a specific subdirectory:

```bash
dfdr add myregistry:datasets/customers.csv -d ./data/customers
```

5. Update your local cache:

```bash
dfdr fetch
```

6. Check the status of your files:

```bash
dfdr status
```

7. Update your working copy:

```bash
dfdr pull
```

8. Push changes back to the registry:

```bash
dfdr push datasets/sales.csv
```

9. Move a file within the working copy:

```bash
dfdr move models/config.json ./configs/model_config.json
```

## Detailed Usage

### Managing Local Registries

```bash
# Add a local registry
dfdr registry add production /path/to/production/data

# List all registries
dfdr registry list

# Remove a registry
dfdr registry remove staging
```

### Working with Files

```bash
# Add a specific file
dfdr add production:datasets/customer_data.csv

# Add a file to a specific subdirectory
dfdr add production:datasets/sales.csv -d ./data/sales

# Add an entire directory
dfdr add production:models

# Fetch all data to local cache
dfdr fetch

# Update working copy (all files)
dfdr pull

# Update a specific file
dfdr pull datasets/customer_data.csv

# Check status of all tracked files
dfdr status

# Push changes back to the registry
dfdr push datasets/customer_data.csv

# Move a file within the working copy
dfdr move models/config.json ./configs/model_config.json
```

## Command Reference

### `dfdr init`

Initialize a new dfdr repository in the current directory.

### `dfdr registry`

Manage local data registries.

- `add <name> <path>`: Add a local data registry
- `list`: List all configured registries
- `remove <name>`: Remove a local data registry

### `dfdr add <remote_name>:<file_path> [-d <destination>]`

Add a specific file or directory from a registry to your working copy. Optionally specify a destination subdirectory.

### `dfdr fetch`

Mirror all data to local `.dfdr` storage from all remotes.

### `dfdr pull [file_path]`

Update working copy from cache (all files or specific file).

### `dfdr status`

Show sync status of files, including origin information.

### `dfdr push <file_path>`

Push changes in a file back to its origin data registry.

### `dfdr move <current_path> <new_path>`

Move a tracked file to a new destination within the working copy.

## Data registry structure

Local data registries should follow this structure:

- files are stored in local directories
- the tool will automatically discover files in the directory

For example:

```
/path/to/registry/
├── datasets/
│   ├── sales.csv
│   └── customers.json
└── models/
    └── config.yaml
```

## Architecture

dfdr is designed with a modular architecture:

- `cli.py`: Command-line interface using Click and Rich for display
- `config.py`: Configuration management (local registries, paths)
- `storage.py`: Local storage management and synchronization
- `fetcher.py`: Local file system operations
- `checksum.py`: MD5 checksum calculation and verification
- `exceptions.py`: Custom exception handling

## Local Storage

dfdr creates a `.dfdr` directory in your project containing:

- `config.json`: Local registry configuration
- `storage/`: Local mirror of registry data
- `*.dfdr` files: YAML files containing MD5 checksums and origin information for each data file (e.g., `sales.csv.dfdr`)

The `.dfdr` files now include additional information:
- MD5 checksum of the file
- Registry name (origin)
- Original path in the registry

This enhanced structure allows for better tracking and management of files across different registries.

### Development Setup

1. Clone the repository
2. Create a virtual environment: `python -m venv venv`
3. Activate the virtual environment:
   - On Windows: `venv\Scripts\activate`
   - On macOS and Linux: `source venv/bin/activate`
4. Install development dependencies: `pip install -r requirements.txt`
5. Install the package in editable mode: `pip install -e .`

## License

This project is licensed under the MIT License - see the [LICENSE](licence.txt) file for details.

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)