dfdr/DEVELOPMENT.md

dfdr Development Guide

Project Structure

dfdr/
├── .gitignore              # Git ignore patterns
├── Makefile               # Development automation
├── README.rst             # Main documentation
├── requirements.txt       # Python dependencies
├── setup.py              # Package configuration
├── DEVELOPMENT.md         # This file
├── dfdr/                 # Main package
│   ├── __init__.py       # Package initialization
│   ├── cli.py            # Command-line interface
│   ├── config.py         # Configuration management
│   ├── checksum.py       # MD5 checksum utilities
│   ├── exceptions.py     # Custom exceptions
│   ├── fetcher.py        # HTTP client with httpx
│   └── storage.py        # Storage and working copy management
├── tests/                # Test suite
│   ├── __init__.py
│   └── test_basic.py     # Basic functionality tests
└── examples/             # Usage examples
    └── basic_usage.py    # Programmatic usage example

Installation

1. Create and activate a virtual environment:

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

2. Install in development mode:

make install-dev
# or manually:
pip install -e .
pip install pytest pytest-cov black flake8 mypy

Development Workflow

Running Tests

make test           # Run basic tests
make test-cov       # Run tests with coverage report

Code Quality

make format         # Format code with black
make lint          # Run linting with flake8 and mypy

Building and Distribution

make build         # Build distribution packages
make clean         # Clean build artifacts

Command Reference

Phase 1 Commands (Implemented)

• dfdr init - Initialize a dfdr repository
• dfdr remote add <name> <url> - Add remote data registry
• dfdr remote list - List configured remotes
• dfdr remote remove <name> - Remove a remote
• dfdr add <remote>:<file> - Add file from remote to working copy
• dfdr fetch - Fetch all files from remotes to storage
• dfdr pull [file] - Update working copy from storage
• dfdr status - Show file sync status

Usage Examples

# Initialize repository
dfdr init

# Add a remote data registry
dfdr remote add mydata https://data.example.com/

# Add specific files
dfdr add mydata:datasets/sales.csv
dfdr add mydata:models/config.json

# Fetch all available files to local storage
dfdr fetch

# Check status
dfdr status

# Update working copy
dfdr pull

Architecture

Core Components

1. Config (config.py): Manages remote registries and local configuration
2. Fetcher (fetcher.py): HTTP client for downloading files and indices
3. Storage (storage.py): Manages local storage and working copy operations
4. Checksum (checksum.py): MD5 checksum calculation and verification
5. CLI (cli.py): Command-line interface using Click and Rich

Data Flow

1. Remote Registry: HTTP server with index.json files listing available files
2. Local Storage: .dfdr/storage/ mirrors remote files
3. Working Copy: Project files with corresponding .dfdr checksum files
4. Configuration: .dfdr/config.json stores remote registry URLs

File Structure

project/
├── .dfdr/
│   ├── config.json       # Remote configurations
│   └── storage/          # Local mirror of remote files
│       └── remote_name/
│           └── file.csv
├── data_file.csv         # Working copy file
└── data_file.csv.dfdr    # MD5 checksum

Protocol Specification

Remote Data Registry

Remote registries must serve files over HTTP with the following structure:

1. Index Files: Each directory contains index.json:

{
  "files": ["file1.csv", "file2.json", "subfolder/file3.yaml"]
}

2. File Access: Files are accessible via direct HTTP GET requests
3. Content Types: Supports CSV, JSON, YAML, TXT files

Local Storage

1. Checksums: Each data file has a corresponding .dfdr file containing MD5 hash
2. Storage Mirror: .dfdr/storage/ contains exact copies of remote files
3. Configuration: JSON configuration file tracks remote registries

Testing

Run the test suite:

python -m pytest tests/ -v

Test coverage:

python -m pytest tests/ --cov=dfdr --cov-report=html

Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes with tests
4. Run make format and make lint
5. Submit a pull request

Future Enhancements

Phase 2 Features

• Directory synchronization
• Nested folder support with recursive index.json
• File metadata tracking (size, modification time)
• Conflict resolution strategies
• Incremental updates based on ETags/Last-Modified headers

Advanced Features

• Authentication support for private registries
• Compression support
• Parallel downloads
• Registry mirroring
• Plugin system for custom protocols