dfdr/DEVELOPMENT.md

# dfdr Development Guide

## Project Structure

```
dfdr/
├── .gitignore              # Git ignore patterns
├── Makefile               # Development automation
├── README.rst             # Main documentation
├── requirements.txt       # Python dependencies
├── setup.py              # Package configuration
├── DEVELOPMENT.md         # This file
├── dfdr/                 # Main package
│   ├── __init__.py       # Package initialization
│   ├── cli.py            # Command-line interface
│   ├── config.py         # Configuration management
│   ├── checksum.py       # MD5 checksum utilities
│   ├── exceptions.py     # Custom exceptions
│   ├── fetcher.py        # HTTP client with httpx
│   └── storage.py        # Storage and working copy management
├── tests/                # Test suite
│   ├── __init__.py
│   └── test_basic.py     # Basic functionality tests
└── examples/             # Usage examples
    └── basic_usage.py    # Programmatic usage example
```

## Installation

1. Create and activate a virtual environment:
```bash
python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate
```

2. Install in development mode:
```bash
make install-dev
# or manually:
pip install -e .
pip install pytest pytest-cov black flake8 mypy
```

## Development Workflow

### Running Tests
```bash
make test           # Run basic tests
make test-cov       # Run tests with coverage report
```

### Code Quality
```bash
make format         # Format code with black
make lint          # Run linting with flake8 and mypy
```

### Building and Distribution
```bash
make build         # Build distribution packages
make clean         # Clean build artifacts
```

## Command Reference

### Phase 1 Commands (Implemented)
- `dfdr init` - Initialize a dfdr repository
- `dfdr remote add <name> <url>` - Add remote data registry
- `dfdr remote list` - List configured remotes
- `dfdr remote remove <name>` - Remove a remote
- `dfdr add <remote>:<file>` - Add file from remote to working copy
- `dfdr fetch` - Fetch all files from remotes to storage
- `dfdr pull [file]` - Update working copy from storage
- `dfdr status` - Show file sync status

### Usage Examples

```bash
# Initialize repository
dfdr init

# Add a remote data registry
dfdr remote add mydata https://data.example.com/

# Add specific files
dfdr add mydata:datasets/sales.csv
dfdr add mydata:models/config.json

# Fetch all available files to local storage
dfdr fetch

# Check status
dfdr status

# Update working copy
dfdr pull
```

## Architecture

### Core Components

1. **Config** (`config.py`): Manages remote registries and local configuration
2. **Fetcher** (`fetcher.py`): HTTP client for downloading files and indices
3. **Storage** (`storage.py`): Manages local storage and working copy operations
4. **Checksum** (`checksum.py`): MD5 checksum calculation and verification
5. **CLI** (`cli.py`): Command-line interface using Click and Rich

### Data Flow

1. **Remote Registry**: HTTP server with `index.json` files listing available files
2. **Local Storage**: `.dfdr/storage/` mirrors remote files
3. **Working Copy**: Project files with corresponding `.dfdr` checksum files
4. **Configuration**: `.dfdr/config.json` stores remote registry URLs

### File Structure

```
project/
├── .dfdr/
│   ├── config.json       # Remote configurations
│   └── storage/          # Local mirror of remote files
│       └── remote_name/
│           └── file.csv
├── data_file.csv         # Working copy file
└── data_file.csv.dfdr    # MD5 checksum
```

## Protocol Specification

### Remote Data Registry

Remote registries must serve files over HTTP with the following structure:

1. **Index Files**: Each directory contains `index.json`:
```json
{
  "files": ["file1.csv", "file2.json", "subfolder/file3.yaml"]
}
```

2. **File Access**: Files are accessible via direct HTTP GET requests
3. **Content Types**: Supports CSV, JSON, YAML, TXT files

### Local Storage

1. **Checksums**: Each data file has a corresponding `.dfdr` file containing MD5 hash
2. **Storage Mirror**: `.dfdr/storage/` contains exact copies of remote files
3. **Configuration**: JSON configuration file tracks remote registries

## Testing

Run the test suite:
```bash
python -m pytest tests/ -v
```

Test coverage:
```bash
python -m pytest tests/ --cov=dfdr --cov-report=html
```

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make changes with tests
4. Run `make format` and `make lint`
5. Submit a pull request

## Future Enhancements

### Phase 2 Features
- Directory synchronization
- Nested folder support with recursive index.json
- File metadata tracking (size, modification time)
- Conflict resolution strategies
- Incremental updates based on ETags/Last-Modified headers

### Advanced Features
- Authentication support for private registries
- Compression support
- Parallel downloads
- Registry mirroring
- Plugin system for custom protocols