dfdr/readme.rst

A Python command-line tool for managing remote data registries, inspired by DVC and Git but focused specifically on data registry functionality.

Overview
--------

dfdr allows you to:

- Declare remote data sources (web servers serving CSV, JSON, YAML, TXT files over HTTP)
- Add specific files from data registries to your working copy
- Mirror data locally for efficient access
- Track file changes with checksums
- Keep your working copy synchronized with remote registries
DefDer (Data Registry Definition) - dfdr

A Python command-line tool for managing local data registries, inspired by DVC and Git but focused specifically on data registry functionality.

Overview
--------

dfdr allows you to:

- Declare local data sources (folders containing CSV, JSON, YAML, TXT files)
- Add specific files from data registries to your working copy
- Mirror data locally for efficient access
- Track file changes with checksums
- Keep your working copy synchronized with local registries
========================================

A Python command-line tool for managing remote data registries, inspired by DVC and Git but focused specifically on data registry functionality.

Overview
--------

dfdr allows you to:

- Declare remote data sources (web servers serving CSV, JSON, YAML, TXT files over HTTP)
- Add specific files from data registries to your working copy
- Mirror data locally for efficient access
- Track file changes with checksums
- Keep your working copy synchronized with remote registries

Installation
------------

.. code-block:: bash

    pip install dfdr

Or install from source:

.. code-block:: bash

    git clone https://defder.fr/dfdr.git
    cd dfdr
    pip install -e .

Quick Start
-----------

First, run 

.. code-block:: bash

    dfdr init


1. Add a local data registry:

.. code-block:: bash

    dfdr registry add myregistry /path/to/local/data/folder

2. Add files from the registry to your working copy:

.. code-block:: bash

    dfdr add myregistry:datasets/sales.csv
    dfdr add myregistry:models/config.json

3. Update your local cache:

.. code-block:: bash

    dfdr fetch

4. Check the status of your files:

.. code-block:: bash

    dfdr status

5. Update your working copy:

.. code-block:: bash

    dfdr pull

Commands
--------

Registry Management
~~~~~~~~~~~~~~~~~~~

``dfdr registry add <name> <path>``
    Add a local data registry

``dfdr registry list``
    List all configured registries

``dfdr registry remove <name>``
    Remove a local data registry

Data Management
~~~~~~~~~~~~~~~

``dfdr add <remote_name>:<file_path>``
    Add a specific file from a registry to your working copy

``dfdr fetch``
    Mirror all data to local ``.dfdr`` storage from all remotes

``dfdr pull [file_path]``
    Update working copy from cache (all files or specific file)

``dfdr status``
    Show sync status of files

Data Registry Structure
-----------------------

Local data registries should follow this structure:

- Files are stored in local directories
- No ``index.json`` file is required; the tool will automatically discover files in the directory

For example:

.. code-block:: text

    /path/to/registry/
    ├── datasets/
    │   ├── sales.csv
    │   └── customers.json
    └── models/
        └── config.yaml

Local Storage
-------------

dfdr creates a ``.dfdr`` directory in your project containing:

- ``config.json`` - Local registry configuration
- ``storage/`` - Local mirror of registry data
- ``*.dfdr`` files - MD5 checksums for each data file (e.g., ``sales.csv.dfdr``)

License
-------

MIT License

Contributing
------------

Contributions are welcome! Please feel free to submit a Pull Request.