165 lines
3.6 KiB
ReStructuredText
165 lines
3.6 KiB
ReStructuredText
|
|
A Python command-line tool for managing remote data registries, inspired by DVC and Git but focused specifically on data registry functionality.
|
|
|
|
Overview
|
|
--------
|
|
|
|
dfdr allows you to:
|
|
|
|
- Declare remote data sources (web servers serving CSV, JSON, YAML, TXT files over HTTP)
|
|
- Add specific files from data registries to your working copy
|
|
- Mirror data locally for efficient access
|
|
- Track file changes with checksums
|
|
- Keep your working copy synchronized with remote registries
|
|
DefDer (Data Registry Definition) - dfdr
|
|
|
|
A Python command-line tool for managing local data registries, inspired by DVC and Git but focused specifically on data registry functionality.
|
|
|
|
Overview
|
|
--------
|
|
|
|
dfdr allows you to:
|
|
|
|
- Declare local data sources (folders containing CSV, JSON, YAML, TXT files)
|
|
- Add specific files from data registries to your working copy
|
|
- Mirror data locally for efficient access
|
|
- Track file changes with checksums
|
|
- Keep your working copy synchronized with local registries
|
|
========================================
|
|
|
|
A Python command-line tool for managing remote data registries, inspired by DVC and Git but focused specifically on data registry functionality.
|
|
|
|
Overview
|
|
--------
|
|
|
|
dfdr allows you to:
|
|
|
|
- Declare remote data sources (web servers serving CSV, JSON, YAML, TXT files over HTTP)
|
|
- Add specific files from data registries to your working copy
|
|
- Mirror data locally for efficient access
|
|
- Track file changes with checksums
|
|
- Keep your working copy synchronized with remote registries
|
|
|
|
Installation
|
|
------------
|
|
|
|
.. code-block:: bash
|
|
|
|
pip install dfdr
|
|
|
|
Or install from source:
|
|
|
|
.. code-block:: bash
|
|
|
|
git clone https://defder.fr/dfdr.git
|
|
cd dfdr
|
|
pip install -e .
|
|
|
|
Quick Start
|
|
-----------
|
|
|
|
First, run
|
|
|
|
.. code-block:: bash
|
|
|
|
dfdr init
|
|
|
|
|
|
1. Add a local data registry:
|
|
|
|
.. code-block:: bash
|
|
|
|
dfdr registry add myregistry /path/to/local/data/folder
|
|
|
|
2. Add files from the registry to your working copy:
|
|
|
|
.. code-block:: bash
|
|
|
|
dfdr add myregistry:datasets/sales.csv
|
|
dfdr add myregistry:models/config.json
|
|
|
|
3. Update your local cache:
|
|
|
|
.. code-block:: bash
|
|
|
|
dfdr fetch
|
|
|
|
4. Check the status of your files:
|
|
|
|
.. code-block:: bash
|
|
|
|
dfdr status
|
|
|
|
5. Update your working copy:
|
|
|
|
.. code-block:: bash
|
|
|
|
dfdr pull
|
|
|
|
Commands
|
|
--------
|
|
|
|
Registry Management
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
``dfdr registry add <name> <path>``
|
|
Add a local data registry
|
|
|
|
``dfdr registry list``
|
|
List all configured registries
|
|
|
|
``dfdr registry remove <name>``
|
|
Remove a local data registry
|
|
|
|
Data Management
|
|
~~~~~~~~~~~~~~~
|
|
|
|
``dfdr add <remote_name>:<file_path>``
|
|
Add a specific file from a registry to your working copy
|
|
|
|
``dfdr fetch``
|
|
Mirror all data to local ``.dfdr`` storage from all remotes
|
|
|
|
``dfdr pull [file_path]``
|
|
Update working copy from cache (all files or specific file)
|
|
|
|
``dfdr status``
|
|
Show sync status of files
|
|
|
|
Data Registry Structure
|
|
-----------------------
|
|
|
|
Local data registries should follow this structure:
|
|
|
|
- Files are stored in local directories
|
|
- No ``index.json`` file is required; the tool will automatically discover files in the directory
|
|
|
|
For example:
|
|
|
|
.. code-block:: text
|
|
|
|
/path/to/registry/
|
|
├── datasets/
|
|
│ ├── sales.csv
|
|
│ └── customers.json
|
|
└── models/
|
|
└── config.yaml
|
|
|
|
Local Storage
|
|
-------------
|
|
|
|
dfdr creates a ``.dfdr`` directory in your project containing:
|
|
|
|
- ``config.json`` - Local registry configuration
|
|
- ``storage/`` - Local mirror of registry data
|
|
- ``*.dfdr`` files - MD5 checksums for each data file (e.g., ``sales.csv.dfdr``)
|
|
|
|
License
|
|
-------
|
|
|
|
MIT License
|
|
|
|
Contributing
|
|
------------
|
|
|
|
Contributions are welcome! Please feel free to submit a Pull Request.
|