rougail/docs/structured_data/data_integrity.rst

Data integrity
===============

Data integrity refers to the fact that data must be reliable and accurate throughout its lifecycle.

This means that the value must be:

- of high quality
- appropriate to the overall context

Data quality
------------

The values of the variables must be individually of good quality.

Typing
~~~~~~

The typed variable concept refers to associating a data type with a variable, which defines what kind of values the variable can hold and what operations can be performed on it.

Typed variables are a fundamental feature in Rougail. Type validation is the first and most important quality check.

Strongly-typed
''''''''''''''

Rougail is a strongly-typed DSL.

This means that loading user data requires attention to the variable types.

The :term:`integrator` define the variable type and :term:`operator` has to conform with it.

.. For untyped user data (such as environment variables), the value type will be adapted during preprocessing.

Dynamicly-typed
'''''''''''''''

During the structured data definition, the type is dynamic.

That is to say, the :term:`integrator` can change the type of the variable at any time.

However, the :term:`operator` who adapts the value does not have the possibility of redefining the type of the variable.

.. seealso:: :doc:`the variable mutability <../concepts>`

Type inference
''''''''''''''

Type inference is the process where Rougail automatically figures out the types without explicit annotations.
In fact, the variable type comes from the type of its default value.

The type inference is in particular use in :term:`short-hand notation`.

Variables typing
~~~~~~~~~~~~~~~~

Standard types
''''''''''''''

Rougail accepts the following standard types:

- string (default type of a variable)
- integer
- float
- boolean

.. seealso:: :doc:`the variable documentation <variable>`

Specialized type
''''''''''''''''

But we will also find a whole series of specialized types:

- IP
- domainname
- port
- MAC         
- choice
- secret
- ...

.. seealso:: :doc:`the variable documentation <variable>`

Type parameters
'''''''''''''''

For certain types, there are a number of parameters that can be used to further type the variables.

.. seealso:: :doc:`the variable documentation <variable>`

Nullable variable
'''''''''''''''''

The `null` type (or `None` in Python) does not exist in Rougail.

All variables, regardless of their type, can be nullable (see the remarks on the list type below).

Even if all types can accept this value, by default, they do not.

In reality, the variable not nullable with the value to `null` is not accessible in `read-only` mode (which is the case during the output steps).
In `read/write` mode, the access is indeed granted.

.. seealso:: tutorial with a real world sample :doc:`nullable variable <../tutorial/nullable>`

The `list` type
'''''''''''''''

A list is not a type either.
It is a property of a variable.
This means that all type could have multiple value by a list can only contain values of an uniq type.

A multiple variable could be nullable. This means that `null` can be accepted as a value in the list (not permitted by default).
But the multiple variable could also be without value (not permitted too by default).

.. seealso:: tutorial with a real world sample :doc:`multiple variable <../tutorial/multiple>`

Families typing
~~~~~~~~~~~~~~~

The `mapping` type
''''''''''''''''''

A mapping is a collection of key: value pairs.

This is a JSON example:

.. code-block:: json

    {
      "my_variable": "my_value"
    }

In Rougail the mapping type is call :term:`family`.
It is a container designed to hold variables.

Families manage the tree structure. It is possible to create subfamilies.

.. seealso:: :doc:`the family documentation <family>`

The dynamically built family
''''''''''''''''''''''''''''

A dynamically built family is a special family.

.. glossary::

   dynamically built family

        A dynamically built family is a fictitious family linked to a list of uniq identifiers.

        This means that families will appear or disappear folling the context.

.. seealso:: :doc:`the dynamically built family documentation <family>`

Homogeneous elements sequence
'''''''''''''''''''''''''''''

.. glossary::

   sequence
   homogeneous elements sequence

        A specific type of mapping, called a `sequence`, allows you to have a list of homogeneous objects.

        For example, if I want to be able to create an unlimited number of users associated with a secret, you can't use lists; I need a list of objects.

        In JSON we will have:

        .. code-block:: json

           [
                {
                  "user": "login1",
                  "secret": "MySecret"
                },
                {
                  "user": "login2",
                  "secret": "MySecret"
                }
           ]

Validation
~~~~~~~~~~

Type validation is the first and most important check.
But it is possible to add additional validations.

For example, one might want numbers, but only odd numbers.

Overall coherence
-----------------

An isolated variable can be considered to be of quality but become inconsistent depending on the context.

Consistency
~~~~~~~~~~~

For example, if a minimum value and then a maximum value are requested, the minimum must be less than the maximum.

Overall consistency is initially managed by personality validators who will validate the value of a variable in relation to others.

Access control
~~~~~~~~~~~~~~

Access control is implemented as soon as an attempt is made to access a variable.

There are two main types of access control:

- Properties
- Modes

Properties
'''''''''''

We already see the property access control.

Remember, we talked about :doc:`the hidden and disabled variables <../concepts>`.

These properties become fully meaningful when managing overall consistency.

Why ask for the domain name of a service if we haven't activated that service just before?

Mode
''''

By default, the mode is not configured. It is an optional feature.

Let's start by defining what we want to do with the modes.

We'll present a common example, but you'll need to define your own modes according to your needs.

Here is our classic case of mode definition. We'll use three modes:

- basic: automatically sets mandatory variables without default values (in this case, the actor adapting the configuration will have to enter values) and manually sets variables that the actor defining the variables deems necessary.
- standard: automatically sets all other variables
- advanced: sets variables that the actor defining the variables decides to include. These variables are intended for a knowledgeable audience who know what to do.