rougail/docs/structured_data/data_integrity.rst

Data integrity
===============

Data integrity refers to the fact that data must be reliable and accurate throughout its lifecycle.

This means that the value must be:

- of high quality
- appropriate to the overall context

Data quality
------------

The values of the variables must be individually of good quality.

Typing
~~~~~~

The typed variable concept refers to associating a data type with a variable, which defines what kind of values the variable can hold and what operations can be performed on it.

Typed variables are a fundamental feature in Rougail. Type validation is the first and most important quality check.

Strongly-typed
''''''''''''''

Rougail is a strongly-typed DSL.

This means that loading user data requires attention to the variable types.

The :term:`integrator` define the variable type and the :term:`operator` has to conform with it.

.. For untyped user data (such as environment variables), the value type will be adapted during preprocessing.

Dynamicaly-typed
'''''''''''''''''

During the structured data definition, the type is dynamic.

That is to say, the :term:`integrator` can change the type of the variable at any time.

However, the :term:`operator` who adapts the value does not have the possibility of redefining the type of the variable.

.. seealso:: :ref:`the variable mutability <variable_mutability>`


Type inference
''''''''''''''

Type inference is the process where Rougail automatically figures out the types without explicit annotations.
In fact, the variable type comes from the type of its default value.

The type inference is in particular use with the :term:`short-hand notation`.

Variables typing
~~~~~~~~~~~~~~~~

Standard types
''''''''''''''

Rougail accepts the following standard types:

- `string` (default type of a variable)
- `integer`
- `float`
- `boolean`

.. FIXME: est-ce que integer et float -> number type ????

.. seealso:: :doc:`the variable documentation <variable>`

Specialized types
''''''''''''''''''

But we will also find a whole series of specialized types:

- IP
- domainname
- port
- MAC         
- secret
- ...

.. seealso:: :doc:`the variable documentation <variable>`

A variable with a list of possible values
'''''''''''''''''''''''''''''''''''''''''

Sometimes a variable can only hold a finite list of possible values. We call it a variable with a list of possible values.

.. seealso:: The tutorial with a real world sample about :doc:`a variable with a list of possible values <../tutorial/choice>`

Type parameters
'''''''''''''''

For certain types, there are a number of parameters that can be used to further type the variables.

.. seealso:: :doc:`the variable documentation <variable>`

.. _nullable_variable:

Custom type
'''''''''''

It is not possible to create every possible type of variable.

Therefore, it is necessary to provide a simple way to create these custom types.

There are two ways to create a type:

- creating a :term:`Tiramisu` type
- creating a type from an existing variable, have a look at the tutorial with a real world sample :doc:`custom type <../tutorial/customtype>`.

Nullable variable
'''''''''''''''''

The `null` type (or `None` in Python) does not exist in Rougail.

All variables, regardless of their type, can be nullable (see the remarks on the list type below).

Even if all types can accept this value, by default, they do not.

In reality, the variable not nullable with the value to `null` is not accessible in `read-only` mode (which is the case during the output steps).
In `read/write` mode, the access is indeed granted.

.. seealso:: The tutorial with a real world sample about a :doc:`nullable variable <../tutorial/nullable>`

The `list` type
'''''''''''''''

A list is not a type either.
It is a property of a variable. There is no such thing as a container type (like a python `list` type for instance).
This means that all type can have multiple values in a list but it can only contain values of an unique type.

A multiple variable could be nullable. This means that `null` can be accepted as a value in the list (not permitted by default).
But the multiple variable could also be without value (not permitted too by default).

.. seealso:: The tutorial with a real world sample about a :doc:`multiple variable <../tutorial/multiple>`

Families typing
~~~~~~~~~~~~~~~

The `mapping` type
''''''''''''''''''

A mapping is a collection of key: value pairs.

This is a JSON example:

.. code-block:: json

    {
      "my_variable": "my_value"
    }

In Rougail the mapping type is called a :term:`family`.
It is a container designed to hold variables.

The whole :term:`configuration` tree structure is handled by the families definitions. 
It is possible to create subfamilies.

.. seealso:: The tutorial with a real world sample about :doc:`the family documentation <family>`

The dynamically built family
''''''''''''''''''''''''''''

A dynamically built family is a special family.

.. glossary::

   dynamically built family

        A dynamically built family is a fictitious family linked to a list of unique identifiers.

        This means that families will appear or disappear folling the :term:`context`.

.. seealso:: The tutorial with a real world sample about :doc:`the dynamically built family documentation <family>`

Homogeneous elements sequence
'''''''''''''''''''''''''''''

.. glossary::

   sequence
   homogeneous elements sequence

        A specific type of mapping, called a `sequence`, allows you to have a list of homogeneous objects.

        For example, if I want to be able to create an unlimited number of users associated with a secret, you can't use lists; I need a list of objects.

        In JSON we will have:

        .. code-block:: json

           [
                {
                  "user": "login1",
                  "secret": "MySecret"
                },
                {
                  "user": "login2",
                  "secret": "MySecret"
                }
           ]

Validation
~~~~~~~~~~

Type validation is the first and most important check.
But it is possible to add additional validations.

For example, one might want numbers, but only odd numbers.

.. index:: overall coherence

Overall coherence
-----------------

An isolated variable can be considered to be of quality but become inconsistent depending on the context.

.. index:: consistency

Consistency
~~~~~~~~~~~

For example, if a minimum value and then a maximum value are requested, the minimum must be lesser than the maximum.

Overall consistency is initially managed by personalized validators which will validate the value of a variable in relation to others.

Access control
~~~~~~~~~~~~~~

Access control occurs as soon as an attempt is made to access a variable.

There are two main types of access control:

- Properties
- Modes

Properties
'''''''''''

We already see the property access control.

Remember, we talked about the :ref:`hidden variable <hidden>` and :ref:`disabled variable <disabled>` variables.

These properties become fully meaningful when managing overall consistency.

Why ask for the domain name of a service if we haven't activated that service just before?

.. index:: mode

Mode
''''

By default, the mode is not configured. It is an optional feature.

Let's start by defining what we want to do with the modes.

We'll present a common example, but you'll need to define your own modes according to your needs.

Here is our classic use case of mode definition. We'll use three modes:

- `basic`: automatically sets mandatory variables without default values (in this case, the actor adapting the configuration will have to enter values) and manually sets variables that the actor defining the variables deems necessary.
- `standard`: automatically sets all other variables
- `advanced`: sets variables that the actor defining the variables decides to include. These variables are intended for a knowledgeable audience who know what to do.