Metadata-Version: 2.3
Name: elasticsearch-pii-redacter
Version: 1.11.0
Summary: Redacting field data from your Elasticsearch indices and Searchable Snapshots
Project-URL: Homepage, https://github.com/elastic/elastic-pii-redacter
Project-URL: Bug Tracker, https://github.com/elastic/elastic-pii-redacter/issues
Author-email: Elastic <info@elastic.co>
License: Apache-2.0
Keywords: elasticsearch,index,pii,redact
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: System Administrators
Classifier: License :: OSI Approved :: Apache Software License
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3.8
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Requires-Python: >=3.8
Requires-Dist: es-client==8.13.5
Provides-Extra: doc
Requires-Dist: sphinx; extra == 'doc'
Requires-Dist: sphinx-rtd-theme; extra == 'doc'
Provides-Extra: test
Requires-Dist: mock; extra == 'test'
Requires-Dist: pytest-cov; extra == 'test'
Requires-Dist: pytest>=7.2.1; extra == 'test'
Requires-Dist: requests; extra == 'test'
Description-Content-Type: text/x-rst

.. _readme:

elastic-pii-redacter
====================

Did you find PII (Personally Identifiable Information) in your Elasticsearch
indices that doesn't belong there? This is the tool for you!

The elastic-pii-redacter can help you redact information from even Searchable
Snapshot mounted indices. It works with deeply nested fields, too!


Client Configuration
--------------------

The tool connects using the ``es_client`` Python module.

You can use command-line options, or a YAML configuration file to configure the client connection.
If using a configuration file is desired, the configuration file structure requires
``elasticsearch`` at the root level as follows::

    ---
    elasticsearch:
      client:
        hosts: https://10.11.12.13:9200
        cloud_id:
        request_timeout: 60
        verify_certs:
        ca_certs:
        client_cert:
        client_key:
      other_settings:
        username:
        password:
        api_key:
          id:
          api_key:
          token:

    logging:
      loglevel: INFO
      logfile: /path/to/file.log
      logformat: default
      blacklist: []


`REDACTIONS_FILE` Configuration
-------------------------------

NOTE: If, under ``forcemerge``, ``only_expunge_deletes`` is ``True``, any configured value
for ``max_num_segments`` will be ignored, and only documents marked for delete will be
cleared. It is important to note this distinction as the default behavior is to merge
to 1 segment per shard.

::

  ---
  redactions:
    - job_name_20240506_redact_hot:
        pattern: hot-*
        query: {'match': {'message': 'message1'}}
        fields: ['message']
        message: REDACTED
        expected_docs: 1
        restore_settings: {'index.routing.allocation.include._tier_preference': 'data_warm,data_hot,data_content'}
    - job_name_20240506_redact_cold:
        pattern: restored-cold-*
        query: {'match': {'nested.key': 'nested19'}}
        fields: ['nested.key']
        message: REDACTED
        expected_docs: 1
        restore_settings: {'index.routing.allocation.include._tier_preference': 'data_warm,data_hot,data_content'}
        forcemerge:
          max_num_segments: 1
    - job_name_20240506_redact_frozen:
        pattern: partial-frozen-*
        query: {'range': {'number': {'gte': 8, 'lte': 11}}}
        fields: ['deep.l1.l2.l3']
        message: REDACTED
        expected_docs: 4
        forcemerge:
          only_expunge_deletes: True
