Importer bricht ab

Ich habe einen großen Fehler gemacht und wollte auf die aktuelle Version Updaten. Da dabei ja auch Postgres geupdatet werden muss, habe ich alle meine Daten gesichert, Postgres aktualisiert (so wie hier beschrieben), habe dann mit „docker-compose run --rm webserver createsuperuser“ einen neuen Superuser angelegt und wollte dann meine Daten wieder importieren.

Leider scheitert es daran mit folgender FM:

oliver@Zion:/volume1/docker/paperless-ngx/config$ sudo docker-compose exec webserver document_importer ../export
System check identified some issues:

WARNINGS:
?: OCR output mode "skip_noarchive" is deprecated and will be removed in a future version. Please use PAPERLESS_OCR_SKIP_ARCHIVE_FILE instead.
Found existing user(s), this might indicate a non-empty installation
Found existing documents(s), this might indicate a non-empty installation
Checking the manifest
Database import failed
No version information present
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
psycopg.errors.NotNullViolation: null value in column "pdf_layout" of relation "paperless_mail_mailrule" violates not-null constraint
DETAIL:  Failing row contains (3, Shell-Tankbelege, INBOX, null, Ihr Shell SmartPay Tankbeleg, null, 2000, 3, null, 1, 4, 3, 5, 5, 2, 1, *.pdf, 1, null, null, t, null, t, null).

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/paperless/src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 413, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/paperless/src/documents/management/commands/document_importer.py", line 280, in handle
    self.load_data_to_database()
  File "/usr/src/paperless/src/documents/management/commands/document_importer.py", line 223, in load_data_to_database
    raise e
  File "/usr/src/paperless/src/documents/management/commands/document_importer.py", line 204, in load_data_to_database
    call_command("loaddata", manifest_path)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 459, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/core/management/commands/loaddata.py", line 103, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/commands/loaddata.py", line 164, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/commands/loaddata.py", line 254, in load_label
    if self.save_obj(obj):
       ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/core/management/commands/loaddata.py", line 210, in save_obj
    obj.save(using=self.using)
  File "/usr/local/lib/python3.12/site-packages/django/core/serializers/base.py", line 265, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 998, in save_base
    updated = self._save_table(
              ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 1161, in _save_table
    results = self._do_insert(
              ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/db/models/base.py", line 1202, in _do_insert
    return manager._insert(
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/db/models/manager.py", line 87, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/db/models/query.py", line 1847, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1836, in execute_sql
    cursor.execute(sql, params)
  File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 79, in execute
    return self._execute_with_wrappers(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 92, in _execute_with_wrappers
    return executor(sql, params, many, context)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 100, in _execute
    with self.db.wrap_database_errors:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/db/utils.py", line 91, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "/usr/local/lib/python3.12/site-packages/django/db/backends/utils.py", line 105, in _execute
    return self.cursor.execute(sql, params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/psycopg/cursor.py", line 97, in execute
    raise ex.with_traceback(None)
django.db.utils.IntegrityError: Problem installing fixture '/usr/src/paperless/export/manifest.json': Could not load paperless_mail.MailRule(pk=3): null value in column "pdf_layout" of relation "paperless_mail_mailrule" violates not-null constraint
DETAIL:  Failing row contains (3, Shell-Tankbelege, INBOX, null, Ihr Shell SmartPay Tankbeleg, null, 2000, 3, null, 1, 4, 3, 5, 5, 2, 1, *.pdf, 1, null, null, t, null, t, null).
ERROR: 1
oliver@Zion:/volume1/docker/paperless-ngx/config$ 

Kann mir jemand helfen? Ich hab das „kleine“ Paket, also die Config von hier und alles so gemacht wie beschrieben.

Danke.

O.

Sorry, vergessen, paperless läuft im Docker auf meiner Synology!

Der Importer funktioniert nur, wenn Du den obigen Schritt nach der Installation eben NICHT machst.

Oh! Wie kann ich das fixen?

Die Installation wiederholen (alles vorher löschen, nicht nur die Ordner leeren) und eben keinen User anlegen. Der Importer legt Dir denselben User an, welchen Du vorher schon hattest. Aber eben nur, wenn es noch gar keinen anderen User gibt.

Also den Inhalt von media und data? Oder was noch? Anschließend “sudo docker-compose pull” und dann importieren?

De beiden Ordner wirklich löschen und neu erstellen, danach den pull und den Importer ausführen.

1 „Gefällt mir“

Leider nein:

oliver@Zion:/volume1/docker/paperless-ngx/config$ sudo docker-compose pull
Pulling broker    ... done
Pulling db        ... done
Pulling webserver ... done
oliver@Zion:/volume1/docker/paperless-ngx/config$ sudo docker-compose exec webserver document_importer ../export
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/paperless/data/log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/paperless/src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 382, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.12/site-packages/django/conf/__init__.py", line 81, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.12/site-packages/django/conf/__init__.py", line 68, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/conf/__init__.py", line 166, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/src/paperless/src/paperless/settings.py", line 749, in <module>
    LOGGING_DIR.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.12/pathlib.py", line 1316, in mkdir
    self.mkdir(mode, parents=False, exist_ok=exist_ok)
  File "/usr/local/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/paperless/data/log'
ERROR: 1
oliver@Zion:/volume1/docker/paperless-ngx/config$ 

Nach dem Pull ein paar Minuten warten bis du Paperless über den Browser erreichst.

Danach erst den Importer ausführen.

Leider nicht, ich habe media und data (ohne Unterverzeichnisse) angelegt, gewartet, Paperless im Browser aufgerufen, ich kann mich anmelden, trotzdem weiterhin folgendes:

oliver@Zion:/volume1/docker/paperless-ngx/config$ sudo docker-compose exec webserver document_importer ../export
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/paperless/data/log'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/paperless/src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 382, in execute
    settings.INSTALLED_APPS
  File "/usr/local/lib/python3.12/site-packages/django/conf/__init__.py", line 81, in __getattr__
    self._setup(name)
  File "/usr/local/lib/python3.12/site-packages/django/conf/__init__.py", line 68, in _setup
    self._wrapped = Settings(settings_module)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/conf/__init__.py", line 166, in __init__
    mod = importlib.import_module(self.SETTINGS_MODULE)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 999, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/usr/src/paperless/src/paperless/settings.py", line 749, in <module>
    LOGGING_DIR.mkdir(parents=True, exist_ok=True)
  File "/usr/local/lib/python3.12/pathlib.py", line 1316, in mkdir
    self.mkdir(mode, parents=False, exist_ok=exist_ok)
  File "/usr/local/lib/python3.12/pathlib.py", line 1311, in mkdir
    os.mkdir(self, mode)
FileNotFoundError: [Errno 2] No such file or directory: '/usr/src/paperless/data/log'
ERROR: 1
oliver@Zion:/volume1/docker/paperless-ngx/config$ 

Laufen wirklich alle Container die Paperless benötigt?
Kann man ja schön im container Manager kontrollieren.

Synology Firewall deaktiviert?

Deine dockerCompose wäre auch interessant, Passwörter usw. unkenntlich machen oder löschen

[quote]Laufen wirklich alle Container die Paperless benötigt?
Kann man ja schön im container Manager kontrollieren. [/quote]
Ja laufen alle.

Ja.

Wie mache ich das, bzw. wie komm ich da ran?

Irgendwo in deinem alten Paperless Verzeichnis gibt es die datei docker-compose.yml
Mit der Datei wird Paperless installiert.

Wurde der Export eigentlich ohne Fehler ausgeführt?

# docker-compose file for running paperless from the Docker Hub.
# This file contains everything paperless needs to run.
# Paperless supports amd64, arm and arm64 hardware.
#
# All compose files of paperless configure paperless in the following way:
#
# - Paperless is (re)started on system boot, if it was running before shutdown.
# - Docker volumes for storing data are managed by Docker.
# - Folders for importing and exporting files are created in the same directory
#   as this file and mounted to the correct folders inside the container.
# - Paperless listens on port 8000.
#
# In addition to that, this docker-compose file adds the following optional
# configurations:
#
# - Instead of SQLite (default), PostgreSQL is used as the database server.
#
# To install and update paperless with this file, do the following:
#
# - Copy this file as 'docker-compose.yml' and the files 'docker-compose.env'
#   and '.env' into a folder.
# - Run 'docker-compose pull'.
# - Run 'docker-compose run --rm webserver createsuperuser' to create a user.
# - Run 'docker-compose up -d'.
#
# For more extensive installation and update instructions, refer to the
# documentation.

version: "3.4"
services:
  broker:
    image: docker.io/library/redis:6.0
    restart: unless-stopped
    volumes:
      - redisdata:/data

  db:
    image: docker.io/library/postgres:17
    restart: unless-stopped
    volumes:
      - pgdata:/var/lib/postgresql/data
    environment:
      POSTGRES_DB:
      POSTGRES_USER:
      POSTGRES_PASSWORD:

  webserver:
    image: ghcr.io/paperless-ngx/paperless-ngx:latest #2.14.7
    restart: unless-stopped
    depends_on:
      - db
      - broker
    ports:
      - 8000:8000
    healthcheck:
      test: ["CMD", "curl", "-fs", "-S", "--max-time", "2", "http://localhost:8000"]
      interval: 30s
      timeout: 10s
      retries: 5
    volumes:
      - ../data:/usr/src/paperless/data
      - ../media:/usr/src/paperless/media
      - ../export:/usr/src/paperless/export
      - /volume1/scaninput:/usr/src/paperless/consume
    env_file: docker-compose.env
    environment:
      PAPERLESS_REDIS: redis://broker:6379
      PAPERLESS_DBHOST: db


volumes:
  data:
  media:
  pgdata:
  redisdata:

Ja, der Export war erfolgreich!

Ich würde alle Volume Pfade in der docker-compose auf reele Pfade umstellen, auch vom Broker und der Datenbank.

Beispiel, musst du halt bei dir anpassen auf volume1:

image

Das würde ich auch empfehlen, aber das hat trotzdem nichts mit dem Fehler zu tun.
Auch wenn es nach „trial & error“ klingt, würde ich Paperless einmal neu starten.

Abgesehen davon irritiert mich, dass du oben schreibst „…du kannst dich anmelden..“. Anmelden vor oder nach dem Import? Vor dem Import hast du keinen User zum Anmelden.

Yep, wohl nicht.
Die yaml schaut auch recht unvollständig aus, Container Name ist nicht vorhanden oder er wurde gelöscht.

Hab ich auch grade gesehen, und wieso ist Version 2.14 installiert wenn in der yaml :latest steht, da müsste Version 2.18 installiert werden.

Der Import muss immer schon in eine leere Installation erfolgen ohne vorheriges anlegen des SuperUser.
Der SU wird ohnehin bei der ersten Anmeldung angelegt und mit importieren in die leere Installation hast du 1:1 wieder alles wie davor.

Korrekt, dann würde aber auch im Log stehen bereits ein User vorhanden ist und der Import deswegen abgebrochen wird, das hab ich bisher noch in keinem Screenshot gesehen.

So, ich habe noch mal alles neu gestartet, Inc. der Synology, Paperless angehalten, einen pull gestartet, Paperless gestartet. Natürlich media und data gelöscht und neu angelegt!

Anschließend habe ich gewartet bis ich im Browser das Anmeldefenster sehe (anmelden geht ja nicht), danach habe ich versucht zu importieren.

Mit folgendem Ergebnis:

oliver@Zion:/volume1/docker/paperless-ngx/config$ sudo docker-compose exec webserver document_importer ../export
Found existing user(s), this might indicate a non-empty installation
Found existing documents(s), this might indicate a non-empty installation
Version mismatch: Currently 2.18.1, importing 2.14.7. Continuing, but import may fail.
Checking the manifest
Database import failed
Version mismatch: Currently 2.18.1, importing 2.14.7
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/django/core/serializers/python.py", line 195, in _handle_object
    data[field.name] = field.to_python(field_value)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/db/models/fields/__init__.py", line 1503, in to_python
    raise exceptions.ValidationError(
django.core.exceptions.ValidationError: ['“2023-03-21T23:00:00Z” value has an invalid date format. It must be in YYYY-MM-DD format.']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/paperless/src/manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 442, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 436, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 416, in run_from_argv
    self.execute(*args, **cmd_options)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/paperless/src/documents/management/commands/document_importer.py", line 246, in handle
    self._run_import()
  File "/usr/src/paperless/src/documents/management/commands/document_importer.py", line 288, in _run_import
    self.load_data_to_database()
  File "/usr/src/paperless/src/documents/management/commands/document_importer.py", line 221, in load_data_to_database
    raise e
  File "/usr/src/paperless/src/documents/management/commands/document_importer.py", line 207, in load_data_to_database
    call_command("loaddata", manifest_path)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/__init__.py", line 194, in call_command
    return command.execute(*args, **defaults)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/core/management/base.py", line 460, in execute
    output = self.handle(*args, **options)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/core/management/commands/loaddata.py", line 103, in handle
    self.loaddata(fixture_labels)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/commands/loaddata.py", line 164, in loaddata
    self.load_label(fixture_label)
  File "/usr/local/lib/python3.12/site-packages/django/core/management/commands/loaddata.py", line 252, in load_label
    for obj in objects:
               ^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/django/core/serializers/python.py", line 123, in __iter__
    yield from self._handle_object(obj)
  File "/usr/local/lib/python3.12/site-packages/django/core/serializers/json.py", line 78, in _handle_object
    yield from super()._handle_object(obj)
  File "/usr/local/lib/python3.12/site-packages/django/core/serializers/python.py", line 197, in _handle_object
    raise base.DeserializationError.WithData(
django.core.serializers.base.DeserializationError: Problem installing fixture '/usr/src/paperless/export/manifest.json': ['“2023-03-21T23:00:00Z” value has an invalid date format. It must be in YYYY-MM-DD format.']: (documents.document:pk=72) field_value was '2023-03-21T23:00:00Z'
ERROR: 1
oliver@Zion:/volume1/docker/paperless-ngx/config$ 

Ich weiß weder wo die existieren User her kommen sollen, noch die Dokumente. Die Verzeichnisse wurden gelöscht und wieder angelegt, sie sind leer!