Unrecognized input header error with Docker containers – Solving the problem

This tutorial addresses a common error encountered when using Docker containers: Unrecognized input header: 50. This issue often occurs when integrating Docker with logging tools like Loki and Promtail. We will explore various community-provided solutions and workarounds.

Steps to Reproduce the Error

Start Loki 2.5.0.

Start Promtail 2.5.0 to tail Docker container logs.

Run a Docker container with a command like:

docker run --rm \
      --log-driver local --log-opt max-size=10m \
      alpine echo hello world

Common Error Scenario

• Environment: Archlinux, Docker 20.10.14, deployed using ansible.
• Promtail Config:

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
        refresh_interval: 5s
        filters:
          - name: name
            values: [loggingtesting]
    relabel_configs:
      - source_labels: ['__meta_docker_container_name']
        regex: '/(.*)'
        target_label: 'container'

• Error Message:

level=warn ts=2022-04-18T18:36:24.283252527Z caller=target.go:120 target=docker/d127e5083600a4618b773825eac65f8b260f3605ba0ed33b43cabb66a60a39ed msg="could not transfer logs" written=0 container=d127e5083600a4618b773825eac65f8b260f3605ba0ed33b43cabb66a60a39ed err="Unrecognized input header: 50"

Community Answers and Solutions

• Update Docker Version: An update to the latest Docker version may resolve the issue temporarily. However, the issue may reoccur.
• Workaround Using Static Configs:
  • Switch Promtail to read logs directly from JSON files:

static_configs:
  - targets:
      - localhost
    labels:
      __path__: /var/lib/docker/containers/*/*-json.log

Docker Driver Configuration: Ensure the Docker driver is configured correctly with JSON logs. Example configuration:

{
  "storage-driver": "overlay2",
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "5m",
    "max-file": "5"
  }
}

• Verify log validity using JSON processors like jq.
• TTY Configuration Issue: A correlation exists between the error and containers running with TTY (tty: true). Removing tty: true from Docker Compose or Docker configurations might solve the issue. However, this is not feasible for containers requiring TTY for proper functioning.
• Use of Docker Plugin: Installing the Promtail Docker plugin can sometimes fix the issue:

docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions

• Code Modification in Docker Package: A workaround involves modifying the stdcopy.go file in the Docker package to handle ASCII symbols in stdWriterFdIndex. However, this approach might lead to looping container logs.
• Prioritizing Log Files: Targeting log files directly is a safer approach, especially when permissions for reading container log files are limited. This approach is useful if docker_sd is considered experimental or unreliable.
• Pull Request for Fix: A pull request (#7829) addressing the issue was merged, which may provide a more permanent solution.
• Fallback to Alternative Logging Clients: If the issue persists, consider using alternative logging clients like Logstash for container log management.

Conclusion

The “Unrecognized input header: 50” error in Docker containers can be addressed through various methods, including updating Docker, modifying configurations, removing TTY settings, using Docker plugins, or directly targeting log files. Community contributions and ongoing development in Docker and related tools are vital in resolving such issues.