Apache NiFi: Data Flow Automation & Pipeline Design Guide

Apache NiFi is a visual, real-time dataflow platform that makes it easy to automate data movement between systems without writing code. Drag, drop, and connect processors in a browser-based UI to build powerful, fault-tolerant data pipelines.

What is Apache NiFi?

Apache NiFi is an open-source data integration platform originally developed by the NSA and donated to the Apache Software Foundation in 2014. It is designed for automated, reliable data flow between heterogeneous systems: files, databases, REST APIs, message queues, cloud storage, IoT devices, and more. The browser-based UI lets users build and modify pipelines visually without touching code — making it accessible to both engineers and operations teams.

Every piece of data in NiFi is represented as a FlowFile — a content payload plus metadata attributes. Processors read, transform, route, and write FlowFiles. Data provenance is tracked for every FlowFile through every processor, providing a complete audit trail of where data came from and every transformation it underwent. NiFi includes built-in security (TLS, multi-tenant authorization), clustering for scalability, and Site-to-Site for data transfer between NiFi instances.

Docker Compose Quickstart

# docker-compose.yml — Apache NiFi single-node
version: '3.8'
services:
  nifi:
    image: apache/nifi:1.27.0
    ports:
      - "8443:8443"    # HTTPS UI (NiFi 1.14+ enforces TLS)
    environment:
      NIFI_WEB_HTTPS_PORT: '8443'
      SINGLE_USER_CREDENTIALS_USERNAME: admin
      SINGLE_USER_CREDENTIALS_PASSWORD: adminpassword123
    volumes:
      - nifi_conf:/opt/nifi/nifi-current/conf
      - nifi_data:/opt/nifi/nifi-current/data
      - nifi_logs:/opt/nifi/nifi-current/logs
      - ./flow_templates:/opt/nifi/nifi-current/flow_templates

volumes:
  nifi_conf:
  nifi_data:
  nifi_logs:

# Access UI at: https://localhost:8443/nifi
# Default credentials set via SINGLE_USER_CREDENTIALS_* env vars

Creating Processors via REST API

# NiFi REST API — create a processor via API (Python)
import requests
import json

BASE = "https://localhost:8443/nifi-api"
HEADERS = {"Content-Type": "application/json"}
SESSION = requests.Session()
SESSION.verify = False  # disable for self-signed cert in dev

# 1. Get root process group ID
root = SESSION.get(f"{BASE}/process-groups/root").json()
pg_id = root["id"]

# 2. Create a GetFile processor
processor_config = {
    "revision": {"version": 0},
    "component": {
        "type": "org.apache.nifi.processors.standard.GetFile",
        "name": "Get Orders CSV",
        "position": {"x": 400, "y": 200},
        "config": {
            "properties": {
                "Input Directory": "/data/input",
                "File Filter": "orders_.*\.csv",
                "Keep Source File": "false",
            },
            "schedulingPeriod": "10 sec",
            "schedulingStrategy": "TIMER_DRIVEN",
        },
    },
}
resp = SESSION.post(
    f"{BASE}/process-groups/{pg_id}/processors",
    json=processor_config
)
print("Created processor:", resp.json()["component"]["name"])

NiFi Expression Language

# NiFi Expression Language — FlowFile attribute manipulation
# Used in Processor properties to dynamically route and transform

# Access FlowFile attributes
${filename}                          # original filename
${uuid}                             # FlowFile UUID
${fileSize}                         # size in bytes
${entryDate}                        # ingestion timestamp

# String operations
${filename:toLower()}               # orders_2024.CSV -> orders_2024.csv
${filename:replace('.csv', '.parquet')}
${filename:substring(0, 8)}          # first 8 chars

# Conditionals — used in RouteOnAttribute processor
${amount:gt(1000)}                  # route high-value orders
${status:equals('error')}           # route error records
${region:in('us', 'eu', 'ap')}      # allow-list routing

# Date formatting for S3/HDFS partition paths
${now():format('yyyy/MM/dd')}        # 2024/01/15
${entryDate:toNumber():format('yyyy-MM-dd', 'UTC')}

# Combining attributes in PutS3Object Key property:
# output/${now():format('yyyy/MM/dd')}/${uuid}.json

NiFi vs Kafka Connect vs Apache Airflow

Common Processor Patterns

• GetFile → SplitText → PublishKafka — read CSV line-by-line and publish each row to Kafka
• ConsumeKafka → EvaluateJsonPath → RouteOnAttribute → PutDatabaseRecord — consume, extract fields, route by value, and write to database
• ListS3 → FetchS3Object → ConvertAvroToJSON → PutElasticsearchRecord — S3 to Elasticsearch with format conversion
• QueryDatabaseTable → ConvertRecord → PutParquet — CDC (change data capture) from DB to Parquet files
• Use HandleHttpRequest / HandleHttpResponse to expose NiFi as an HTTP endpoint for receiving data from external systems
• Use ExecuteScript (Python/Groovy) only when built-in processors cannot handle the logic — maintain testability