DFIRVault

CSV2ELK

Why I Built This

As a DFIR professional, I constantly deal with:

  • CSV exports from SIEMs, firewalls, and malware analysis tools

  • Time wasted manually mapping fields to Elasticsearch

  • Credential fatigue from repeated Elasticsearch logins

CSV2ELK v0.1 solves this with:
✅ One-click ingestion of forensic data into ELK
✅ Automatic timestamp detection (supports 10+ formats)
✅ Secure credential storage (no hardcoded passwords)

Link here => https://github.com/dfirvault/CSV2ELK

How It Works

1. Data Flow Diagram

[CSV File] → 
  [CSV2ELK] → (Auto-Detect Fields) → 
    [Elasticsearch Cluster]  
      ↓  
[Kibana Dashboard Ready!]

2. Key Features

A. Smart CSV Parsing

  • Handles malformed CSVs (skips bad lines, preserves data)

  • Auto-converts:

    • Timestamps → @timestamp

    • IPs → source.ip (if field matches *ip*)

    • URLs → url.original

B. Secure Authentication

[User Input] → 
  [config.txt] → 
    (Encrypted in Transit) → 
      [Elasticsearch HTTPS]

C. CLI + GUI Modes

# CLI for automation:
python CSV2ELK.py --csv iocs.csv --index threat_intel-$(date +%Y%m%d)

# GUI for manual analysis:

Use Cases

1. Malware Log Analysis

[VirusTotal CSV] → 
  [CSV2ELK] → 
    [ELK Dashboard] → 
      (Correlate with EDR alerts)

2. Firewall/SIEM Data

[Palo Alto Log Export] → 
  [CSV2ELK] → 
    [GeoIP Visualizations]

Getting Started

  1. Install:

     
    pip install pandas requests tqdm

  2. Configure: Edit config.txt:

    #NOTE: on first run, this file will be created and managed by the application. If you do not see this, then it will be created on the first run of the application.
    ELASTICSEARCH_URL=https://your-elk:9200
USERNAME=elastic
PASSWORD=YOUR_PASSWORD

  3. Run:

     
    python CSV2ELK.py

Behind the Scenes

Technical Deep Dive

  • Python Libraries Used:

    • pandas → CSV parsing

    • requests → Elasticsearch API calls

    • tqdm → Progress bars

  • Error Handling:

    • Retries failed Elasticsearch batches (3x)

    • Validates timestamp formats before upload

Download & Contribute

💡 Pro Tip: Use with Elasticsearch’s ILM policies to auto-manage index retention.

Final Thoughts

CSV2ELK saves hours per investigation by eliminating manual data wrangling. Try it with:

  • Threat intel feeds (MISP, AlienVault OTX)

  • Cloud audit logs (AWS GuardDuty, Azure Sentinel)

Questions? Email me at dfirvault@gmail.com.

Tags: