File Monitor Dual-Log Configuration Guide

Date: 2025-10-06 Status: ✅ Implemented Version: 2.0

Overview

The file monitor now supports full internal control of both output streams without requiring shell redirection. Both trace logs and event logs can be configured via CLI flags.

---

✅ What's New (v2.0)

Previous Limitation (v1.0)

# Required shell redirection for dual logs
./monitor /path --format json > events.log 2> trace.log

New Capability (v2.0)

# Monitor handles both logs internally - NO shell redirection needed
./monitor /path \
  --trace-log .coditect/logs/trace.log \
  --events-log .coditect/logs/events.log

---

🎯 CLI Flags Reference

Trace Log Control

Flag	Type	Default	Description
--trace-log <PATH>	Option<PathBuf>	stderr	Lifecycle events (INFO/WARN/ERROR logs)
--no-logs	bool	false	Disable all trace logging completely

Events Log Control

Flag	Type	Default	Description
--events-log <PATH>	Option<PathBuf>	stdout	JSON file events output
--format <FMT>	json | pretty	pretty	Event output format

---

📋 Usage Examples

Example 1: Dual Internal Logging (Recommended)

./monitor /home/hal/v4/PROJECTS \
  --recursive \
  --checksums \
  --format json \
  --trace-log .coditect/logs/trace.log \
  --events-log .coditect/logs/events.log

Result:

• ✅ Trace logs → .coditect/logs/trace.log
• ✅ Event logs → .coditect/logs/events.log
• ✅ No shell redirection needed
• ✅ Clean process management

When to use: Production deployment, systemd services, Docker containers

---

Example 2: Events to File, Traces to stderr

./monitor /path/to/watch \
  --recursive \
  --checksums \
  --format json \
  --events-log events.log

Result:

• ✅ Trace logs → stderr (terminal)
• ✅ Event logs → events.log

When to use: Development, want to see lifecycle logs in terminal while capturing events

---

Example 3: Events to stdout, Traces to File

./monitor /path/to/watch \
  --recursive \
  --format json \
  --trace-log trace.log

Result:

• ✅ Trace logs → trace.log
• ✅ Event logs → stdout (terminal or pipe)

When to use: Piping events to another process: ./monitor ... | jq .

---

Example 4: Events Only (No Traces)

./monitor /path/to/watch \
  --recursive \
  --format json \
  --no-logs \
  --events-log events.log

Result:

• ❌ Trace logs → disabled
• ✅ Event logs → events.log

When to use: Minimal output, events-only pipeline, no lifecycle visibility needed

---

Example 5: Pretty Format to File

./monitor /path/to/watch \
  --recursive \
  --checksums \
  --format pretty \
  --trace-log trace.log \
  --events-log events.log

Result:

• ✅ Trace logs → trace.log (with emoji and formatting)
• ✅ Event logs → events.log (human-readable with icons)

When to use: Human review, debugging, not machine parsing

---

Example 6: Background Service

nohup ./monitor /home/hal/v4/PROJECTS \
  --recursive \
  --checksums \
  --format json \
  --trace-log .coditect/logs/trace.log \
  --events-log .coditect/logs/events.log \
  &

echo $! > .coditect/logs/monitor.pid

Result:

• ✅ Runs in background
• ✅ Both logs written to files
• ✅ PID saved for management
• ✅ No shell redirection confusion

When to use: Long-running monitoring, system integration

---

Example 7: Polling Mode (No inotify)

./monitor /home/hal/v4 \
  --recursive \
  --format json \
  --poll \
  --poll-interval 5 \
  --trace-log .coditect/logs/trace.log \
  --events-log .coditect/logs/events.log

Result:

• ✅ Uses polling instead of inotify (no 3-minute startup delay)
• ✅ Works with large directory trees (1,857+ directories)
• ✅ No kernel memory limits
• ⚠️ Higher CPU usage, 5-second detection latency

When to use: Very large directories, NFS/FUSE filesystems, inotify limits

---

🔍 Output Stream Behavior

Default Behavior (No Flags)

./monitor /path --format pretty

Stream	Destination	Content
Trace logs	stderr	Lifecycle events with observability
Event logs	stdout	Pretty-formatted events

---

With --trace-log Only

./monitor /path --trace-log trace.log --format json

Stream	Destination	Content
Trace logs	trace.log	Lifecycle events
Event logs	stdout	JSON events

---

With --events-log Only

./monitor /path --events-log events.log --format json

Stream	Destination	Content
Trace logs	stderr	Lifecycle events
Event logs	events.log	JSON events

---

With Both Flags

./monitor /path --trace-log trace.log --events-log events.log --format json

Stream	Destination	Content
Trace logs	trace.log	Lifecycle events
Event logs	events.log	JSON events

✅ Complete internal control - no shell redirection needed!

---

With --no-logs

./monitor /path --no-logs --events-log events.log --format json

Stream	Destination	Content
Trace logs	❌ Disabled	None
Event logs	events.log	JSON events only

---

🏗️ Implementation Details

Code Changes

1. CLI Flags (monitor.rs:50-56)

/// Trace log file for observability (lifecycle events, defaults to stderr)
#[structopt(long)]
trace_log: Option<PathBuf>,

/// Events log file for JSON events (defaults to stdout)
#[structopt(long)]
events_log: Option<PathBuf>,

2. Trace Log Setup (monitor.rs:94-122)

if !cli.no_logs {
    if let Some(log_path) = &cli.trace_log {
        // Open trace log file
        let log_file = OpenOptions::new()
            .create(true)
            .append(true)
            .open(log_path)?;

        tracing_subscriber::fmt()
            .with_writer(log_file)
            .with_ansi(false)
            .with_target(false)
            .init();
    } else {
        // Default to stderr
        // ...
    }
}

3. Events Log Setup (monitor.rs:170-180)

// Open events log file if specified
let mut events_writer: Box<dyn std::io::Write + Send> =
    if let Some(events_path) = &cli.events_log {
        let events_file = OpenOptions::new()
            .create(true)
            .append(true)
            .open(events_path)?;
        Box::new(events_file)
    } else {
        Box::new(std::io::stdout())
    };

4. Unified Display Function (monitor.rs:211-272)

fn display_event(
    event: &AuditEvent,
    format: &OutputFormat,
    writer: &mut dyn std::io::Write
) -> anyhow::Result<()> {
    match format {
        OutputFormat::Json => {
            writeln!(writer, "{}", serde_json::to_string(event)?)?;
            writer.flush()?;
        }
        OutputFormat::Pretty => {
            // Format and write to provided writer
            write!(writer, "...")?;
            writeln!(writer)?;
            writer.flush()?;
        }
    }
    Ok(())
}

---

🎨 File Output Characteristics

Trace Log Format

File: .coditect/logs/trace.log

2025-10-06T14:15:49.370913Z  INFO new: File monitor created path=/home/hal/v4 recursive=true
2025-10-06T14:15:49.443821Z  INFO start: File monitoring started path=/home/hal/v4 mode=Recursive
2025-10-06T14:32:17.891234Z  WARN process_event: Checksum timeout exceeded file=/large-file.bin
2025-10-06T14:45:03.123456Z  INFO shutdown: Graceful shutdown initiated

Features:

• RFC3339 timestamps with nanosecond precision
• Log level (INFO, WARN, ERROR)
• Span names (new, start, process_event, shutdown)
• Structured key-value pairs
• No ANSI colors (plain text)

---

Events Log Format (JSON)

File: .coditect/logs/events.log

{"id":"d423df9e-285c-4544-9664-e4f901ca3d2d","timestamp_utc":"2025-10-06T14:02:43.700758101Z","event_type":{"type":"created"},"file_path":"/home/hal/v4/test.txt","user_id":"hal","process_name":"monitor","checksum":"c160bd30","file_size":40,"metadata":{}}
{"id":"8a92ef1c-4d3a-4e2b-9f1d-2e8c7b6a5d4c","timestamp_utc":"2025-10-06T14:02:43.705868234Z","event_type":{"type":"modified","modification_type":"content"},"file_path":"/home/hal/v4/test.txt","user_id":"hal","process_name":"monitor","checksum":"d8f3b21a","file_size":87,"metadata":{}}

Features:

• One JSON object per line (JSONL format)
• Immediate flush after each event
• UUID for each event
• Nanosecond-precision timestamps
• Structured metadata field
• Machine-parseable

---

Events Log Format (Pretty)

File: .coditect/logs/events.log

✨ CREATED    /home/hal/v4/test.txt (40B) [c160bd30] by hal
📝 MODIFIED   /home/hal/v4/test.txt (87B) [d8f3b21a] by hal
🗑️  DELETED    /home/hal/v4/old-file.txt by hal
📋 RENAMED    /home/hal/v4/old.txt -> /home/hal/v4/new.txt

Features:

• Emoji icons for event types
• Fixed-width columns
• Human-readable file sizes (B, KB, MB, GB)
• First 8 chars of checksum
• User attribution
• Color-free (file-safe)

---

🚀 Production Deployment

Systemd Service Example

[Unit]
Description=File Monitor Audit Service
After=network.target

[Service]
Type=simple
User=hal
WorkingDirectory=/home/hal/v4
ExecStart=/usr/local/bin/file-monitor \
  /home/hal/v4/PROJECTS \
  --recursive \
  --checksums \
  --format json \
  --trace-log /var/log/file-monitor/trace.log \
  --events-log /var/log/file-monitor/events.log \
  --poll \
  --poll-interval 2
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

---

Docker Container Example

FROM debian:trixie-slim

# Install file-monitor binary
COPY target/release/file-monitor /usr/local/bin/

# Create log directory
RUN mkdir -p /.coditect/logs

# Run with internal log handling
CMD ["/usr/local/bin/file-monitor", \
     "/workspace", \
     "--recursive", \
     "--checksums", \
     "--format", "json", \
     "--trace-log", "/.coditect/logs/trace.log", \
     "--events-log", "/.coditect/logs/events.log"]

---

📊 Comparison: v1.0 vs v2.0

Feature	v1.0 (Shell Redirection)	v2.0 (Internal Control)
Trace log destination	2> trace.log	--trace-log trace.log
Events log destination	> events.log	--events-log events.log
Command complexity	High (shell syntax)	Low (simple flags)
Systemd compatibility	Requires /bin/sh wrapper	Direct ExecStart
Docker compatibility	Requires shell entrypoint	Direct CMD
Cross-platform	Shell-dependent	Universal
File handle management	Shell's responsibility	Monitor's control
Flush control	Shell buffer	Immediate flush
Error handling	Shell exit codes	Rust Result<>

---

🐛 Troubleshooting

Issue: No events appearing in events.log

Check:

# Verify file is being written
ls -lh .coditect/logs/events.log

# Check if monitor is running
ps aux | grep file-monitor

# Check trace log for errors
tail -f .coditect/logs/trace.log

Solution: Events are flushed immediately, so if file is empty, either:

1. No events detected yet (wait for file changes)
2. Monitor crashed (check trace.log)
3. Wrong path being monitored

---

Issue: Trace log not updating

Check:

# Verify trace logging is enabled
# Should NOT have --no-logs flag

# Check trace log exists and is writable
ls -lh .coditect/logs/trace.log

# Test with explicit trace log
./monitor /tmp/test --trace-log /tmp/trace.log

Solution: Trace logs are buffered, wait for:

• Monitor lifecycle events (start, shutdown)
• Warnings or errors
• Use --format pretty for more verbose output

---

Issue: Permission denied on log files

Solution:

# Ensure log directory exists and is writable
mkdir -p .coditect/logs
chmod 755 .coditect/logs

# Check file ownership
ls -la .coditect/logs/
chown -R $USER:$USER .coditect/logs/

---

🎯 Recommended Configuration

For Production

./monitor /workspace/projects \
  --recursive \
  --checksums \
  --format json \
  --poll \
  --poll-interval 2 \
  --trace-log /var/log/file-monitor/trace.log \
  --events-log /var/log/file-monitor/events.log

Why:

• ✅ Polling avoids inotify limits
• ✅ JSON format for machine parsing
• ✅ Checksums for integrity
• ✅ Explicit log file paths
• ✅ No shell redirection complexity

---

For Development

./monitor /workspace/PROJECTS/t2 \
  --recursive \
  --checksums \
  --format pretty \
  --trace-log trace.log \
  --events-log events.log

Why:

• ✅ Pretty format for human reading
• ✅ Local log files
• ✅ Smaller directory (faster startup)
• ✅ Easy to tail and debug

---

For CI/CD Pipeline

./monitor /workspace \
  --recursive \
  --no-logs \
  --format json \
  --events-log events.jsonl

Why:

• ✅ No trace noise in CI logs
• ✅ Events only for analysis
• ✅ JSONL format for parsing
• ✅ Minimal output

---

📚 Related Documentation

• Timing Analysis: docs/file-monitor/TIMING-analysis.md
• inotify Performance: docs/file-monitor/inotify-performance-issue.md
• Database Integration: docs/file-monitor/database-integration.md
• Test Results: docs/status-reports/file-monitor-test-success.md

---

🎓 Summary

You asked for: Monitor configured for dual output without shell redirection

We delivered:

• ✅ --trace-log <PATH> for lifecycle events
• ✅ --events-log <PATH> for JSON events
• ✅ Both handled internally by monitor
• ✅ No shell redirection needed
• ✅ Production-ready with systemd/Docker examples

Example usage:

./monitor /path/to/watch \
  --trace-log trace.log \
  --events-log events.log \
  --format json \
  --recursive \
  --checksums

Both logs written. No shell magic. Pure Rust control. ✨

---

Version: 2.0 Date: 2025-10-06 Status: ✅ Ready for Production