custom-ocean.xyz-dashboard/project_structure.md

Enhanced Project Structure Documentation

This document provides a comprehensive overview of the Bitcoin Mining Dashboard project architecture, component relationships, and technical design decisions.

Directory Structure

bitcoin-mining-dashboard/
│
├── App.py                      # Main application entry point and Flask routes
├── config.py                   # Configuration management utilities
├── config.json                 # User configuration file
├── data_service.py             # Service for fetching mining/market data
├── models.py                   # Data models and conversion utilities
├── state_manager.py            # Manager for persistent state and history
├── worker_service.py           # Service for worker data management
│
├── templates/                  # HTML templates
│   ├── base.html              # Base template with common elements
│   ├── boot.html              # Boot sequence animation
│   ├── dashboard.html         # Main dashboard template
│   ├── workers.html           # Workers overview template
│   ├── blocks.html            # Bitcoin blocks template
│   └── error.html             # Error page template
│
├── static/                     # Static assets
│   ├── css/                   # CSS files
│   │   ├── common.css         # Shared styles across all pages
│   │   ├── dashboard.css      # Main dashboard styles
│   │   ├── workers.css        # Workers page styles
│   │   ├── boot.css           # Boot sequence styles
│   │   ├── blocks.css         # Blocks page styles
│   │   ├── error.css          # Error page styles
│   │   └── retro-refresh.css  # Floating system monitor styles
│   │
│   └── js/                    # JavaScript files
│       ├── main.js            # Main dashboard functionality
│       ├── workers.js         # Workers page functionality
│       ├── blocks.js          # Blocks page functionality
│       ├── block-animation.js # Block mining animation
│       └── BitcoinProgressBar.js # System monitor implementation
│
├── logs/                       # Application logs directory
├── requirements.txt            # Python dependencies
├── Dockerfile                  # Docker configuration
├── setup.py                    # Setup script for organizing files
├── deployment_steps.md         # Deployment documentation
├── project_structure.md        # This document
└── README.md                   # Project overview and instructions

Core Components

Backend Services

App.py

The main Flask application that serves as the entry point. It:

• Initializes the application and its components
• Configures routes and middleware
• Sets up the background scheduler for data updates
• Manages Server-Sent Events (SSE) connections
• Handles error recovery and graceful shutdown

Key features:

• Custom middleware for error handling
• Connection limiting for SSE to prevent resource exhaustion
• Watchdog process for scheduler health
• Metrics caching with controlled update frequency

data_service.py

Service responsible for fetching data from external sources:

• Retrieves mining statistics from Ocean.xyz
• Collects Bitcoin network data (price, difficulty, hashrate)
• Calculates profitability metrics
• Handles connection issues and retries

Notable implementations:

• Concurrent API requests using ThreadPoolExecutor
• Multiple parsing strategies for resilience against HTML changes
• Intelligent caching to reduce API load
• Unit normalization for consistent display

worker_service.py

Service for managing worker data:

• Fetches worker statistics from Ocean.xyz
• Simulates worker data when real data is unavailable
• Provides filtering and search capabilities
• Tracks worker status and performance

Key features:

• Fallback data generation for testing or connectivity issues
• Smart worker count synchronization
• Hashrate normalization across different units

state_manager.py

Manager for application state and history:

• Maintains hashrate history and metrics over time
• Provides persistence via Redis (optional)
• Implements data pruning to prevent memory growth
• Records indicator arrows for value changes

Implementation details:

• Thread-safe collections with locking
• Optimized storage format for Redis
• Data compression techniques for large state objects
• Automatic recovery of critical state

Frontend Components

Templates

The application uses Jinja2 templates with a retro-themed design:

• base.html: Defines the common layout, navigation, and includes shared assets
• dashboard.html: Main metrics display with hashrate chart and financial calculations
• workers.html: Grid layout of worker cards with filtering controls
• blocks.html: Bitcoin block explorer with detailed information
• boot.html: Animated terminal boot sequence
• error.html: Styled error page with technical information

JavaScript Modules

Client-side functionality is organized into modular JavaScript files:

• main.js: Dashboard functionality, real-time updates, and chart rendering
• workers.js: Worker grid rendering, filtering, and mini-chart creation
• blocks.js: Block explorer with data fetching from mempool.guide
• block-animation.js: Interactive block mining animation
• BitcoinProgressBar.js: Floating system monitor with uptime and connection status

Key client-side features:

• Real-time data updates via Server-Sent Events (SSE)
• Automatic reconnection with exponential backoff
• Cross-tab synchronization using localStorage
• Data normalization for consistent unit display
• Animated UI elements for status changes

Architecture Overview

Data Flow

1. Data Acquisition:

   • data_service.py fetches data from Ocean.xyz and blockchain sources
   • Data is normalized, converted, and enriched with calculated metrics
   • Results are cached in memory

2. State Management:

   • state_manager.py tracks historical data points
   • Maintains arrow indicators for value changes
   • Optionally persists state to Redis

3. Background Updates:

   • Scheduler runs periodic updates (typically once per minute)
   • Updates are throttled to prevent API overload
   • Watchdog monitors scheduler health

4. Real-time Distribution:

   • New data is pushed to clients via Server-Sent Events
   • Clients process and render updates without page reloads
   • Connection management prevents resource exhaustion

5. Client Rendering:

   • Browser receives and processes JSON updates
   • Chart.js visualizes hashrate trends
   • DOM updates show changes with visual indicators
   • BitcoinProgressBar shows system status

System Resilience

The application implements multiple resilience mechanisms:

Server-Side Resilience

• Scheduler Recovery: Auto-detects and restarts failed schedulers
• Memory Management: Prunes old data to prevent memory growth
• Connection Limiting: Caps maximum concurrent SSE connections
• Graceful Degradation: Falls back to simpler data when sources are unavailable
• Adaptive Parsing: Multiple strategies to handle API and HTML changes

Client-Side Resilience

• Connection Recovery: Automatic reconnection with exponential backoff
• Fallback Polling: Switches to traditional AJAX if SSE fails
• Local Storage Synchronization: Shares data across browser tabs
• Visibility Handling: Optimizes updates based on page visibility

Technical Design Decisions

Server-Sent Events vs WebSockets

The application uses SSE instead of WebSockets because:

• Data flow is primarily one-directional (server to client)
• SSE has better reconnection handling
• Simpler implementation without additional dependencies
• Better compatibility with proxy servers

Single Worker Model

The application uses a single Gunicorn worker with multiple threads because:

• Shared in-memory state is simpler than distributed state
• Reduces complexity of synchronization
• Most operations are I/O bound, making threads effective
• Typical deployments have moderate user counts

Optional Redis Integration

Redis usage is optional because:

• Small deployments don't require persistence
• Makes local development simpler
• Allows for flexible deployment options

Hashrate Normalization

All hashrates are normalized to TH/s internally because:

• Provides consistent basis for comparisons
• Simplifies trend calculations and charts
• Allows for unit conversion on display

Component Interactions

┌─────────────────┐      ┌─────────────────┐      ┌─────────────────┐
│  Ocean.xyz API  │      │  blockchain.info │      │  mempool.guide  │
└────────┬────────┘      └────────┬────────┘      └────────┬────────┘
         │                        │                        │
         ▼                        ▼                        ▼
┌────────────────────────────────────────────────────────────────────┐
│                           data_service.py                           │
└────────────────────────────────┬───────────────────────────────────┘
                                 │
                                 ▼
┌────────────────────────────────────────────────────────────────────┐
│                             App.py                                  │
├────────────────────────────────────────────────────────────────────┤
│                                                                     │
│  ┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐ │
│  │  worker_service │    │  state_manager  │    │ Background Jobs  │ │
│  └─────────────────┘    └─────────────────┘    └─────────────────┘ │
│                                                                     │
└───────────────────────────────┬────────────────────────────────────┘
                                │
                                ▼
┌────────────────────────────────────────────────────────────────────┐
│                         Flask Routes & SSE                          │
└───────────────────────────────┬────────────────────────────────────┘
                                │
         ┌────────────────────────────────────────────┐
         │                                            │
         ▼                                            ▼
┌─────────────────┐                          ┌─────────────────┐
│  Browser Tab 1  │                          │  Browser Tab N  │
└─────────────────┘                          └─────────────────┘

Performance Considerations

Memory Usage

• Arrow history is pruned to prevent unbounded growth
• Older data points are stored at reduced resolution
• Regular garbage collection cycles are scheduled
• Memory usage is logged for monitoring

Network Optimization

• Data is cached to reduce API calls
• Updates are throttled to reasonable frequencies
• SSE connections have a maximum lifetime
• Failed connections use exponential backoff

Browser Performance

• Charts use optimized rendering with limited animation
• DOM updates are batched where possible
• Data is processed in small chunks
• CSS transitions are used for smooth animations

Future Enhancement Areas

1. Database Integration: Option for SQL database for long-term metrics storage
2. User Authentication: Multi-user support with separate configurations
3. Mining Pool Expansion: Support for additional mining pools beyond Ocean.xyz
4. Mobile App: Dedicated mobile application with push notifications
5. Advanced Analytics: Profitability projections and historical analysis