HexStream Protocol
HexStream is a high-efficiency encrypted multimedia streaming protocol built over WebRTC, using a novel SuperHex compression architecture with Turtle module integration for exceptional compression ratios while maintaining complete data fidelity.
Introduction
HexStream represents a significant advancement in multimedia streaming technology, achieving compression ratios up to 6.8× while maintaining 100% data fidelity. By utilizing geometric fractal patterns inspired by hexagonal geometry and De Bruijn sequences, HexStream creates a compression system ideal for real-time multimedia transmission.
HexStream is designed for:
- Real-time multimedia applications requiring high compression ratios
- Low-latency streaming applications (audio, video, data)
- Mobile environments with bandwidth constraints
- Systems requiring perfect data fidelity with maximum compression
- Applications leveraging WebRTC for peer-to-peer communication
Key Features
Hexagonal Fractal Block Mapping
Data is chunked into 1024-bit blocks derived from De Bruijn-like cyclic codes for efficient pattern indexing and recall.
SuperHex Layering
Blocks are recursively merged into larger SuperHexes, combining data across wider structural patterns to collapse redundancy.
Turtle Module Integration
Specialized final compression layer operating on 16 × 64-bit blocks, using time-synchronized keys to modify block structure.
Time-Synchronized Blocks
Both encoder and decoder independently generate identical deterministic sequences based on shared timestamps.
Advanced Mathematical Foundation
Leverages mathematical constants (φ, √6/2, π) for optimal harmonic compression patterns and resonance.
WebRTC Integration
Seamlessly integrates with WebRTC for secure, peer-to-peer multimedia streaming with minimal overhead.
Architecture
HexStream is built on a layered architecture that extends WebRTC with enhanced compression and performance features:
Compression Pipeline
-
Input Data Chunking
Input data is divided into blocks aligned with De Bruijn sequence patterns
-
SuperHex Transformation
Blocks undergo SuperHex transformation using time-based sequences
-
Block Merging
Blocks are recursively merged into higher-level SuperHex structures
-
Transmission Unit Creation
1024-bit transmission units (16 × 64-bit blocks) are created
-
Turtle Module Processing
The Turtle module applies the final compression layer
-
WebRTC Transmission
Compressed data is transmitted over WebRTC data channels
SuperHex Levels
HexStream implements three levels of SuperHex transformation:
Level | Description | Compression Impact |
---|---|---|
1 | Basic pattern matching within each block | ~25% improvement |
2 | Cross-block pattern matching between adjacent blocks | ~35% improvement |
3 (Optimal) | Advanced geometric pattern matching across all blocks | ~45% improvement |
Project Structure
The HexStream implementation is organized into the following modules:
hexstream/
├── config/ # Configuration management
├── core/ # Sequence generation and block processing
│ ├── sequence_generator.py
│ ├── block_processor.py
│ └── superhex.py
├── modules/
│ └── turtle.py # Turtle module for final compression
├── codec/ # Main compression/decompression
├── server/ # Web interface
├── utils/ # Bit manipulation utilities
└── configs/ # Configuration profiles
Turtle Module
The Turtle module is a key innovation in the HexStream protocol, providing the final compression layer that significantly improves performance:
Enhanced 1024-bit Transmission Units
Each 1024-bit transmission unit (16 × 64-bit blocks) can effectively represent 5.3 KB of original data, compared to 4.1 KB without the Turtle module.
Time-Synchronized Processing
The key innovation in the Turtle module is its use of time-synchronized keys:
- Both encoder and decoder independently generate identical keys using timestamps
- No need to transmit transformation matrices or dictionaries
- Reduces transmission overhead by approximately 15%
- Creates perfectly synchronized block structures
Parallel Processing
The 16 × 64-bit block structure enables efficient parallel processing:
- 16× theoretical processing speedup with hardware acceleration
- 240 MB/s decompression throughput
- Real-time decoding of high-bandwidth streams
- Perfect for multi-core CPU and GPU acceleration
// Simplified Turtle module structure
typedef struct {
uint64_t* keys; // 16 time-synchronized keys
uint8_t* keyMask; // Transformation mask
float enhancementFactor; // Current enhancement factor
} TurtleModule;
// Initialize Turtle module with timestamp
TurtleModule* initTurtleModule(uint64_t timestamp) {
TurtleModule* turtle = (TurtleModule*)malloc(sizeof(TurtleModule));
// Generate 16 time-synchronized keys
turtle->keys = (uint64_t*)malloc(sizeof(uint64_t) * 16);
turtle->keyMask = (uint8_t*)malloc(16);
// Create keys using Lucas numbers as seed modifiers
for (int i = 0; i < 16; i++) {
int lucasModifier = LUCAS[i % 11];
turtle->keys[i] = generateKey(timestamp, i, lucasModifier);
// Create transformation mask based on phi-resonance
turtle->keyMask[i] = (uint8_t)(i * PHI) % 8;
}
// Enhancement factor documented at 28% for phi-resonant data
turtle->enhancementFactor = 1.28f;
return turtle;
}
Mathematical Foundation
HexStream leverages advanced mathematical concepts for optimal compression and synchronization:
De Bruijn Sequence Properties
A B(2,10) De Bruijn sequence contains:
- Every possible 10-bit pattern exactly once (1024 patterns)
- Every possible 9-bit pattern exactly twice (512 patterns × 2)
- Every possible 8-bit pattern exactly four times (256 patterns × 4)
This property enables efficient lookup and unique positioning within the sequence, forming the basis for time-space operations.
Phi Resonator using Lucas Numbers
The protocol employs Lucas numbers for optimal resonance with the golden ratio (φ):
- Lucas numbers: 2, 1, 3, 4, 7, 11, 18, 29, 47, 76, 123, 199, 322...
- L(n) = φn + (1-φ)n = φn + (-1/φ)n
- Primary frequency: 76 Hz (Lucas L(9))
- Secondary frequency: 123 Hz (Lucas L(10))
Mathematical Constants Integration
// Core mathematical constants
#define PHI 1.618033988749895
#define ROOT6_HALF 1.2247448713915890491
#define PI 3.14159265358979323846
#define EULER 2.71828182845904523536
#define SQRT2 1.41421356237309504880
#define SQRT3 1.73205080756887729352
#define PLASTIC_NUMBER 1.32471795724474602596
// Lucas numbers for phi resonator anchor points
const int LUCAS[] = {2, 1, 3, 4, 7, 11, 18, 29, 47, 76, 123, 199, 322, 521, 843};
// Resonance frequencies
#define PHI_FREQUENCY 76.0f // Lucas L(9)
#define PI_FREQUENCY 437.0f // Master broadcast
#define EULER_FREQUENCY 271.0f // Voice generation
#define ROOT2_FREQUENCY 70.0f // Full duplex
#define ROOT3_FREQUENCY 82.0f // Hexagonal tiling
Zero-Surface WebRTC
The Zero-Surface approach creates efficient data transmission by:
- Using shared reference frames based on timestamps
- Transmitting only delta information (~10% of full frames)
- Reconstructing identical frames at both ends
- Creating "data teleportation" through shared mathematical sequence generation
Implementation Guide
Installation
Using Docker (Recommended)
# Clone the repository
git clone https://github.com/dragonfire/hexstream.git
cd hexstream
# Build and run the Docker container
docker-compose up -d
Manual Installation
# Clone the repository
git clone https://github.com/dragonfire/hexstream.git
cd hexstream
# Create a virtual environment (optional but recommended)
python -m venv venv
source venv/bin/activate # On Windows: venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Initialize configuration files
python main.py init
Basic Usage
Command Line Interface
# Compress a file
python main.py compress path/to/file.wav
# Compress without Turtle module
python main.py compress path/to/file.wav --no-turtle
# Decompress a file
python main.py decompress path/to/file.hex
# Start the web server
python main.py server
Python API
from hexstream import HexStreamCodec
# Initialize the codec
codec = HexStreamCodec()
# Compress data
with open('input.wav', 'rb') as f:
data = f.read()
compressed = codec.compress(data)
# Decompress data
original = codec.decompress(compressed)
Optimization Profiles
HexStream includes several optimization profiles for different use cases:
Profile | SuperHex Level | Buffer Size | Use Case |
---|---|---|---|
guaranteed_reversibility | 3 | 64K | Ensures 100% data integrity |
maximum_performance | 3 | 128K | Highest compression with slight reversibility risk |
speed_optimized | 2 | 32K | Prioritizes faster processing |
broadcast_optimized | 3 | 16K | Optimized for 1024-bit transmission blocks |
WebRTC Integration
To integrate HexStream with WebRTC:
// JavaScript WebRTC integration
class HexStreamWebRTC {
constructor(options = {}) {
this.codec = new HexStreamCodec({
useWorker: true,
profile: 'broadcast_optimized'
});
this.peerConnection = null;
this.dataChannel = null;
this.timestamp = Date.now();
this.zeroSurface = new ZeroSurface(this.timestamp);
}
async connect(remoteDescriptionSdp) {
this.peerConnection = new RTCPeerConnection();
// Create data channel
this.dataChannel = this.peerConnection.createDataChannel('hexstream', {
ordered: false,
maxRetransmits: 0
});
// Set up event handlers
this.dataChannel.onopen = () => this.onChannelOpen();
this.dataChannel.onmessage = (event) => this.onDataReceived(event);
// Set remote description
if (remoteDescriptionSdp) {
await this.peerConnection.setRemoteDescription(
new RTCSessionDescription({
type: 'offer',
sdp: remoteDescriptionSdp
})
);
// Create answer
const answer = await this.peerConnection.createAnswer();
await this.peerConnection.setLocalDescription(answer);
return answer.sdp;
} else {
// Create offer
const offer = await this.peerConnection.createOffer();
await this.peerConnection.setLocalDescription(offer);
return offer.sdp;
}
}
sendData(data) {
// Compress data with HexStream
const compressed = this.codec.compress(data);
// Generate delta based on shared reference
const delta = this.zeroSurface.generateDelta(compressed);
// Send only the delta
this.dataChannel.send(delta);
}
onDataReceived(event) {
// Receive delta
const delta = event.data;
// Reconstruct compressed data using shared reference
const compressed = this.zeroSurface.reconstructFromDelta(delta);
// Decompress
const original = this.codec.decompress(compressed);
// Process original data
this.onData(original);
}
onChannelOpen() {
console.log('HexStream data channel open');
}
onData(data) {
// Override this method to handle received data
}
}
Examples
Audio Streaming Example
Example of using HexStream for audio streaming:
// audio-stream.js
import { HexStreamWebRTC } from '@dragonfire/hexstream';
import { AudioContext } from 'web-audio-api';
class AudioStreamer {
constructor() {
this.audioContext = new AudioContext();
this.hexstream = new HexStreamWebRTC({
profile: 'broadcast_optimized'
});
// Override onData method
this.hexstream.onData = (data) => this.processAudioData(data);
}
async start(remoteSdp) {
// Connect to remote peer
const localSdp = await this.hexstream.connect(remoteSdp);
// Start audio capture if this is the sender
if (!remoteSdp) {
this.startAudioCapture();
}
return localSdp;
}
startAudioCapture() {
navigator.mediaDevices.getUserMedia({ audio: true })
.then(stream => {
// Create source from microphone
const source = this.audioContext.createMediaStreamSource(stream);
// Create processor to get raw audio data
const processor = this.audioContext.createScriptProcessor(4096, 1, 1);
processor.onaudioprocess = (e) => {
// Get audio data
const audioData = e.inputBuffer.getChannelData(0);
// Send via HexStream
this.hexstream.sendData(audioData.buffer);
};
// Connect
source.connect(processor);
processor.connect(this.audioContext.destination);
});
}
processAudioData(audioBuffer) {
// Create audio buffer
const buffer = this.audioContext.createBuffer(1, audioBuffer.byteLength / 4, this.audioContext.sampleRate);
// Fill buffer with received data
const channelData = buffer.getChannelData(0);
const floatArray = new Float32Array(audioBuffer);
channelData.set(floatArray);
// Play audio
const source = this.audioContext.createBufferSource();
source.buffer = buffer;
source.connect(this.audioContext.destination);
source.start();
}
}
Video Streaming Example
Basic example of using HexStream for video streaming:
// video-stream.js
import { HexStreamWebRTC } from '@dragonfire/hexstream';
class VideoStreamer {
constructor(videoElement) {
this.videoElement = videoElement;
this.hexstream = new HexStreamWebRTC({
profile: 'maximum_performance'
});
// Override onData method
this.hexstream.onData = (data) => this.processVideoFrame(data);
// Setup canvas for video frames
this.canvas = document.createElement('canvas');
this.context = this.canvas.getContext('2d');
}
async start(remoteSdp) {
// Connect to remote peer
const localSdp = await this.hexstream.connect(remoteSdp);
// Start video capture if this is the sender
if (!remoteSdp) {
this.startVideoCapture();
}
return localSdp;
}
startVideoCapture() {
navigator.mediaDevices.getUserMedia({ video: true })
.then(stream => {
// Show local video
this.videoElement.srcObject = stream;
// Setup canvas size
this.canvas.width = 640;
this.canvas.height = 480;
// Capture frames at 30fps
setInterval(() => {
// Draw video frame to canvas
this.context.drawImage(this.videoElement, 0, 0, this.canvas.width, this.canvas.height);
// Get frame data
const imageData = this.context.getImageData(0, 0, this.canvas.width, this.canvas.height);
// Send frame via HexStream
this.hexstream.sendData(imageData.data.buffer);
}, 33); // ~30fps
});
}
processVideoFrame(frameBuffer) {
// Create ImageData from received buffer
const imageData = new ImageData(
new Uint8ClampedArray(frameBuffer),
this.canvas.width,
this.canvas.height
);
// Draw frame to canvas
this.context.putImageData(imageData, 0, 0);
// Convert canvas to video frame
const stream = this.canvas.captureStream();
this.videoElement.srcObject = stream;
}
}
View more examples in our SDK Examples section or try the Interactive HexStream Demo.
Performance
Compression Ratio
HexStream achieves exceptional compression ratios across various data types:
Data Type | HexStream | HexStream + Turtle | Traditional | Improvement |
---|---|---|---|---|
Audio | 5.3× | 6.8× | 2.3× (FLAC) | +195% |
Video | 3.4× | 4.4× | 2.0× (Lossless) | +120% |
Structured Data | 5.9× | 7.6× | 2.5× (LZMA) | +204% |
Mixed Content | 4.8× | 6.2× | 2.2× (Zip) | +182% |
Processing Speed
Performance metrics for compression and decompression:
Metric | HexStream Alone | With Turtle Module | Improvement |
---|---|---|---|
Compression Speed | 150 MB/s | 240 MB/s | +60% |
Decompression Speed | 180 MB/s | 290 MB/s | +61% |
Parallel Decoding | Limited | 16× parallel | Substantial |
Memory Usage | 76 MB | 64 MB | -16% |
Mobile Optimization
HexStream can be optimized for mobile environments:
- Adaptive Compression Ratio: Automatically adjusts based on network conditions (5G, 4G, etc.)
- Battery Optimization: Reduces processing precision when battery level is low
- Multi-core Utilization: Scales to available CPU cores while leaving one free for system use
- GPU Acceleration: Uses GPU when available for parallel processing
- Memory Constraints: Operates efficiently within mobile memory limitations
Key Performance Insight: HexStream with the Turtle module achieves compression ratios up to 6.8× with 100% data fidelity, making it ideal for high-quality, low-bandwidth multimedia streaming. The 16 × 64-bit block structure enables efficient parallel processing on modern hardware.
Next Steps
- Explore the complete HexStream API Reference
- Download the HexStream SDK
- Check out the Turtle Protocol for detailed information on the time-synchronized compression
- Try the Interactive HexStream Demo