DragonFire Developer Portal

Merlin API

Related APIs

Resources

Merlin API Reference

Version 2.4.7

The DragonFire Merlin API provides advanced recursive harmonic data compression and processing capabilities. This API enables your applications to compress and process data with unprecedented efficiency using mathematically optimal patterns.

IMPORTANT: The Merlin API requires an active DragonFire service connection. Initialize the Hexwave connection before using Merlin's services.

Overview

The Merlin API provides recursive harmonic data compression and intelligent data processing capabilities. By leveraging mathematical constants and patterns found in nature, Merlin achieves compression ratios that exceed traditional algorithms while preserving data integrity.

Key Features

  • Recursive Compression: Multi-pass recursive compression using harmonic patterns
  • Adaptive Algorithms: Self-optimizing compression based on data characteristics
  • Pattern Recognition: Identifies and optimizes for mathematical patterns in data
  • Streaming Support: Efficient compression of streaming data
  • Lossless and Lossy Modes: Configurable precision for different use cases

Compression Architecture

Merlin Compression Architecture

Client SDK Requirements

To use the Merlin API, you'll need the DragonFire Merlin Client SDK:

# Using npm
npm install @dragonfire/merlin-client

# Using yarn
yarn add @dragonfire/merlin-client

Initialization

Before using the Merlin API, you need to initialize a client instance with a connection to the DragonFire service.

Creating a Merlin Client

MerlinClient constructor

Creates a new client for interacting with the DragonFire Merlin service.

Syntax
constructor(dragonfire: DragonFireClient, options?: MerlinClientOptions)
Parameters
Name Type Description
dragonfire DragonFireClient An initialized and connected DragonFire client instance.
options MerlinClientOptions Optional. Configuration options for the Merlin client.
Example
import { DragonFireClient } from '@dragonfire/client';
import { MerlinClient } from '@dragonfire/merlin-client';

// Initialize DragonFire client
const dragonfire = new DragonFireClient({
  apiKey: 'YOUR_API_KEY',
  region: 'us-west'
});
await dragonfire.connect();

// Initialize Merlin client
const merlin = new MerlinClient(dragonfire, {
  defaultPattern: 'phi',        // Mathematical pattern to use (phi, pi, sqrt2, sqrt3, e)
  adaptiveMode: true,           // Automatically select optimal patterns
  recursionDepth: 3,            // Maximum recursion depth for compression
  compressThreshold: 1024,      // Min size in bytes to trigger compression
  cacheResults: true            // Cache compression results
});

Configuration Options

The Merlin client can be configured with various options:

Option Type Default Description
defaultPattern string 'phi' Default mathematical pattern for compression.
adaptiveMode boolean true Automatically select optimal compression patterns based on data.
recursionDepth number 3 Maximum recursion depth for compression (1-7).
compressThreshold number 1024 Minimum size in bytes to trigger compression.
cacheResults boolean true Cache compression results for improved performance.
workerCount number 2 Number of parallel workers for compression.
asyncCompression boolean true Use asynchronous compression when possible.
hashAlgorithm string 'phi-hash' Hash algorithm for data checksums.

Compression

The Merlin API provides methods for compressing data using harmonic patterns.

Basic Compression

compress()

Compresses data using the configured harmonic patterns.

Syntax
async compress(data: Buffer | ArrayBuffer | Uint8Array | string, options?: CompressionOptions): Promise
Parameters
Name Type Description
data Buffer | ArrayBuffer | Uint8Array | string The data to compress. Strings will be encoded using UTF-8.
options CompressionOptions Optional. Additional options for the compression operation.
Returns

A Promise that resolves to a CompressionResult object containing the compressed data and metadata.

Example
// Compress a string
const textData = 'Lorem ipsum dolor sit amet, consectetur adipiscing elit...';
const result = await merlin.compress(textData);

console.log(`Original size: ${result.originalSize} bytes`);
console.log(`Compressed size: ${result.compressedSize} bytes`);
console.log(`Compression ratio: ${result.ratio.toFixed(2)}x`);
console.log(`Pattern used: ${result.patternUsed}`);

// Compress binary data with options
const binaryData = new Uint8Array([/* ... */]);
const result2 = await merlin.compress(binaryData, {
  pattern: 'pi',           // Override default pattern
  quality: 'max',          // Maximum compression quality
  lossless: true,          // Ensure lossless compression
  priority: 'ratio',       // Prioritize compression ratio over speed
  recursionDepth: 5,       // Higher recursion depth
  metadata: {              // Custom metadata
    contentType: 'image/png',
    fileName: 'example.png'
  }
});

Compression Options

Configure compression behavior with various options:

Option Type Description
pattern string Mathematical pattern to use ('phi', 'pi', 'sqrt2', 'sqrt3', 'e', 'auto').
quality string Compression quality ('low', 'medium', 'high', 'max'). Affects speed vs. ratio.
lossless boolean Whether to use lossless compression. Default is true.
priority string Optimization priority ('speed', 'ratio', 'balanced').
recursionDepth number Override the default recursion depth (1-7).
metadata object Custom metadata to include with the compressed data.
preAnalyze boolean Whether to analyze data before compression to optimize parameters.
segmentSize number Size of data segments for parallel processing (in bytes).

Batch Compression

compressBatch()

Compresses multiple data items in a single operation.

Syntax
async compressBatch(items: Array<{data: Buffer | ArrayBuffer | Uint8Array | string, options?: CompressionOptions}>, batchOptions?: BatchOptions): Promise>
Returns

A Promise that resolves to an array of CompressionResult objects, one for each input item.

Example
// Compress multiple items
const results = await merlin.compressBatch([
  { 
    data: textData1,
    options: { pattern: 'phi', quality: 'high' }
  },
  { 
    data: imageData,
    options: { pattern: 'pi', lossless: true }
  },
  { 
    data: jsonData,
    options: { priority: 'ratio' }
  }
], {
  parallel: true,          // Process in parallel
  progressCallback: (progress) => {
    console.log(`Compression progress: ${progress.toFixed(2)}%`);
  }
});

// Log results
results.forEach((result, index) => {
  console.log(`Item ${index + 1}:`);
  console.log(`  Original size: ${result.originalSize} bytes`);
  console.log(`  Compressed size: ${result.compressedSize} bytes`);
  console.log(`  Ratio: ${result.ratio.toFixed(2)}x`);
});

Decompression

The Merlin API provides methods for decompressing previously compressed data.

Basic Decompression

decompress()

Decompresses data that was previously compressed with Merlin.

Syntax
async decompress(compressedData: Buffer | ArrayBuffer | Uint8Array, options?: DecompressionOptions): Promise
Parameters
Name Type Description
compressedData Buffer | ArrayBuffer | Uint8Array The compressed data to decompress.
options DecompressionOptions Optional. Additional options for the decompression operation.
Returns

A Promise that resolves to a DecompressionResult object containing the decompressed data and metadata.

Example
// Decompress data
const compressedData = result.data; // From previous compression
const decompressed = await merlin.decompress(compressedData);

console.log(`Decompressed size: ${decompressed.data.byteLength} bytes`);
console.log(`Original checksum verified: ${decompressed.checksumVerified}`);

// Get decompressed data as string
const textContent = decompressed.toString('utf8');

// Decompress with options
const decompressed2 = await merlin.decompress(compressedData, {
  outputFormat: 'string',    // Get result as string
  encoding: 'utf8',          // Output encoding
  verifyChecksum: true,      // Verify data integrity
  metadata: true             // Include original metadata
});

Decompression Options

Configure decompression behavior with various options:

Option Type Description
outputFormat string Output format ('buffer', 'arrayBuffer', 'uint8Array', 'string').
encoding string Character encoding when outputFormat is 'string' (e.g., 'utf8', 'ascii').
verifyChecksum boolean Whether to verify data integrity using the embedded checksum.
metadata boolean Whether to extract and include the original metadata.
forceLossless boolean Force lossless decompression mode if possible.

Batch Decompression

decompressBatch()

Decompresses multiple compressed data items in a single operation.

Syntax
async decompressBatch(items: Array<{data: Buffer | ArrayBuffer | Uint8Array, options?: DecompressionOptions}>, batchOptions?: BatchOptions): Promise>
Returns

A Promise that resolves to an array of DecompressionResult objects, one for each input item.

Example
// Decompress multiple items
const decompressedResults = await merlin.decompressBatch([
  { data: compressedItem1 },
  { data: compressedItem2, options: { outputFormat: 'string' } },
  { data: compressedItem3 }
], {
  parallel: true,
  progressCallback: (progress) => {
    console.log(`Decompression progress: ${progress.toFixed(2)}%`);
  }
});

Harmonic Patterns

Merlin uses mathematical patterns derived from natural constants to optimize compression. Different patterns are suited to different types of data.

Available Patterns

Pattern Mathematical Basis Optimal Data Types
phi Golden Ratio (φ = 1.618...) Natural language text, organic imagery, recursive structures
pi Pi (π = 3.14159...) Scientific data, circular/cyclical patterns, audio waveforms
sqrt2 Square Root of 2 (√2 = 1.414...) Binary data, computer-generated content, geometric data
sqrt3 Square Root of 3 (√3 = 1.732...) 3D coordinates, triangular meshes, crystal structures
e Euler's Number (e = 2.718...) Growth patterns, logarithmic data, financial information
auto Adaptive selection Mixed content, unknown data types

Pattern Analysis

analyzePattern()

Analyzes data to determine the optimal compression pattern.

Syntax
async analyzePattern(data: Buffer | ArrayBuffer | Uint8Array | string, options?: AnalysisOptions): Promise
Returns

A Promise that resolves to a PatternAnalysisResult object containing the recommended patterns and analysis details.

Example
// Analyze data to find optimal pattern
const analysis = await merlin.analyzePattern(data, {
  detailed: true,        // Include detailed breakdown
  testCompression: true  // Run test compressions
});

console.log('Recommended pattern:', analysis.recommendedPattern);
console.log('Pattern scores:');
for (const [pattern, score] of Object.entries(analysis.patternScores)) {
  console.log(`  ${pattern}: ${score.toFixed(2)}`);
}

// Use recommended pattern for compression
const result = await merlin.compress(data, {
  pattern: analysis.recommendedPattern
});

Understanding Pattern Selection

The effectiveness of each pattern depends on the inherent mathematical structures within your data. When in doubt, use the 'auto' setting to let Merlin analyze and select the optimal pattern, or run the analyzePattern() method for a detailed analysis.

Custom Patterns

Advanced users can define custom compression patterns:

// Define a custom harmonic pattern
const customPattern = {
  name: 'fibonacci',
  baseValues: [1, 1, 2, 3, 5, 8, 13, 21, 34, 55],
  transformations: [
    { type: 'normalize', factor: 0.1 },
    { type: 'rotate', angle: Math.PI / 4 }
  ],
  recursiveFunction: (n) => n <= 1 ? n : customPattern.recursiveFunction(n-1) + customPattern.recursiveFunction(n-2)
};

// Register custom pattern
await merlin.registerPattern(customPattern);

// Use custom pattern
const result = await merlin.compress(data, {
  pattern: 'fibonacci'
});

Streaming API

For large data sets or continuous data streams, Merlin provides a streaming API that processes data in chunks.

Compression Stream

createCompressionStream()

Creates a transform stream for compressing data.

Syntax
createCompressionStream(options?: CompressionOptions): MerlinCompressionStream
Returns

A MerlinCompressionStream instance that can be used in a standard Node.js pipeline.

Example
import { pipeline } from 'stream/promises';
import fs from 'fs';

// Create compression stream
const compressionStream = merlin.createCompressionStream({
  pattern: 'phi',
  quality: 'high',
  chunkSize: 65536  // 64 KB chunks
});

// Set up event listeners
compressionStream.on('progress', (progress) => {
  console.log(`Compression progress: ${progress.toFixed(2)}%`);
});

compressionStream.on('stats', (stats) => {
  console.log(`Current ratio: ${stats.currentRatio.toFixed(2)}x`);
  console.log(`Bytes processed: ${stats.bytesProcessed}`);
});

// Set up a pipeline
await pipeline(
  fs.createReadStream('largefile.dat'),
  compressionStream,
  fs.createWriteStream('largefile.dat.mln')
);

console.log('Compression complete');
console.log('Final stats:', compressionStream.getFinalStats());

Decompression Stream

createDecompressionStream()

Creates a transform stream for decompressing data.

Syntax
createDecompressionStream(options?: DecompressionOptions): MerlinDecompressionStream
Returns

A MerlinDecompressionStream instance that can be used in a standard Node.js pipeline.

Example
import { pipeline } from 'stream/promises';
import fs from 'fs';

// Create decompression stream
const decompressionStream = merlin.createDecompressionStream({
  verifyChecksum: true
});

// Set up a pipeline
await pipeline(
  fs.createReadStream('largefile.dat.mln'),
  decompressionStream,
  fs.createWriteStream('largefile_restored.dat')
);

console.log('Decompression complete');

Web Streaming API

For browser environments, Merlin supports the Web Streams API:

// Browser environment using Web Streams API
// Compress a stream from fetch
const response = await fetch('https://example.com/largefile.dat');
const readableStream = response.body;

// Create compression transformer
const compressionTransform = merlin.createCompressionTransformer({
  pattern: 'sqrt2',
  quality: 'high'
});

// Create transformed stream
const compressedStream = readableStream.pipeThrough(compressionTransform);

// Use the compressed stream
// e.g., save to a file using StreamSaver.js
const fileStream = streamSaver.createWriteStream('compressed.mln');
compressedStream.pipeTo(fileStream);

Optimization

Merlin provides methods to optimize compression for specific use cases and data types.

Compression Profiles

createProfile()

Creates a reusable compression profile optimized for specific data types.

Syntax
async createProfile(name: string, sampleData: Array, options?: ProfileOptions): Promise
Returns

A Promise that resolves to a CompressionProfile object that can be used for future compressions.

Example
// Create a profile optimized for JSON data
const jsonSamples = [
  JSON.stringify({ id: 1, name: 'Test', data: [...] }),
  JSON.stringify({ id: 2, name: 'Sample', data: [...] }),
  // More samples...
];

const jsonProfile = await merlin.createProfile('json-data', jsonSamples, {
  iterationCount: 5,        // Number of optimization iterations
  optimizeFor: 'ratio',     // Optimize for maximum compression
  persistent: true          // Save profile for future sessions
});

console.log('Profile created:', jsonProfile.name);
console.log('Estimated compression ratio:', jsonProfile.estimatedRatio);

// Use the profile for compression
const compressedJson = await merlin.compress(jsonData, {
  profile: 'json-data'      // Reference profile by name
});

Managing Profiles

// List available profiles
const profiles = await merlin.listProfiles();
console.log('Available profiles:', profiles);

// Get profile details
const profile = await merlin.getProfile('json-data');
console.log('Profile details:', profile);

// Update an existing profile with new samples
await merlin.updateProfile('json-data', newSamples, {
  merge: true  // Merge with existing optimization data
});

// Delete a profile
await merlin.deleteProfile('json-data');

Performance Tuning

optimizePerformance()

Optimizes Merlin's performance for the current environment.

Syntax
async optimizePerformance(options?: PerformanceOptions): Promise
Returns

A Promise that resolves to a PerformanceResult object with recommended settings.

Example
// Optimize Merlin for the current environment
const performance = await merlin.optimizePerformance({
  testDuration: 5000,           // Run tests for 5 seconds
  balanceFactors: {
    speed: 0.7,                 // Prioritize speed (70%)
    memory: 0.3                 // Consider memory usage (30%)
  }
});

console.log('Optimized settings:', performance.recommendedSettings);

// Apply recommended settings
merlin.applySettings(performance.recommendedSettings);

Error Handling

The Merlin API provides structured error handling for managing different types of failures.

Error Types

Merlin defines several error types for specific scenarios:

Error Class Description Common Causes
MerlinError Base class for all Merlin errors General Merlin-related issues
CompressionError Error during compression Invalid data format, resource limitations
DecompressionError Error during decompression Corrupted data, incompatible format
ChecksumError Data integrity verification failed Data corruption, transmission errors
PatternError Error with compression pattern Invalid pattern definition, incompatible data
ResourceError Insufficient resources Memory limitations, CPU constraints

Error Handling Examples

// Try-catch error handling
try {
  const compressed = await merlin.compress(data);
  // Use compressed data...
} catch (error) {
  if (error instanceof CompressionError) {
    console.error('Compression failed:', error.message);
    console.error('Details:', error.details);
  } else if (error instanceof ResourceError) {
    console.error('Resource limitation:', error.message);
    console.error('Recommended max size:', error.recommendedMaxSize);
  } else {
    console.error('Unexpected error:', error);
  }
}

// Handling decompression errors
try {
  const decompressed = await merlin.decompress(compressedData);
  // Use decompressed data...
} catch (error) {
  if (error instanceof ChecksumError) {
    console.error('Data integrity check failed:', error.message);
    // Attempt recovery
    try {
      const recovered = await merlin.decompress(compressedData, {
        verifyChecksum: false,
        attemptRecovery: true
      });
      console.log('Data recovered with potential loss');
    } catch (recoveryError) {
      console.error('Recovery failed:', recoveryError.message);
    }
  } else {
    console.error('Decompression error:', error);
  }
}

Error Recovery

Some Merlin errors provide recovery methods:

try {
  const decompressed = await merlin.decompress(compressedData);
} catch (error) {
  if (error.canRecover()) {
    // Get recovery options
    const recoveryOptions = error.getRecoveryOptions();
    console.log('Recovery options available:', recoveryOptions);
    
    // Attempt recovery with recommended options
    try {
      const recovered = await error.recover();
      console.log('Recovery successful. Recovered data:', recovered);
    } catch (recoveryError) {
      console.error('Recovery failed:', recoveryError);
    }
  } else {
    console.error('Unrecoverable error:', error);
  }
}